MongoDB Upgrade Paths

MongoDB Upgrade Paths

Operational reference for MongoDB major-version upgrades on self-managed deployments. Covers the supported version sequence, Feature Compatibility Version (FCV) lifecycle, rolling replica-set and sharded-cluster procedures, driver matrices, pre-/post-upgrade verification, and the common failure modes that show up in real customer upgrades.

Active customer context (2026): Goldman Sachs “Cookie” 7.0 → 8.0 and Straight-to-8 self-managed initiatives. Both rely on every section below being correct.

When to use this skill

When NOT to use this skill

Quick reference

Question Answer
Supported upgrade hop One major version at a time: 4.4 → 5.0 → 6.0 → 7.0 → 8.0
Can I skip a major version? No. Even “Straight-to-8” walks through each hop sequentially.
Replica-set upgrade order Secondaries (one at a time) → arbiters → primary (via rs.stepDown())
Sharded-cluster order Stop balancer → config servers → shards → mongos → re-enable balancer → FCV
FCV command db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true })
Atlas FCV pin window 4 weeks — auto-unpins on next maintenance after expiry
Self-managed FCV pin window No enforced limit — burn-in 1–4 weeks recommended by policy
Java driver for 8.0 5.1+ (recommended 5.4+), JRE 11+ recommended
Top failure mode Driver mismatch (Section 10.1)
Customer-visible regression that has no upstream SOP Cold WiredTiger cache after rolling upgrade (Section 8)

1. Version upgrade paths

MongoDB enforces strictly sequential major-version upgrades. You cannot skip a major release on a single cluster. The supported sequence today is:

4.4  →  5.0  →  6.0  →  7.0  →  8.0

Each arrow above is one full upgrade cycle: bump binaries → wait for cluster to stabilize → set FCV → only then start the next hop.

Hard rules

“Straight-to-8” jump pattern

“Straight-to-8” is a customer-facing label for rapid sequential upgrades on the same cluster, not a literal skip. The pattern compresses the 4.4 → 8.0 ladder into a small number of well-rehearsed maintenance windows by:

  1. Doing back-to-back binary-only upgrades through 5.0, 6.0, 7.0 with the FCV held at the source version for each hop (so each hop remains downgrade-eligible until validation).
  2. Validating workload + drivers between hops, but deferring setFeatureCompatibilityVersion until the cluster is sitting on the 8.0 binary set with all health and driver tests passing.
  3. Pinning FCV to "7.0" before the 7.0 → 8.0 hop so you retain a full 4-week binary-downgrade window from 8.0 back to 7.0 if 8.0 misbehaves under production load.

Straight-to-8 is not a documented MongoDB product feature — it is an operational rollout pattern. The supported upgrade matrix is unchanged: 4.4 → 5.0 → 6.0 → 7.0 → 8.0.

Pre-flight: identify your starting version honestly

db.version();                                            // binary
db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 });  // FCV

If FCV lags binary by more than one major version (e.g., binary 7.0 with FCV "5.0"), stop. You must walk FCV forward one hop at a time before any further binary change.


2. Feature Compatibility Version (FCV)

FCV is the gate between binary upgrade and feature activation. It exists so customers can roll new binaries first, verify stability, then opt into the new on-disk and protocol features once they are confident no downgrade is needed.

Key facts

db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true });

Downgrade preservation window

Burn-in recommendation

After binary upgrade, run without advancing FCV for a deliberate burn-in (1–4 weeks is typical). Only set FCV to the new value when:


3. Rolling upgrades — replica sets

The rolling upgrade contract: at any moment, the replica set retains a writable primary and a majority of voters.

Order

  1. Secondaries, one at a time. Shut down mongod, swap the binary, restart. Wait for the node to return to SECONDARY (it may transit STARTUP2 or RECOVERING first — this is normal). Do not start the next node until rs.status() shows the current one back in SECONDARY.
  2. Arbiters (if any). Arbiters are stateless and have priority 0 by default. Upgrade them at any point after the secondaries but before the primary. Treat them like a tiny secondary: stop, swap, start, confirm ARBITER state. Do not make the arbiter the last node — its election votes matter when the primary steps down.
  3. Primary, last. Connect mongosh to the primary and run:
rs.stepDown();   // default freeze: 60s on 6.0+, 120s on older versions

Stepping down is preferable to a hard shutdown because it triggers a clean election. Then upgrade the now-secondary former primary.

Election timing

Arbiter handling


4. Sharded cluster upgrades

Sharded clusters have three component classes — config servers, shards, mongos — and the upgrade ordering is strict.

Order (8.0 example, applies to every major hop)

Pre-step 0 — Disable the balancer before any binary change:

sh.stopBalancer();
sh.isBalancerRunning();   // confirm "false"

Then walk the tiers in this strict order:

  1. Config server replica set (CSRS). Apply the rolling replica-set procedure (Section 3) to the config servers.
  2. Shards next. For each shard (replica set), repeat the rolling procedure. Upgrade least-critical shards first if possible — most v8 sharded upgrade incidents originate in the shard phase, and walking from least to most critical limits blast radius.
  3. mongos routers last. mongos is stateless, so you can drain connections and restart them in parallel groups.

Post-step — Re-enable balancer, then set FCV only after all three tiers are upgraded and stable:

sh.startBalancer();
db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true });

Version-skew rules

Config shard caveat (8.0+)

If your cluster uses the config shard topology (config server doubling as a data shard, introduced in 7.0), you must run transitionToDedicatedConfigServer before downgrading FCV below 8.0. There is no equivalent path forward — this is a downgrade-only gate.


5. Driver compatibility

A driver mismatch is the most common day-1 production incident after a server upgrade. The driver must support the target server version before binaries change.

Java driver (the matrix that matters for most enterprise customers)

MongoDB Server Minimum Java driver Notes
4.4 4.1+
5.0 4.3+ First version supporting timeseries and load-balanced topology.
6.0 4.7+ Required for queryable encryption beta.
7.0 4.10+ 4.10.x is the LTS line that paves the way to 5.x.
8.0 5.1+ (recommended 5.4+) Requires Java 8 minimum; Java 11+ recommended; Java 17 LTS supported.

Retry semantics

Other drivers — quick rules

The canonical matrix lives at https://www.mongodb.com/docs/drivers/ — verify there for the exact patch level before any production upgrade.


6. Pre-upgrade checks

Run these checks before touching binaries. Missing any one of them is how upgrades fail in the middle of the maintenance window.

Required gates

  1. All nodes on the prerequisite version, FCV matches:
    db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 });
  2. No in-flight index builds:
    db.currentOp({ "command.createIndexes": { $exists: true } });
    db.currentOp({ msg: /Index Build/ });
    An interrupted index build leaves the collection in unfinished state. Either let the build complete or dropIndex and restart it after the upgrade.
  3. Change-stream consumers are resumable: confirm every consumer is storing _id (resume token) durably so it can resume past the maintenance window. If oplog rolls beyond the last-seen token during upgrade, the consumer must do a full reseed — surface this risk to the customer in writing.
  4. Oplog window comfortably exceeds expected maintenance duration:
    rs.printReplicationInfo();   // shows oplog length
    Aim for ≥ 4× the expected maintenance window (e.g., 4-hour oplog for a 1-hour upgrade). If short, raise oplog size before upgrade with replSetResizeOplog.
  5. Backup verified, not just taken. Restore the latest snapshot into a scratch cluster and run a smoke query. Untested backups are not backups.
  6. Driver compatibility deployed. Drivers should be on the target-server-compatible version for at least 7 days before the upgrade.
  7. Replica lag < 5 seconds on every secondary:
    rs.printSecondaryReplicationInfo();
  8. Disk free space: at minimum 25% headroom. WiredTiger needs room for the new on-disk format files even when FCV is held.
  9. Compatibility scan: review the target version’s “Compatibility Changes” doc and grep the application for any removed/deprecated command names (e.g., $listLocalSessions semantics, geoNear aggregation pipeline equivalents).

Index builds and commit quorum (8.0 nuance)

Starting in MongoDB 8.0, the commit quorum specifies how many nodes must be ready to finish the index build before the primary commits, while the write concern specifies how many nodes must replicate the commit oplog entry before the command returns success. This is a semantic change vs. 7.0 and earlier. If your application sets commitQuorum explicitly, audit those calls.

Default commitQuorum is votingMembers (all voting data-bearing members). Lowering it (e.g., majority) can prevent index builds from stalling on a lagging secondary during the upgrade window.


7. Upgrade rollback

Binary downgrade window

The rollback path depends entirely on whether FCV was pinned before the upgrade.

State Downgrade possible?
Binary upgraded, FCV still at N-1, no new features used Yes — binary downgrade is safe and reversible.
Binary upgraded, FCV advanced to N, no new features used Possible only by reversing FCV first (setFeatureCompatibilityVersion: "N-1", confirm: true) then binary downgrade.
Binary upgraded, FCV advanced to N, new features used (encrypted ranges, new index types, etc.) Not supported without removing the persisted feature data first. May require MongoDB Support assistance.

The “Point of No Return” command

setFeatureCompatibilityVersion with the new version is the Point of No Return for the rollback window. Once executed and confirmed, the FCV gate closes:

Practical rollback playbook

  1. Detect regression (latency, error rate, missing feature support in application).
  2. Confirm FCV state. If FCV is still "N-1":
    • Apply rolling binary downgrade in reverse order: primary last for replica sets; mongos → shards → config servers for sharded clusters.
  3. If FCV has been advanced:
    • Audit for use of new-version-only features. Atlas indexes that depend on new operators, new index types, sharded time-series with reshardCollection, etc.
    • If clean, run setFeatureCompatibilityVersion: "N-1", confirm: true. If features are in use, open a Premium Support case before touching anything.
  4. Document the rollback as a near-miss event regardless of whether downgrade succeeded. The data is the most valuable input for the next upgrade.

Status: known operational gap, manual SOP, not documented in upstream MongoDB upgrade procedures.

The problem

Goldman Sachs “Cookie” cluster upgrades from 7.0 → 8.0 exposed a recurring symptom: secondaries that come back online after the binary swap return to SECONDARY state quickly, but their WiredTiger block cache is cold. When the load balancer (or a stepDown on the primary) shifts read traffic onto a freshly upgraded secondary, query latency spikes 10–100× until the cache rewarms. For latency-sensitive workloads (Cookie’s read SLA is sub-10ms p99), this looks like a production incident.

Why MongoDB docs don’t cover it

The WiredTiger cache is in-process memory; it always starts empty after mongod restart and warms naturally with traffic. Standard MongoDB guidance is “let it warm”. For most workloads that is fine. For low-latency, predictable-workload customers it is not fine — the natural warm-up takes minutes and the latency degradation is customer-visible.

Pre-warming SOP (manual, customer-driven)

This is the pattern Goldman Sachs operators developed. There is no automated tool from MongoDB.

  1. After binary swap and SECONDARY state, hold the node out of routing (e.g., hidden: true, priority: 0, or remove from driver readPreference rotation).
  2. Run a scripted touch query workload that mimics the production read pattern — typically a parallel sweep over the most-frequently-accessed indexes (covered queries against the hot collections), constrained to a small key range so the OS cache and WT cache fill in a controlled order.
  3. Monitor cache pressure with:
    db.serverStatus().wiredTiger.cache;
    Watch bytes currently in the cache and pages read into cache — when growth flattens, warm-up is complete.
  4. Reset hidden: false (or re-add to routing rotation) only after the cache fills to ~70% of cacheSizeGB and p99 latency from the touch workload matches pre-upgrade baseline.

Recommendation to MongoDB Engineering

This SOP is a documentation and tooling gap. The TAM team should propose:

Until then, the manual SOP above is the supported pattern for latency-critical customers.


9. Upgrade event coverage

An upgrade is a scheduled maintenance event, not a deploy. TAM-owned upgrades follow a three-phase template with explicit sign-offs.

Pre-event (T-7 days through T-0)

During event (T-0 → T-end)

Post-event (T-end → T+72 hours)

Sign-off artefacts

A complete upgrade event produces:

Anything less and the event is not auditable.


10. Common upgrade failures

The recurring failure modes, ranked by frequency from real customer post-mortems:

10.1 Driver mismatch

Symptom: application throws connection errors immediately after the primary steps down, or after the cluster is fully on the new binary.

Root cause: driver was not updated to a server-compatible version before the upgrade. Common in environments where multiple application teams share a database — one team upgrades, ten others discover they can’t connect.

Mitigation:

10.2 FCV unpinned too early

Symptom: customer wants to roll back after the upgrade because of unrelated production incident; rollback path is closed.

Root cause: operator (or Atlas auto-unpin) advanced FCV before the burn-in window completed.

Mitigation:

10.3 Index build conflicts

Symptom: secondary returns to RECOVERING and never reaches SECONDARY, or replSetReconfig rejects the new member set because of inconsistent indexes.

Root cause: an in-flight index build was interrupted by the binary swap, leaving the collection in an inconsistent state across members. Or, the new version changed default index behavior (e.g., 4.2 → 4.4 wildcard index, 7.0 → 8.0 commit-quorum semantics).

Mitigation:

10.4 mongos version skew

Symptom: queries fail intermittently with IncompatibleServerVersion or routing errors after partial upgrade.

Root cause: mongos pool contains a mix of N-1 and N binaries while FCV has been advanced to N, or shards on mixed versions.

Mitigation:

10.5 PSA topology stepdown failure

Symptom: during the upgrade of the data secondary in a Primary-Secondary-Arbiter (PSA) topology, the primary steps down and cannot find a majority for a new election. Cluster is read-only.

Root cause: PSA only has two data-bearing voters. While the secondary is offline for upgrade, the primary alone cannot achieve majority for any write that requires w: majority, and the arbiter cannot help. If the primary then steps down, the cluster has no eligible primary until the secondary returns.

Mitigation:

10.6 Oplog window overflow during long maintenance

Symptom: secondary returns from upgrade and immediately enters full initial sync because its lag exceeded the oplog window.

Root cause: maintenance ran longer than expected; oplog size was sized for steady-state replication lag, not for a multi-hour maintenance window.

Mitigation:

10.7 Cold-cache latency regression (Section 8)

Symptom: post-upgrade p99 read latency spikes for minutes after each rolling upgrade step.

Root cause: WiredTiger cache reset on mongod restart; default warm-up is uncontrolled.

Mitigation: pre-warming SOP from Section 8. Document gap; advocate for upstream tooling.


Quick command reference

// Identify state
db.version();
db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 });
rs.status();
rs.printReplicationInfo();
rs.printSecondaryReplicationInfo();

// Replica set upgrade
rs.stepDown();        // step down primary cleanly

// FCV — the Point of No Return
db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true });

// Sharded cluster
sh.stopBalancer();
sh.startBalancer();
sh.status();

// Index / op state
db.currentOp({ "command.createIndexes": { $exists: true } });
db.adminCommand({ listShards: 1 });

// Oplog resize
db.adminCommand({ replSetResizeOplog: 1, size: 102400 });

// WiredTiger cache state (Cookie pre-warm SOP)
db.serverStatus().wiredTiger.cache;

Sources