Common Failure Modes
Parent: MongoDB Ops Manager and Cloud Manager · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-6a1c409b355b
↓ Facts as markdownall context files
Source-anchored research for Common Failure Modes. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- Cloud Manager rolling changes can stall when DNS or firewall problems prevent the MongoDB Agent from inspecting other members. Check inter-member reachability and agent access to cloud.mongodb.com over HTTPS before treating an In Progress deployment as a database defect. (low confidence) [source]
- Ops Manager 8.0.20 release notes document a fix for job workers stuck waiting on a data-fetch lock that stopped scheduled-job progress. When jobs stop advancing, check the installed patch level as well as configuration and resource health. (low confidence) (verified as of 2026-02-19) [source]
- Cloud Manager troubleshooting requires SAN-bearing TLS certificates for MongoDB Agents built with Go versions that no longer permit hostname validation through CommonName alone. Check certificate names when agent upgrades expose TLS failures. (low confidence) [source]
- Insufficient disk capacity can break backups or restores. A standalone backup blockstore can go offline when full; restoring compressed snapshots also requires space for both the archive and extracted database files. (medium confidence) [source]
Problems, failure modes and limitations
- TLS errors can block different control paths: an Ops Manager operator user reported an agent hostname/certificate mismatch, while another reported application startup failure caused by an untrusted certificate chain. Cloud Manager documents the requirement for SAN-bearing certificates. Check hostname coverage and CA trust on each connection. (medium confidence) [source]
- Automated restore can fail when source and target storage settings differ, including storage.engine, storage.directoryPerDB, WiredTiger block compression, or directoryForIndexes. Ops Manager reports mismatches after the attempt; correct target settings before retrying. (high confidence) [source]
- For recovery of Ops Manager itself, backing up database data without the original gen.key leaves a startup/decryption failure. The recovery guide also warns that starting Ops Manager before restoring its backing databases can create inconsistent state. (low confidence) [source]
- Online automation-config encryption adds a KMIP dependency. If an agent cannot decrypt its local backup and cannot reach Ops Manager, it enters an error state and retries. Hardware or agent-version changes can invalidate local artifacts, but reachable Ops Manager can supply fresh configuration automatically. (low confidence) [source]
- Historical operator defects can resemble configuration errors: Enterprise Kubernetes Operator 1.14.0 fixed failure to mount a custom CA into the Backup Daemon in hybrid mode, and 1.18.0 fixed a liveness probe that could report healthy after the automation agent died. These are documented fixes, not evidence that current versions retain those bugs. (low confidence) [source]
- DNS or firewall failures between MongoDB processes can prevent automation from determining peer state and completing a rolling change. Backup agents can likewise fail when replica-set connectivity or authentication fails. Test from the affected agent host. (medium confidence) [source]
- If backup processing cannot keep up before oplog history rolls over, backup can require resynchronization. Point-in-time recovery requires continuous oplog history; gaps can make a requested recovery point unusable. (low confidence) [source]
- If a source backup uses KMIP-based encryption at rest, the restore target needs access to the same KMIP server. The database can fail to start after restoration when that server is unavailable. (low confidence) [source]
- Do not classify every incomplete snapshot or frequent agent connection as a failure. A snapshot with complete:false may still be running, and short-lived automation connections are routine state checks. (medium confidence) [source]
Comparisons and alternatives
- Backup sizing guidance treats 2 TB per replica set or shard as a planning recommendation rather than an Ops Manager limit. Resource consumption and recovery objectives require capacity planning rather than treating that number as a hard failure threshold. (medium confidence) [source]
Facts and statements
- A Cloud Manager automation change can stall with open /data/db/mongod.lock: permission denied. The agent user needs ownership and read/write access to the database and log paths. (low confidence) [source]
- A September 2023 practitioner report describes SSL peer certificate validation failed: self signed certificate after enabling Cloud Manager TLS. The thread has no resolution; it demonstrates a client validation symptom, not a proven Cloud Manager defect or an established certificate fix. (low confidence) [source]
- An automated restore of only one shard fails; Ops Manager requires restoring all shards. Restored encrypted data also requires the target to reach the same KMIP server, otherwise mongod fails to start. (low confidence) [source]
- IBM Software Hub 5.2.x documents Ops Manager backups as disabled by default in its integration. A deployed or Running Ops Manager resource therefore does not by itself prove backups are configured; the integration requires configuring backup stores and explicitly enabling backup. (low confidence) (verified as of 2026-10-23) [source]
- A published third-party storage-backup experiment waits for the Ops Manager snapshot job to reach READY before taking array snapshots, catalogs complete sets, then checks restored document counts against captured baselines. This provides a concrete validation pattern; its demonstrated restore time is not a general Ops Manager RTO guarantee. (low confidence) [source]
- An automated sharded-cluster restore must restore all shards; attempting to restore one shard in isolation fails. (high confidence) [source]
- A reproduced third-party backup lab using Ops Manager demonstrates that dropping a database propagates through replica-set members. Replication therefore does not protect against that logical deletion; recovery requires a usable earlier backup. (low confidence) [source]
Children
- No children recorded.