Crash Stack Trace Symbolization

Parent: KB-Backed Troubleshooting · Published reference · snapshot 2026-10-15

Depth-first rabbithole dossier for Crash Stack Trace Symbolization; source-anchored research pack.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

How it works

[low] When mongod and mongos processes receive a SIGUSR2 signal, they generate a backtrace for crash stack trace symbolization. Sources: https://www.mongodb.com/docs/manual/tutorial/manage-mongodb-processes.md [source]
[low] The symbolization process extracts the MongoDB version from the processInfo field of the trace document. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]
[low] If the binary name does not end with .debug, the symbolization process appends .debug to the binary name to locate the debug symbols file. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]
[low] The symbolization process retries if the ratio of not-found frames is 0.5 or higher, waiting 60 seconds between retries. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]
[low] Symbol lines are formatted as 'CODE {path:} {addr:}' during the symbolization process. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]

Measurements and reference values

[low] The symbolization tool identifies backtrace sections using the indicator '{"backtrace":' and skips lines containing 'Frame: 0x'. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]

Problems, failure modes and limitations

[low] WiredTiger panics, such as WT_PANIC with error code -31804, can trigger crash stack traces in mongod. Sources: https://www.percona.com/blog/resolving-mongodb-stack-trace/ [source]
[low] The symbolization tool exits with an error if it cannot find a JSON backtrace object in the input. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]

Facts and statements

[low] The default symbolizer path for crash stack trace symbolization is /opt/mongodbtoolchain/v5/bin/llvm-symbolizer. (verified as of 2026-10-15) Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]
[low] Crash stack traces may include stripped mongod binaries, such as ELF 64-bit LSB executable x86-64 for GNU/Linux 2.6.9. Sources: https://www.percona.com/blog/resolving-mongodb-stack-trace/ [source]
[low] Crash stack traces may include non-stripped mongod binaries with BuildID information, such as ELF 64-bit LSB shared object x86-64 for GNU/Linux 2.6.32. Sources: https://www.percona.com/blog/resolving-mongodb-stack-trace/ [source]
[low] Crash stack traces include mongod frames with mangled names like _ZN5mongo15printStackTraceERSo and demangled names like mongo::printStackTrace. Sources: https://www.percona.com/blog/resolving-mongodb-stack-trace/ [source]
[low] Crash stack traces include frames for assertion failures, such as mongo::fassertFailedNoTraceWithLocation and __wt_err_func. Sources: https://www.percona.com/blog/resolving-mongodb-stack-trace/ [source]
[low] Crash stack traces include frames for WiredTiger panic handling, such as __wt_panic, and system library frames like gsignal, abort, and clone. Sources: https://www.percona.com/blog/resolving-mongodb-stack-trace/ [source]
[low] Crash stack traces may include unresolved mongod frames with only offset addresses, such as mongod(+0x22744D9). Sources: https://www.percona.com/blog/resolving-mongodb-stack-trace/ [source]
[low] Crash stack traces for WiredTiger panics include frames for printStackTrace, fassertFailed, __wt_panic, __wt_bm_read, and WiredTigerIndexStandard::_insert. Sources: https://jira.mongodb.org/browse/SERVER-25067 [source]
[low] Crash stack traces for WiredTiger panics include frames for replication operations like applyOperation_inlock and ThreadPool::_workerThreadBody. Sources: https://jira.mongodb.org/browse/SERVER-25067 [source]
[low] Crash stack traces for WiredTiger panics include frames for status conversion like wtRCToStatus_slow and cursor operations like WiredTigerRecordStore6Cursor::seekExact. Sources: https://jira.mongodb.org/browse/SERVER-25067 [source]
[low] Crash stack traces for WiredTiger panics include frames for query execution stages like IDHackStage::work, UpdateStage::work, and update operations. Sources: https://jira.mongodb.org/browse/SERVER-25067 [source]
[low] If no trace document is provided via stdin, the symbolization tool prompts the user to provide the backtrace through stdin. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]
[low] The symbolization tool defaults to using the 'pr' option for --debug-file-resolver and 'classic' for --input-format. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]
[low] llvm-symbolizer accepts positional arguments or stdin values preceded by 'DATA' or 'CODE' to specify symbolization type, defaulting to 'CODE' if unspecified. Sources: https://llvm.org/docs/CommandGuide/llvm-symbolizer.html [source]
[low] Address calculation for symbolization determines addr_base based on ELF type (0 for type 3, frame['b'] for type 2, or soinfo vmaddr) and adds the frame offset. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]
[low] Frames missing the 'b' field are ignored during symbolization, with a reference to SERVER-58863 for discussions. Sources: https://raw.githubusercontent.com/mongodb/mongo/master/buildscripts/mongosymb.py [source]

Related concepts

Trace — is a part of Crash Stack Trace Symbolization
Crash — is a part of Crash Stack Trace Symbolization
Stack — is a part of Crash Stack Trace Symbolization
Symbolization — is a part of Crash Stack Trace Symbolization

Children

No children recorded.

← the whole tree · 3D view· how to read this page