Performance Tuning Checklist
Parent: MongoDB Spark Connector and Databricks Integration · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a8bd98441f1a
↓ Facts as markdownall context files
Source-anchored research for Performance Tuning Checklist. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- Attach a connector comment to read or write operations and inspect the MongoDB Database Profiler to connect Spark activity to database work. On the server, investigate inefficient indexes, query structure, memory pressure, and connection load before treating the connector as the sole bottleneck. (medium confidence) [source]
How it works
- Distinguish connection establishment timeout from query execution control. MongoDB documents connectTimeoutMS for the connection phase, warns against using socketTimeoutMS to prevent long-running server operations, and recommends maxTimeMS() when the server should cancel a long-running operation. (low confidence) [source]
- Treat connector-upgrade performance regressions as workload-specific until reproduced. A 2023 user reported a v10.1 regression relative to v2.4 and attributed recovery to checkpointing after a read; the claimed generated-pipeline cause is an unverified self-diagnosis, not a general reason to checkpoint every MongoDB read. (low confidence) [source]
- Verify effective configuration before benchmarking: when the same connector setting appears in connection.uri and separately, the URI value takes precedence. This can make a read or write tuning experiment appear ineffective. (medium confidence) [source]
How-to and procedures
- Avoid increasing schema sampleSize to the entire collection as a routine fix for missing fields. MongoDB documents sampling and schemaHints; a February 2026 Databricks thread reports a driver restart after sampleSize=100000 and recommends supplying a schema. The thread does not provide a verified heap analysis. (medium confidence) [source]
- Use Spark UI spill metrics and task-duration distribution to guide shuffle tuning. AQE can split eligible skewed join partitions when both skew thresholds are exceeded; enabling AQE does not guarantee that every skewed stage is repaired. (high confidence) [source]
- Use AQE to evaluate downstream shuffle skew: Spark can split skewed join partitions, while Databricks cautions that eligibility and size thresholds govern intervention. AQE skew handling is not evidence that MongoDB source partitions or all join types will be balanced automatically. (high confidence) [source]
Comparisons and alternatives
- Correlate Spark concurrency with MongoDB connection and CPU pressure. MongoDB recommends increasing maxPoolSize or active threads when capacity is underused, but decreasing them when excessive connection attempts raise CPU. Budget pools across clients and replica-set members rather than treating one pool limit as a deployment-wide cap. (low confidence) [source]
- Cache reusable Spark DataFrames selectively and unpersist them after use. Spark documents a memory-versus-compression tradeoff for cached columnar batches: larger batches may help compression but can cause out-of-memory failures. (low confidence) [source]
- For sampled or size-based batch reads, smaller target partition sizes create more partitions, but sampled boundaries and average document sizes do not guarantee equally sized partitions. Compare actual task durations when testing partition sizes instead of assuming equal work. (medium confidence) [source]
Facts and statements
- Begin with the longest Spark stage and inspect task-duration variation and spill before increasing compute. This distinguishes uneven work and memory pressure from a general lack of resources; Spark's runtime statistics and query-plan estimates provide additional diagnostic evidence. (high confidence) [source]
- Treat connector read partitions and Spark shuffle partitions as separate controls. MongoDB batch partitioners divide collection reads; spark.sql.shuffle.partitions controls shuffle work for joins and aggregations. Spark file-source partition settings such as spark.sql.files.maxPartitionBytes do not tune MongoDB collection scans. (high confidence) [source]
- Benchmark partition size and partitioner against representative document sizes and task balance. Smaller connector partitions create more tasks, not a guaranteed speedup; the 2020 practitioner benchmark used connector 2.4.1 and does not establish the best setting for newer releases. (medium confidence) [source]
- Reduce transferred data with a selective aggregation.pipeline, but check partition compatibility: the connector documentation warns that $group does not work with partitioners that create multiple partitions. (low confidence) [source]
- For batch writes, benchmark maxBatchSize and ordered alongside operationType and required writeConcern. These options control bulk operation count, ordering, insert/update/replace behavior and acknowledgment semantics; the documentation does not prescribe a universally optimal batch size. (low confidence) [source]
- Reduce unnecessary rows and columns before increasing memory. The connector accepts an aggregation.pipeline that runs before data reaches Spark; Databricks recommends reducing processed rows and columns to relieve execution-memory pressure. Confirm the custom pipeline remains compatible with the partitioner. (high confidence) [source]
- Do not add a global $group to a custom aggregation pipeline while using a connector partitioner that creates multiple partitions: the read documentation explicitly rules out that combination. Partitioners also do not tune streaming reads. (low confidence) [source]
- Benchmark maxBatchSize as a maximum operation count per bulk write, alongside ordered, operationType, upsertDocument, and writeConcern settings. These settings affect batching or write semantics; the configuration reference does not establish an optimal batch size or a universal throughput improvement from unordered writes. (low confidence) [source]
- Do not assume increasing executor concurrency will scale throughput indefinitely. MongoDB documents that concurrent requests can overwhelm database capacity, while Databricks documents that skew leaves other cores idle. Diagnose database saturation and Spark task imbalance before scaling the cluster. (high confidence) [source]
Children
- No children recorded.