Connector Error Patterns
Parent: MongoDB Spark Connector and Databricks Integration · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a04dd23fc577
↓ Facts as markdownall context files
Source-anchored research for Connector Error Patterns. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- For reads, the current connector defaults to FAILFAST, which throws when a document does not match the expected schema; PERMISSIVE sets incompatible values to null and can store the invalid document as Extended JSON, while DROPMALFORMED ignores the document. Batch schema inference samples records, so separate document/schema parsing failures from connection and runtime errors. (high confidence) (verified as of 2026-10-23) [source]
Problems, failure modes and limitations
- Connector/runtime compatibility is version-specific: MongoDB lists Connector 11.1.0 with Spark 4.0 or later and MongoDB 4.2 or later; the 11.0 release adds Spark 4.0 support, which requires Scala 2.13, and Spark says Scala applications must use the same Scala version as the Spark build. Compare the connector version, Spark runtime, and Scala artifact suffix when a linkage error appears. (high confidence) (verified as of 2026-10-23) [source]
- A connection timeout can come from a versioned option-name mismatch: a Databricks report for Connector 10.0.5 timed out while using spark.mongodb.input.uri, and the author later marked spark.mongodb.read.connection.uri as the working option. Current MongoDB read configuration uses the spark.mongodb.read. namespace. Check the configuration key expected by the installed connector before treating a timeout as proof of network failure. (medium confidence) (verified as of 2026-10-14) [source]
- On Databricks standard access compute with Unity Catalog, a MongoDB read can fail with SparkSecurityException [INSUFFICIENT_PERMISSIONS] for SELECT on ANY FILE even when the same runtime and package work on dedicated compute. Databricks attributes this case to custom data-source access not being governed by Unity Catalog and recommends Lakehouse Federation or dedicated compute; its current Spark data-source guidance also excludes custom third-party Spark JARs from standard and serverless compute. (high confidence) (verified as of 2026-09-28) [source]
- Connector 10.0 and later uses the Spark provider name mongodb. A MongoDB forum report with Failed to find the data source: mongo shows format("mongo") even though the author says the connector JAR was installed; the thread does not confirm the cause. Databricks documentation says custom connector JARs must be installed on the cluster and that third-party Spark JARs are supported only on dedicated compute. Check the provider name, artifact/Scala suffix, and runtime classpath before treating a provider-discovery error as a database failure. (medium confidence) (verified as of 2026-10-14) [source]
- When MongoTypeConversionException says Cannot cast STRING into a DoubleType, Databricks identifies a mismatch between the source value and the Spark schema, with ANSI compliance enforcing strict conversion. Its guidance is to validate and align source types; disabling ANSI can return nulls and hide invalid values. MongoDB's schemaHints option can specify known field types during inference, but it does not repair inconsistent source data. (medium confidence) (verified as of 2026-10-14) [source]
- If a read fails with the MongoDB server response Invalid $project: FieldPath field names may not start with '$', inspect the generated projection and field-path semantics. A report for Connector 10.2.2/Spark 3.3.4 shows this server-side error for a dollar-prefixed key. MongoDB stores dollar-prefixed field names but restricts their direct queryability; its documentation describes using $getField with $literal to retrieve such a field. This signature alone does not indicate a classpath or transport failure. (medium confidence) [source]
Comparisons and alternatives
- One Databricks community report describes a driver restart after sampleSize=100000 and mergeSchema=true on documents with 100+ fields; a responder attributes it to schema-inference memory and suggests an explicit schema or smaller sample. The report uses the older spark.mongodb.connection.uri option and the thread questions whether it is the newer connector, so treat this as a version-specific lead rather than a current connector threshold. (low confidence) (verified as of 2026-02-10) [source]
- A legacy 2019 dependency-resolution report for mongo-spark-connector_2.11:2.2.0 showed module not found alongside Maven Central and Bintray hostname-resolution failures; the repository maintainer also said that requested release was no longer available. A module-not-found error can therefore originate in artifact coordinates, repository availability, or network resolution rather than connector runtime behavior. This v2 example does not diagnose current v10 or v11 installations. (low confidence) [source]
Facts and statements
- Community reports show NoSuchMethodError in version-alignment cases: a July 2023 Databricks report for Connector 10.2.0 on Databricks 13.2/Spark 3.4.0 showed missing DataType.sameType and later reported this case fixed in Databricks Runtime 13.3 LTS; a separate November 2023 upgrade report showed missing RowEncoder.apply on Databricks 14.x/Spark 3.5.0, and a MongoDB forum report showed StructType.toAttributes missing with a Scala 2.13 connector JAR on Spark compiled for Scala 2.12. These examples make API or binary alignment a diagnosis to check, not a universal explanation. (medium confidence) [source]
- Driver-side client success does not rule out Spark JVM TLS or worker reachability failures. One Atlas report shows SSLHandshakeException with PKIX path building failed while CA files were stored in DBFS; a Databricks Runtime 14.3/Spark 3.5.0 DocumentDB report timed out despite PyMongo working, and its author later said that adding a JVM truststore resolved the case. Check JVM certificate trust and executor DNS, routes, and port 27017 access before attributing this signature to credentials or a connector defect; these are user reports, not a universal diagnosis. (medium confidence) (verified as of 2026-01-12) [source]
- Connector compatibility changes by release: MongoDB states that Connector 11.0 adds Spark 4.0 and later support and that Spark 4.0 requires Scala 2.13, while Connector 10.7 supports Spark 3.2 through 3.5. Compare the runtime Spark version and Scala binary suffix with the connector artifact before attributing a NoSuchMethodError to connector code. A Databricks 13.2/Spark 3.4.0 report for Connector 10.2.0 with the _2.12 artifact showed DataType.sameType missing and later said DBR 13.3 LTS fixed that case. (medium confidence) (verified as of 2026-10-14) [source]
Children
- Spark and Scala connector binary compatibility (frontier)
- Connector configuration namespace migration (frontier)
- Databricks cluster JVM TLS and network reachability (frontier)
- Schema inference and document parse modes (frontier)
- Spark connector and runtime compatibility (frontier)
- Databricks connector loading and access modes (frontier)
- JVM TLS and executor reachability (frontier)
- MongoDB type conversion and field projection (frontier)
Frontier under this node: Connector configuration namespace migration, Databricks cluster JVM TLS and network reachability, Databricks connector loading and access modes, JVM TLS and executor reachability, MongoDB type conversion and field projection, Schema inference and document parse modes, Spark and Scala connector binary compatibility, Spark connector and runtime compatibility