Databricks Workspace Integration
Parent: MongoDB Spark Connector and Databricks Integration · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a8bd98441f1a
↓ Facts as markdownall context files
Source-anchored research for Databricks Workspace Integration. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- Microsoft's MongoDB migration tutorial installs org.mongodb.spark:mongo-spark-connector_2.12:3.0.1 and shows legacy input/output settings and "mongo" writes. MongoDB's modern setup guide instead specifies "mongodb" for connector 10.0.0 and later. Treat these examples as different connector generations when adapting workspace notebooks. (high confidence) [source]
How-to and procedures
- Install the MongoDB Spark Connector as a compute library through Compute > cluster > Libraries > Install new > Maven, then restart compute before using it in a notebook. Databricks documents this custom-connector workflow, and Microsoft independently documents it for a MongoDB migration notebook. (high confidence) [source]
Problems, failure modes and limitations
- A Databricks timeout after upgrading the connector can be a configuration error: one reported job targeted localhost:27017 after using spark.mongodb.input.uri, and the author resolved it by changing to spark.mongodb.read.connection.uri. Inspect the effective endpoint and option names before attributing every timeout to network routing. (medium confidence) [source]
- Databricks documents a MongoDB read failure on standard compute with INSUFFICIENT_PERMISSIONS / SELECT on any file, while the same runtime and package work on dedicated compute. A June 2026 community thread also reports UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION and recommends dedicated compute for the MongoDB Spark data-source path. These are documented cases, not an exhaustive error catalog. (medium confidence) (verified as of 2026-06-04) [source]
- Databricks secret redaction is limited to literal secret values and cannot prevent authorized users from retrieving secrets or transformed values from escaping redaction. Scope permissions remain necessary even when notebook output displays REDACTED. (low confidence) [source]
- For dependency distribution, Databricks recommends package repositories, workspace files or Unity Catalog volumes. DBFS-root library storage is deprecated and disabled by default from Databricks Runtime 15.1 onward. This storage guidance is separate from whether a particular connector can execute on the chosen compute access mode. (low confidence) (verified as of 2026-09-11) [source]
- A MongoDB read on standard access mode can fail with INSUFFICIENT_PERMISSIONS and a missing SELECT-on-any-file privilege. Databricks attributes this case to ANY FILE evaluation for a source outside Unity Catalog governance and documents dedicated compute as a remedy for retaining the Spark connector. The current custom-connector guide also directs MongoDB workloads to dedicated compute. (medium confidence) (verified as of 2026-09-28) [source]
Comparisons and alternatives
- A custom connector installed on a Databricks cluster is available only on that cluster. Workspace setup must therefore include the library on the compute that actually executes each notebook or job, rather than assuming one cluster installation makes the connector workspace-wide. (low confidence) [source]
Facts and statements
- Databricks documents MongoDB as a custom third-party Spark connector for dedicated classic compute. Its September 2026 guidance excludes custom third-party Spark JARs from SQL warehouses, serverless and standard access mode compute; generic Maven installation support does not establish MongoDB connector support on those compute types. (high confidence) (verified as of 2026-09-28) [source]
- For the modern connector, notebook DataFrame reads use format("mongodb") with a connection URI, database and collection; writes use the same format and connection options. MongoDB documents separate spark.mongodb.read.connection.uri and spark.mongodb.write.connection.uri SparkSession settings. (high confidence) [source]
- Store database credentials or a credential-bearing connection URI in a Databricks secret scope, assign scope permissions, and retrieve values at runtime with dbutils.secrets.get. Databricks documents this secret workflow and uses secret retrieval in its external Spark data-source examples. (high confidence) [source]
- A first-hand on-premises integration walkthrough tests host/port reachability with socket.create_connection and then checks the MongoDB endpoint with PyMongo before running a Spark read. These are driver-side diagnostic steps; the walkthrough does not demonstrate executor-wide connectivity validation. (low confidence) [source]
- MongoDB's integration article documents a curated PySpark notebook opened through New > Data > Native Integrations > MongoDB. The article was updated December 2, 2025; this establishes a documented onboarding path at that date, not verification that the October 2026 workspace UI still exposes the same navigation. (low confidence) (verified as of 2025-12-02) [source]
- For the modern connector API, use spark.read.format("mongodb") with connection.uri, database, and collection options. Writes use the same format and connection options through DataFrameWriter. MongoDB documents the "mongodb" format for connector 10.0.0 and later, and Databricks supplies matching read/write examples. (high confidence) [source]
- Atlas requires a database user with access to the requested databases; an Atlas application login is a separate identity. The Databricks compute environment also needs an allowed network path: public IP access-list entries, peering, or private endpoints, with outbound firewall access where required. Installing a workspace library alone does not satisfy these prerequisites. (low confidence) [source]
Children
- No children recorded.