Data Federation over customer S3 with partitioning
Parent: Storage and Data-Lifecycle Cost Levers · Published reference · snapshot 2026-10-21 · skill ranked-ab842e47
↓ Facts as markdownall context files
Source-anchored research for Data Federation over customer S3 with partitioning. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- Partition-key predicates are a general scan-control pattern across object-store query engines: AWS Athena describes pruning S3 partitions that do not match query predicates, and Trino's Hive connector documents dynamic partition pruning and an optional requirement for partition filters. These are cross-engine context and do not mean Athena or Trino settings apply to Atlas Data Federation. (medium confidence) [source]
How it works
- Data Federation adds partition attributes even when a stored document already has a field with the same name, creating duplicate field names that MongoDB does not support. Use distinct partition names or `omitAttributes`; a community user also reported that renaming a colliding field resolved their partition query issue. (high confidence) [source]
- Give path partition attributes names that do not collide with fields inside the documents. Atlas warns that it adds path attributes even when a same-named document field exists, creating duplicate field names; a 2023 community report said renaming a colliding ObjectId path attribute restored query filtering. (medium confidence) [source]
How-to and procedures
- Avoid layouts that create many small files or excessive partitions. MongoDB says many small files compound per-file resource use and large files can require unnecessary downloads; AWS Athena separately warns that too many S3 partitions add file and partition-filtering overhead, while too few can increase scans. (high confidence) [source]
Measurements and reference values
- Selective partition matches can reduce Data Federation's processed bytes and charges, but partitioning does not guarantee savings: a query that reads the whole collection still processes the whole collection. MongoDB's example contrasts reading one 1 GB partition from a 10 GB file layout with processing all 10 GB when no partition can be skipped. (high confidence) [source]
- MongoDB's billing page lists Data Federation processing at $5 per TB with a 10 MB minimum per query. (low confidence) (verified as of 2026-10-18) [source]
Problems, failure modes and limitations
- MongoDB recommends ordering S3 partition fields from the most frequently queried to the least frequently queried. A query that skips an earlier path field can use matching earlier partitions more efficiently than later ones; fields absent from the configured path cannot prune files. (high confidence) [source]
Facts and statements
- For customer-managed S3, a collection's `dataSources.path` can map S3 key segments to computed partition attributes, and predicates on those attributes can target matching prefixes and files. Independent OneUptime and Medium write-ups describe the same path-to-field and filter pattern, but provide no measured benchmark. (medium confidence) [source]
- Atlas Data Federation maps configured S3 stores to virtual databases and collections. Its data-source path syntax can parse key or filename segments into typed partition attributes and add those attributes to query documents. (high confidence) [source]
- Atlas charges separately for data processed and data returned or transferred. A filter on mapped S3 path attributes can target matching files and reduce processed bytes, but a broad query such as a blank $match still reads the full collection, so partitioning alone does not guarantee lower processing cost. (high confidence) [source]
Children
- No children recorded.