# Data Federation over customer S3 with partitioning > Source-anchored research for Data Federation over customer S3 with partitioning. Run status: COMPLETED. Parent: [Storage and Data-Lifecycle Cost Levers](https://llms-explorer.com/tree/storage-and-data-lifecycle-cost-levers/) · 6 facets · 10 facts · page: https://llms-explorer.com/tree/data-federation-over-customer-s3-with-partitioning/ ## Definitions - Partition-key predicates are a general scan-control pattern across object-store query engines: AWS Athena describes pruning S3 partitions that do not match query predicates, and Trino's Hive connector documents dynamic partition pruning and an optional requirement for partition filters. These are cross-engine context and do not mean Athena or Trino settings apply to Atlas Data Federation. (medium confidence) — [source](https://docs.aws.amazon.com/athena/latest/ug/ctas-partitioning-and-bucketing-what-is-partitioning.html) ## How it works - Data Federation adds partition attributes even when a stored document already has a field with the same name, creating duplicate field names that MongoDB does not support. Use distinct partition names or `omitAttributes`; a community user also reported that renaming a colliding field resolved their partition query issue. (high confidence) — [source](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md) - Give path partition attributes names that do not collide with fields inside the documents. Atlas warns that it adds path attributes even when a same-named document field exists, creating duplicate field names; a 2023 community report said renaming a colliding ObjectId path attribute restored query filtering. (medium confidence) — [source](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md) ## How-to and procedures - Avoid layouts that create many small files or excessive partitions. MongoDB says many small files compound per-file resource use and large files can require unnecessary downloads; AWS Athena separately warns that too many S3 partitions add file and partition-filtering overhead, while too few can increase scans. (high confidence) — [source](https://www.mongodb.com/docs/atlas/data-federation/admin/optimize-query-performance.md) ## Measurements and reference values - Selective partition matches can reduce Data Federation's processed bytes and charges, but partitioning does not guarantee savings: a query that reads the whole collection still processes the whole collection. MongoDB's example contrasts reading one 1 GB partition from a 10 GB file layout with processing all 10 GB when no partition can be skipped. (high confidence) — [source](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md) - MongoDB's billing page lists Data Federation processing at $5 per TB with a 10 MB minimum per query. (low confidence) (verified as of 2026-10-18) — [source](https://www.mongodb.com/docs/atlas/billing/data-federation/) ## Problems, failure modes and limitations - MongoDB recommends ordering S3 partition fields from the most frequently queried to the least frequently queried. A query that skips an earlier path field can use matching earlier partitions more efficiently than later ones; fields absent from the configured path cannot prune files. (high confidence) — [source](https://www.mongodb.com/docs/atlas/data-federation/admin/optimize-query-performance.md) ## Facts and statements - For customer-managed S3, a collection's `dataSources.path` can map S3 key segments to computed partition attributes, and predicates on those attributes can target matching prefixes and files. Independent OneUptime and Medium write-ups describe the same path-to-field and filter pattern, but provide no measured benchmark. (medium confidence) — [source](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md) - Atlas Data Federation maps configured S3 stores to virtual databases and collections. Its data-source path syntax can parse key or filename segments into typed partition attributes and add those attributes to query documents. (high confidence) — [source](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md) - Atlas charges separately for data processed and data returned or transferred. A filter on mapped S3 path attributes can target matching files and reduce processed bytes, but a broad query such as a blank $match still reads the full collection, so partitioning alone does not guarantee lower processing cost. (high confidence) — [source](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md)