Data Acquisition and Sampling
researched 2026-05-30T13:08:12.300Z· 0 sources · 11 concepts · skill da-3-data-acquisition-sampling
This skill covers the third stage of the data analysis curriculum: getting data into a form the
Data Acquisition and Sampling
- This skill covers the third stage of the data analysis curriculum: getting data into a form the [source]
- analysis can operate on, and constructing a sample that supports valid inference about the target [source]
- population. The two activities are intertwined: the choice of source constrains what sampling design [source]
- is possible, and the sampling plan determines which sources are acceptable. [source]
- A common mistake is to treat acquisition as a logistics problem - "just pull the data" - and discover [source]
- only at the analysis stage that the population was wrong, the sample frame had coverage gaps, the [source]
- schema drifted mid-pull, or the file format made the planned query infeasible. This stage owns the [source]
- responsibility for catching these failures before they contaminate downstream work. [source]
Sub-skill routing table
- This hub absorbs 9 former standalone skills as on-demand reference files. When a task matches a row, Read the listed references/ file before answering - do not rely on this table alone for depth. [source]
1. Data Source Taxonomy
- Three orthogonal axes characterize any source. [source]
1.1 Primary vs secondary
- Primary data is collected directly to answer the current question (surveys you designed, interviews, sensor data, A/B exposure logs). Secondary data was collected by someone else for a different purpose and is reused (Census/BLS, commercial panels, academic archives, third-party API exports). Strong analyses combine both. The trade-off is purpose-fit (primary) versus speed/scale/cost (secondary). [source]
1.2 Structured vs semi-structured vs unstructured
- Structured: row/column tabular with fixed schema (relational DBs, warehouses, CSV/Parquet). [source]
- Semi-structured: hierarchical/self-describing (JSON, XML, logs, NoSQL docs, Avro/Protobuf). [source]
- Unstructured: free-form text, images, audio, video; requires feature extraction (OCR, ASR, embeddings) or a model interface. [source]
- Vector embeddings and LLMs reduced the cost of operating on unstructured data, but the closer a source is to structured form, the cheaper and more deterministic the analysis. Schema-on-read (data lakes) defers structure to query time; schema-on-write (warehouses) enforces it at load time. [source]
1.3 Internal vs external
- Internal sources (production DBs, event logs, CRM, telemetry) are more reliable and granular but may not generalize. External sources (APIs, public datasets, scraped pages, panels) broaden the population but add coverage uncertainty, licensing risk, and schema drift. [source]
2.1 REST, GraphQL, gRPC
- REST (HTTP+JSON): broadest compatibility, HTTP caching, OpenAPI contracts. Downside: over/under-fetching. [source]
- GraphQL: client asks for exactly the fields it needs; eliminates over/under-fetch. Downside: HTTP caching is harder, rate limiting via query-cost budgets, per-field authorization. [source]
- gRPC (HTTP/2 + Protobuf): code-generated clients, multiplexed streaming, 3-10x smaller payloads. Best for internal service-to-service traffic. [source]
- Common 2026 pattern: REST public, GraphQL BFF/frontend, gRPC internal. For acquisition you mostly meet REST and GraphQL. [source]
2.2 Authentication
- API key: shared secret in a header; identifies the app, not a user; TLS only; rotate. [source]
- OAuth 2.0: access + refresh tokens. Authorization Code with PKCE for user apps; Client Credentials for service-to-service. Validate scopes server-side. [source]
- JWT: signed bearer token; stateless verification; keep short-lived; verify the alg header (avoid alg: none). [source]
- mTLS / certificate auth: high-trust internal/financial/healthcare APIs. [source]
- Treat refresh tokens as the most sensitive secret: encrypt at rest, rotate on suspicion, log every refresh. [source]
2.3 Pagination
- Offset/limit and page number: simple but break under concurrent writes. [source]
- Cursor-based (opaque token): stable under writes; preferred for high-volume APIs. [source]
- Keyset/seek (sort key + tiebreaker): cheap on indexed columns. [source]
- Persist the cursor after every page so a partial failure can resume. [source]
2.4 Rate limiting
- Fixed window, sliding window, token bucket, cost-based (GraphQL). Build clients with exponential backoff on 429/503, respecting Retry-After. Cap retries. [source]
2.5 Webhooks
- Inverse of polling. Always verify the signature header (HMAC-SHA256 over the raw body) before trusting the payload; treat unsigned webhooks as untrusted input. [source]
3. Web Scraping
- Acquisition without a contract. Use only when there is no API and the legal/ethical posture is sound. [source]
3.1 Tooling
- requests + BeautifulSoup: static HTML, no JS. [source]
- Scrapy: full crawling framework (concurrency, throttling, retries, pipelines). [source]
- Playwright / Selenium: headless browsers for JS-rendered or auth-gated pages; slower. [source]
- TLS-impersonation tooling (curl_cffi): a signal the site does not want to be scraped. [source]
3.2 Legality
- As of 2026 the hiQ v. LinkedIn line: scraping publicly accessible data generally does not violate the CFAA. But the full risk surface includes CFAA (bypassing auth/access controls), Terms of Service (civil claims), copyright (bulk reproduction), GDPR/CCPA (personal data, stricter in the EU), and EU database rights. [source]
3.3 Ethics
- Respect robots.txt; set a descriptive User-Agent with contact info; throttle (≤1 req/sec for small sites); cache aggressively; avoid PII without a lawful basis; never bypass authentication, paywalls, or rate-limiting controls. [source]
4.1 Bulk export
- SELECT * into CSV/Parquet/Avro. Simple for cold historical data; anti-pattern for large operational tables. Use native utilities (mongoexport, pg_dump, mysqldump --single-transaction, bq extract, Redshift UNLOAD). [source]
4.2 Incremental polling (JDBC/ODBC)
- Connectors (Kafka Connect JDBC, Airbyte, Fivetran, Meltano) poll on a schedule, identifying changes by updated_at or auto-increment id. Gaps: hard deletes invisible, backdated updates missed, high-frequency polling stresses the source, schema changes break connectors. [source]
4.3 Log-based CDC
- Read the transaction log (MySQL binlog, PostgreSQL WAL, MongoDB oplog/change streams, SQL Server CDC). Debezium is the dominant open-source platform. Advantages: captures inserts/updates/deletes, every intermediate state, minimal source load, sub-second latency, stable per-row ordering. Typical pipeline: initial snapshot, then stream the log from the snapshot's LSN/position; persist resume tokens/offsets for recovery. [source]
5.1 Kafka, Kinesis, Pub/Sub
- Kafka (MSK, Confluent, Redpanda): de facto standard, open protocol, strongest ecosystem, highest throughput. Best for multi-cloud and complex stream processing. [source]
- Kinesis Data Streams: AWS-native, shard-based; 2026 trend favors MSK for new AWS deployments unless small/serverless. [source]
- Pub/Sub: GCP-native, serverless, regional exactly-once (2024). [source]
5.2 Exactly-once semantics
- At-most-once / at-least-once / exactly-once. Kafka: idempotent producers + transactions API. Kinesis: KCL checkpoints + idempotent downstream. Pub/Sub: regional exactly-once API. Practical guidance: make consumers idempotent, default to at-least-once, invoke exactly-once only when duplicates are costlier than the coordination. [source]
5.3 Order, partitioning, back-pressure
- Partition key sets parallelism and ordering (same key → same partition → in-order). Hot partitions are the primary failure mode. Handle back-pressure at the consumer: buffer (memory), drop (loss), or pause the source (propagation). [source]
6.1 ETL vs ELT
6.2 The modern data stack
- Ingest/EL: Fivetran, Airbyte, Meltano (Singer), Stitch. [source]
- Transform/T: dbt (needs an orchestrator; doesn't extract/load). [source]
- Orchestration: Airflow, Dagster, Prefect, Mage. [source]
- Warehouse: Snowflake, BigQuery, Databricks, Redshift, ClickHouse, MotherDuck. [source]
- Reverse ETL: Hightouch, Census. [source]
- Catalog/governance: DataHub, OpenMetadata, Atlan, Collibra. [source]
- Observability: Monte Carlo, Bigeye, Lightup, Soda. [source]
6.3 Selection guidance
7.1 Sampling frame
- The operational list of population members that can actually be reached - rarely identical to the target population. This gap creates coverage bias. Document the frame explicitly at design time; if it doesn't match the population, no sample size or design can fix the resulting bias. [source]
7.2 Response bias
- Bias is systematic distortion; it does not shrink with sample size. Forms: nonresponse, acquiescence (yea-saying), social desirability, recall, order effects, mode effects, selection bias. Mitigations: track response rate, reverse-coded items, anonymous administration, randomize order, demographic benchmarking. [source]
7.3 Practical design
- Pilot with 10-20 respondents; use vertical scales for mobile; cap length (completion falls past 5-7 min); use attention checks sparingly; pre-register the analysis plan for high-stakes work. [source]
8.1 Probability sampling
- Every unit has a known, non-zero selection probability - the only basis for valid frequentist inference. [source]
- SRS: equal probability 1/N; the reference design. [source]
- Stratified: sample within mutually exclusive strata. Proportional allocation (size-proportional) vs Neyman optimal allocation (size × stdev; minimizes variance for fixed n). [source]
- Cluster: randomly select clusters, then sample within. Loses precision vs SRS (design effect; effective sample size n / DEFF). [source]
- Systematic: every kth element after a random start; biased if the frame has periodicity matching k. [source]
8.2 Non-probability sampling
- Selection probability unknown or zero; treat as exploratory unless you can model selection. [source]
- Convenience: whoever is at hand. [source]
- Quota: hit target subgroup counts; biased on non-quota dimensions. [source]
- Snowball: respondents refer others; good for hidden populations. [source]
- Purposive/judgment: researcher selects informative units; fine for case studies, never for population estimates. [source]
- Modern hybrid: online panel + post-stratification weighting - weight non-probability panel responses to population marginals. Reduces but does not eliminate selection bias on outcome-correlated dimensions not in the weighting variables. [source]
- <!-- cross-hub-map --> [source]
Children
- Data Sources Taxonomy (frontier)
- APIs and Pagination (frontier)
- Web Scraping (frontier)
- Database Extraction and CDC (frontier)
- Streaming Ingest (frontier)
- ETL vs ELT (frontier)
- Surveys and Primary Collection (frontier)
- Sampling Methodology (frontier)
- Sample Size Determination (frontier)
- Data Contracts (frontier)
- File Formats (frontier)
Frontier under this node: APIs and Pagination, Data Contracts, Data Sources Taxonomy, Database Extraction and CDC, ETL vs ELT, File Formats, Sample Size Determination, Sampling Methodology, Streaming Ingest, Surveys and Primary Collection, Web Scraping