Data Acquisition and Sampling

Data Acquisition and Sampling

This skill covers the third stage of the data analysis curriculum: getting data into a form the analysis can operate on, and constructing a sample that supports valid inference about the target population. The two activities are intertwined: the choice of source constrains what sampling design is possible, and the sampling plan determines which sources are acceptable.

A common mistake is to treat acquisition as a logistics problem — “just pull the data” — and discover only at the analysis stage that the population was wrong, the sample frame had coverage gaps, the schema drifted mid-pull, or the file format made the planned query infeasible. This stage owns the responsibility for catching these failures before they contaminate downstream work.

Sub-skill routing table

This hub absorbs 9 former standalone skills as on-demand reference files. When a task matches a row, Read the listed references/ file before answering — do not rely on this table alone for depth.

Sub-topic When to load Reference file
da-3-1-data-sources Data sources as a category within data collection & acquisition references/da-3-1-data-sources.md
da-3-1-1-primary-vs-secondary Primary vs secondary data sources — provenance, fit, trade-offs references/da-3-1-1-primary-vs-secondary.md
da-3-1-2-internal-vs-external The internal-vs-external dimension of data sources references/da-3-1-2-internal-vs-external.md
da-3-1-3-structured-semi-structured-unstructured Structured, semi-structured, unstructured data types references/da-3-1-3-structured-semi-structured-unstructured.md
da-3-2-collection-methods Overview/comparison of data collection methods references/da-3-2-collection-methods.md
da-3-2-5-web-scraping-crawling Web scraping/crawling — tooling, legality, ethics references/da-3-2-5-web-scraping-crawling.md
da-3-2-6-apis-data-feeds Collecting data via APIs and structured feeds references/da-3-2-6-apis-data-feeds.md
da-3-2-7-web-app-analytics-instrumentation Event instrumentation for web and mobile apps references/da-3-2-7-web-app-analytics-instrumentation.md
da-3-data-collection-acquisition Overview/orientation for Data Collection & Acquisition references/da-3-data-collection-acquisition.md

1. Data Source Taxonomy

Three orthogonal axes characterize any source.

1.1 Primary vs secondary

Primary data is collected directly to answer the current question (surveys you designed, interviews, sensor data, A/B exposure logs). Secondary data was collected by someone else for a different purpose and is reused (Census/BLS, commercial panels, academic archives, third-party API exports). Strong analyses combine both. The trade-off is purpose-fit (primary) versus speed/scale/cost (secondary).

1.2 Structured vs semi-structured vs unstructured

Vector embeddings and LLMs reduced the cost of operating on unstructured data, but the closer a source is to structured form, the cheaper and more deterministic the analysis. Schema-on-read (data lakes) defers structure to query time; schema-on-write (warehouses) enforces it at load time.

1.3 Internal vs external

Internal sources (production DBs, event logs, CRM, telemetry) are more reliable and granular but may not generalize. External sources (APIs, public datasets, scraped pages, panels) broaden the population but add coverage uncertainty, licensing risk, and schema drift.

2. API Ingestion

2.1 REST, GraphQL, gRPC

Common 2026 pattern: REST public, GraphQL BFF/frontend, gRPC internal. For acquisition you mostly meet REST and GraphQL.

2.2 Authentication

Treat refresh tokens as the most sensitive secret: encrypt at rest, rotate on suspicion, log every refresh.

2.3 Pagination

Persist the cursor after every page so a partial failure can resume.

2.4 Rate limiting

Fixed window, sliding window, token bucket, cost-based (GraphQL). Build clients with exponential backoff on 429/503, respecting Retry-After. Cap retries.

2.5 Webhooks

Inverse of polling. Always verify the signature header (HMAC-SHA256 over the raw body) before trusting the payload; treat unsigned webhooks as untrusted input.

3. Web Scraping

Acquisition without a contract. Use only when there is no API and the legal/ethical posture is sound.

3.1 Tooling

3.2 Legality

As of 2026 the hiQ v. LinkedIn line: scraping publicly accessible data generally does not violate the CFAA. But the full risk surface includes CFAA (bypassing auth/access controls), Terms of Service (civil claims), copyright (bulk reproduction), GDPR/CCPA (personal data, stricter in the EU), and EU database rights.

3.3 Ethics

Respect robots.txt; set a descriptive User-Agent with contact info; throttle (≤1 req/sec for small sites); cache aggressively; avoid PII without a lawful basis; never bypass authentication, paywalls, or rate-limiting controls.

4. Database Extraction

4.1 Bulk export

SELECT * into CSV/Parquet/Avro. Simple for cold historical data; anti-pattern for large operational tables. Use native utilities (mongoexport, pg_dump, mysqldump --single-transaction, bq extract, Redshift UNLOAD).

4.2 Incremental polling (JDBC/ODBC)

Connectors (Kafka Connect JDBC, Airbyte, Fivetran, Meltano) poll on a schedule, identifying changes by updated_at or auto-increment id. Gaps: hard deletes invisible, backdated updates missed, high-frequency polling stresses the source, schema changes break connectors.

4.3 Log-based CDC

Read the transaction log (MySQL binlog, PostgreSQL WAL, MongoDB oplog/change streams, SQL Server CDC). Debezium is the dominant open-source platform. Advantages: captures inserts/updates/deletes, every intermediate state, minimal source load, sub-second latency, stable per-row ordering. Typical pipeline: initial snapshot, then stream the log from the snapshot’s LSN/position; persist resume tokens/offsets for recovery.

5. Streaming Ingest

5.1 Kafka, Kinesis, Pub/Sub

5.2 Exactly-once semantics

At-most-once / at-least-once / exactly-once. Kafka: idempotent producers + transactions API. Kinesis: KCL checkpoints + idempotent downstream. Pub/Sub: regional exactly-once API. Practical guidance: make consumers idempotent, default to at-least-once, invoke exactly-once only when duplicates are costlier than the coordination.

5.3 Order, partitioning, back-pressure

Partition key sets parallelism and ordering (same key → same partition → in-order). Hot partitions are the primary failure mode. Handle back-pressure at the consumer: buffer (memory), drop (loss), or pause the source (propagation).

6. ETL vs ELT and the Modern Data Stack

6.1 ETL vs ELT

6.2 The modern data stack

6.3 Selection guidance

7. Surveys and Primary Collection

7.1 Sampling frame

The operational list of population members that can actually be reached — rarely identical to the target population. This gap creates coverage bias. Document the frame explicitly at design time; if it doesn’t match the population, no sample size or design can fix the resulting bias.

7.2 Response bias

Bias is systematic distortion; it does not shrink with sample size. Forms: nonresponse, acquiescence (yea-saying), social desirability, recall, order effects, mode effects, selection bias. Mitigations: track response rate, reverse-coded items, anonymous administration, randomize order, demographic benchmarking.

7.3 Practical design

Pilot with 10-20 respondents; use vertical scales for mobile; cap length (completion falls past 5-7 min); use attention checks sparingly; pre-register the analysis plan for high-stakes work.

8. Sampling Methodology

8.1 Probability sampling

Every unit has a known, non-zero selection probability — the only basis for valid frequentist inference.

8.2 Non-probability sampling

Selection probability unknown or zero; treat as exploratory unless you can model selection.

Modern hybrid: online panel + post-stratification weighting — weight non-probability panel responses to population marginals. Reduces but does not eliminate selection bias on outcome-correlated dimensions not in the weighting variables.

Cross-hub map — where every data-analytics topic lives

Hub Owns
da-1-foundations-theory Data Analysis Foundations & Theory
da-2-data-analysis-lifecycle Data Analysis Lifecycle & Process
da-3-data-acquisition-sampling Data Acquisition, Collection & Sampling
da-analytical-methods Data Analytical Methods (cleaning, EDA, modeling, ML, causal, time-series)
da-data-engineering-platform Data Engineering & Analytics Platform
da-applied-and-communication Applied Analytics, Visualization, Communication & Ethics