Data Analysis Foundations and Theory

Data Analysis: Foundations & Theory

Conceptual grounding for a data analysis effort. This skill answers the before-you-pick-a-tool questions: what is data analysis, how does it relate to neighboring fields, what kind of analysis is called for, what can the data support given how it was measured, and what assumptions ride underneath. It does not execute techniques — it scopes and frames them.

Sub-skill routing table

This hub consolidates 37 foundations sub-skills as on-demand references — match the task to the table and Read the listed references/<name>.md before answering deep questions. The overview below is enough for framing and scoping; load the reference when a question needs depth.

1. What data analysis is (and its scope)

Data analysis is the systematic process of inspecting, cleaning, transforming, and interpreting data to extract useful information, support conclusions, and aid decision-making. In practice it is bounded and goal-directed.

Distinguish four neighboring terms — they overlap but are not synonyms:

• Data analysis — the act of evaluating data to answer a defined question. The process verb.
• Data analytics — the broader practice/field built around analyzing data; typically about the past and present, with more elementary statistics.
• Data science — a wider, multidisciplinary field that includes analytics but reaches into ML, forecasting, and large-scale data engineering; more oriented toward predictive models.
• Statistics — the mathematical discipline of collecting, describing, and drawing inferences from data under uncertainty. Data analysis uses statistics as a toolkit.

Rule of thumb: data analysis is the activity; analytics is the field around it; data science extends it toward modeling and engineering; statistics supplies the inferential mathematics. State which definition you are using.

Analysis vs. synthesis

Analysis breaks a whole into parts; synthesis recombines parts into a new integrated whole or recommendation. Name which mode you are in to avoid presenting raw decomposition as a conclusion.

Quantitative vs. qualitative

• Quantitative analysis operates on numeric measurements (interval and ratio scales).
• Qualitative analysis operates on categorical/textual data (nominal and ordinal scales). The split is about method too, not just data type. Many efforts are mixed-method.

2. The four families of analysis

The four form a maturity progression but are not strictly sequential per project. “Diagnostic” maps loosely onto exploratory work, but don’t conflate the marketing taxonomy with Tukey’s exploratory/confirmatory split.

3. The analysis lifecycle / process

The de facto reference is CRISP-DM (six phases you can revisit):

1. Business Understanding — define the question and success criteria.
2. Data Understanding — collect, describe, explore, verify quality.
3. Data Preparation — select, clean, construct, integrate, format.
4. Modeling — choose technique, build, assess.
5. Evaluation — check against business goal; review process.
6. Deployment — deliver, monitor, report.

The phases are iterative, not a one-way pipeline. A lighter generic framing — define → collect → clean → analyze → interpret → communicate — works for non-mining work.

4. Exploratory vs. confirmatory analysis

• Exploratory Data Analysis (EDA) — Tukey’s approach for summarizing a dataset’s main characteristics, often with graphics, to generate hypotheses and check assumptions. Techniques: box plots, stem-and-leaf, histograms, scatter plots.
• Confirmatory Data Analysis (CDA) — classical hypothesis testing: pick a model before examining the data, then assess inference precision.

Researchers need both. Critical pitfall: running exploratory and confirmatory analysis on the same data introduces systematic bias (double-dipping). Reserve a holdout or fresh sample for confirmation.

5. Measurement theory and levels of measurement

Stevens’ four levels (1946):

Nominal and ordinal are categorical/qualitative; interval and ratio are continuous/quantitative.

Common pitfalls: computing a mean of ordinal codes (median is safer); treating an arbitrary numeric label as quantitative; forgetting interval scales lack a true zero (ratios are meaningless). Know the controversy: Velleman & Wilkinson (1993) and Luce (1997) contested Stevens’ typology. Treat the level of measurement as a useful first filter, not an iron law.

6. The role of theory and assumptions

Data does not interpret itself. Every analysis rides on assumptions: representative sample, measurements meaning what labels claim, model preconditions holding. Two practices:

• State assumptions explicitly and tie each to the analysis family and measurement level.
• Match method to question and to data. Misalignment — a prescriptive recommendation on descriptive data, or a mean on ordinal categories — is the most common foundational error.

Quick decision checklist

1. Term check — analysis, analytics, data science, or statistics? State the definition.
2. Family — descriptive, diagnostic, predictive, or prescriptive?
3. Mode — exploratory or confirmatory? Don’t mix on the same data.
4. Lifecycle — which CRISP-DM phase; what’s next?
5. Measurement — what level is each variable; which statistics are licensed?
6. Assumptions — what must be true; have I checked?

Cross-hub map — where every data-analytics topic lives