Data Analysis Lifecycle

researched 2026-05-30T12:23:03.306Z· 0 sources · 2 concepts · skill da-2-data-analysis-lifecycle

Taxonomy context: Data Analysis > Data Analysis Lifecycle (Process)

Data Analysis Lifecycle (Process)

Taxonomy context: Data Analysis > Data Analysis Lifecycle (Process) [source]
The Data Analysis Lifecycle is the structured, iterative process that carries a project from an [source]
initial business question through data acquisition, preparation, analysis, and interpretation, to [source]
communicated and operationalized insight. No single canonical standard exists; instead several [source]
well-adopted frameworks describe roughly the same phases with different emphasis and vocabulary. [source]
Understanding the lifecycle helps analysts know which phase they are in, what must be true before [source]
advancing, and when to loop back. [source]

Sub-skill routing table

This hub consolidates 13 lifecycle sub-skills as on-demand reference files. When a task matches a [source]
row, Read the listed references/<name>.md before answering - do not rely on this table alone [source]

1. Why a lifecycle matters

Raw data does not automatically answer questions. Each phase in the lifecycle performs a distinct [source]
Reduces ambiguity - turning vague questions into measurable objectives. [source]
Ensures fitness of data - catching quality problems before they corrupt findings. [source]
Separates concerns - keeping exploratory work from confirmatory work, and analysis from [source]
Creates checkpoints - natural gates where the team can confirm alignment with stakeholders [source]
before investing further. [source]
Without an explicit lifecycle, projects commonly suffer from scope creep, premature modeling on [source]
dirty data, and findings that cannot be reproduced or deployed [Source 1, Source 2]. [source]

2.1 CRISP-DM (Cross-Industry Standard Process for Data Mining)

Developed in the late 1990s by Daimler-Chrysler, SPSS, and NCR. Still the most widely cited open [source]
standard for data mining and analytics projects [Source 3]. [source]
Six phases arranged in a cycle (the outer ring can restart after deployment): [source]
Arrows in the CRISP-DM diagram flow in both directions: unsatisfactory evaluation sends the team [source]
back to modeling or data preparation; business understanding may be revised when data understanding [source]
reveals the original question is unmeasurable [Source 3]. [source]

2.2 EMC / Big Data Analytics Lifecycle

Popularized by EMC's Data Science and Big Data Analytics book and Wiley's companion edition. [source]
Six phases with a heavier emphasis on analytic sandboxes and operationalization [Source 4]: [source]
Discovery [source]
Data Preparation (ELT/ETL into sandbox) [source]
Model Planning [source]
Model Building [source]
Communicate Results [source]
Operationalize [source]
Distinct from CRISP-DM in that it explicitly names the sandbox as a prerequisite for phase 2, and [source]
distinguishes "Model Planning" (choosing techniques) from "Model Building" (executing them). [source]

2.3 OSEMN

A minimalist five-step mnemonic popular in academic data science courses: [source]
Obtain [source]
Scrub [source]
Explore [source]
Model [source]
iNterpret [source]
Strengths: simple and memorable. Weaknesses: omits business framing (starts at "Obtain"), ignores [source]
deployment, and treats the process as linear [Source 5]. [source]

2.4 TDSP (Team Data Science Process)

Published by Microsoft in 2017. Closest to CRISP-DM but adds explicit team roles, deliverable [source]
templates, and agile sprint cadence. Five stages: Business Understanding → Data Acquisition and [source]
Understanding → Modeling → Deployment → Customer Acceptance [Source 5]. [source]

3. Canonical phase descriptions

The following synthesizes the phases common across frameworks into a single reference. The [source]
references/ files listed in the Sub-skill routing table cover each phase and framework in detail. [source]

Phase 1 — Problem Definition / Business Understanding

Input: stakeholder intent, existing domain knowledge, prior analyses. [source]
Output: a written problem statement, success criteria (KPIs or evaluation metrics), and an [source]
initial set of hypotheses. [source]
The team works with business owners to translate a vague goal ("improve customer retention") into [source]
a concrete, measurable objective ("predict 30-day churn with precision ≥ 0.75 at recall ≥ 0.60"). [source]
Resources, timeline, and risks are assessed here. [source]
Why it matters: an ill-defined question cannot be answered with data. Changing the question [source]
halfway through wastes preparation and modeling effort. [source]
Pitfall: treating this phase as a formality. Teams that skip or rush it often discover midway [source]
through modeling that the available data cannot actually answer the question they care about [source]

Phase 2 — Data Acquisition and Understanding

Input: problem statement, knowledge of available data sources. [source]
Output: a data inventory, quality assessment report, initial summary statistics, and a [source]
decision on whether the data is sufficient to proceed. [source]
The team collects initial data, examines its structure and provenance, documents quality issues [source]
(nulls, duplicates, encoding errors, date range gaps), and explores distributions and [source]
inter-variable relationships. [source]
Pitfall: trusting that data labeled "clean" actually is clean. Source systems commonly have [source]
undocumented conventions (e.g., sentinel values like -9999 for missing) that only domain knowledge [source]
or careful profiling reveals [Source 1, Source 2]. [source]

Phase 3 — Data Preparation

Input: raw or semi-structured data, quality assessment. [source]
Output: an analysis-ready dataset (feature matrix + target variable, or cleaned tabular data [source]
for descriptive work). [source]
This phase typically consumes 60–80% of total project time. It includes: [source]
Cleaning: removing or imputing nulls; correcting format inconsistencies; deduplication. [source]
Transformation: normalization, encoding categorical variables, date parsing, log transforms. [source]
Integration: joining tables across systems, resolving entity mismatches. [source]
Feature engineering: constructing derived columns that encode domain knowledge. [source]
An analytic sandbox - a compute environment with sufficient CPU, RAM, and storage to hold [source]
working copies of the data - is often set up at the start of this phase [Source 4]. [source]
Pitfall - data leakage: features that encode information from the future (relative to the [source]
prediction point) will inflate model performance metrics while producing a model that fails in [source]
production. Any transformation that aggregates across the full dataset (e.g., computing a [source]
z-score mean on both training and test rows) must be fit on training data only and applied to [source]
test data [Source 6]. [source]
Pitfall - aggressive outlier removal: deleting extreme values simplifies modeling but can [source]
remove the most informative signals, especially in anomaly detection or fraud contexts [Source 6]. [source]

Phase 4 — Analysis / Modeling

Input: analysis-ready dataset, modeling plan (technique selection, validation strategy). [source]
Output: trained model(s) or analytical findings with performance metrics. [source]
For descriptive and exploratory analysis, this phase produces summary statistics, [source]
visualizations, and identified patterns. For predictive analysis, it produces one or more [source]
fitted models with cross-validated performance estimates. [source]
Model planning (choosing the technique family and validation design) is logically distinct from [source]
model building (running training and tuning loops). Conflating them leads to technique choices [source]
driven by familiarity rather than problem fit [Source 4]. [source]
Pitfall - overfitting through hyperparameter tuning: testing many parameter combinations [source]
without a held-out test set causes the model to fit noise in the validation set, producing [source]
strong validation scores that do not transfer to new data [Source 6]. [source]

Phase 5 — Evaluation

Input: model or analysis output, success criteria from Phase 1. [source]
Output: judgment of whether findings meet the original objective; recommendation to proceed [source]
The team compares model performance against the thresholds established in Phase 1, assesses [source]
whether the findings have practical as well as statistical significance, and checks that the [source]
model's behavior makes sense to domain experts (a sanity check that catches leakage and [source]
labeling errors not surfaced by metrics alone). [source]
If evaluation fails, the team loops back - usually to Phase 3 (more features, different [source]
cleaning) or Phase 2 (additional data sources). [source]
Pitfall - confusing statistical and practical significance: a result can be statistically [source]
significant yet too small to matter operationally. A 0.1% improvement in click-through rate may [source]
not justify the cost of implementation [Source 6]. [source]

Phase 6 — Communication of Results

Input: evaluated findings, audience knowledge of domain. [source]
Output: narrative report, dashboard, or presentation that conveys key findings and [source]
recommended actions to decision-makers. [source]
Effective communication requires translating technical outputs into business terms. The team [source]
quantifies business value (revenue impact, cost savings, risk reduction), documents key [source]
assumptions, acknowledges limitations, and prepares supporting materials (code, data [source]
dictionaries, reproducibility documentation). [source]
Pitfall - model explainability missteps: presenting SHAP plots or feature importances [source]
without business context confuses rather than informs stakeholders. Explanation tools are most [source]
useful when tied to a specific decision the audience must make [Source 6]. [source]

Phase 7 — Operationalization / Deployment

Input: approved findings or model, deployment environment specifications. [source]
Output: running system (scheduled report, API endpoint, embedded model), monitoring plan. [source]
The team deploys the model or analysis process so that stakeholders can regularly access results. [source]
Pilot deployments in a controlled environment precede full rollout. Monitoring tracks whether [source]
model performance degrades as data distributions shift over time. [source]
Pitfall - ignoring concept drift: a model trained on historical data may fail silently as [source]
real-world behavior changes. Without a monitoring plan and retraining schedule, model staleness [source]
goes undetected [Source 6]. [source]

4. The iterative nature of the lifecycle

All frameworks represent the lifecycle as cyclic or iterative, not strictly linear. Common [source]
Evaluation → Data Preparation: model fails to meet threshold; team engineers additional [source]
features or acquires more data. [source]
Modeling → Business Understanding: the most predictive variables are ones the business [source]
cannot act on; problem definition must be revised. [source]
Communication → Problem Definition: stakeholders raise a follow-up question not covered [source]
by the original scope; a new project iteration begins. [source]
Operationalization → Data Understanding: production data differs from training data in [source]
distribution; team must re-examine source systems. [source]
Treating the lifecycle as strictly sequential is a recognized anti-pattern. Teams that refuse to [source]
revisit earlier phases when evidence demands it produce analyses that are technically complete but [source]
practically useless [Source 1, Source 5]. [source]

5. Cross-cutting concerns

These concerns apply across all phases rather than belonging to a single one: [source]

Documentation and provenance

Every transformation applied to data should be recorded so that findings can be reproduced and [source]
audited. Missing metadata about data origins is one of the top causes of unreproducible analyses [source]

Stakeholder alignment

Checkpoints between phases - presenting phase outputs to stakeholders before proceeding - catch [source]
misalignment early. The cost of rework grows with each phase completed before a mismatch is [source]

Team roles

In TDSP, roles are explicitly assigned: project lead, data scientist, data engineer, solution [source]
architect, and business analyst each own specific deliverables. In smaller teams one person [source]
covers multiple roles, but the responsibilities remain distinct [Source 5]. [source]

Governance and ethics

Data collection and use must comply with applicable regulations (GDPR, HIPAA, etc.) and internal [source]
data governance policies. These checks are easiest to apply at phase transitions, not after [source]

6. Practical worked example

Scenario: a retail company wants to reduce stockouts. [source]
At evaluation, the team discovers that weather features add noise rather than signal; they loop [source]
back to Phase 3, drop those columns, and re-evaluate - a normal iteration, not a failure. [source]

Sources

"Understanding the data analytics lifecycle from end-to-end," Quadratic HQ. [source]
https://www.quadratichq.com/blog/understanding-the-data-analytics-lifecycle-from-end-to-end [source]
"Data Analytics Lifecycle: Phases And Importance," TechCanvass Business Analyst Blog. [source]
https://businessanalyst.techcanvass.com/data-analytics-lifecycle-phases/ [source]
"CRISP-DM Methodology: Industry Standard for Data Mining Processes," Medium / Learning Data. [source]
https://medium.com/learning-data/crisp-dm-methodology-industry-standard-for-data-mining-processes-f896b33dc5ce [source]
"6 Phases of Data Analytics Lifecycle Every Data Analyst Should Know," DEV Community / BPB Online. [source]
https://dev.to/bpb_online/6-phases-of-data-analytics-lifecycle-every-data-analyst-should-know-1k [source]
"Data Science Life Cycle: CRISP-DM and OSEMN frameworks," Data Rundown. [source]
https://datarundown.com/data-science-life-cycle/ [source]
"Common Pitfalls to Avoid When Analyzing and Modeling Data," freeCodeCamp. [source]
https://www.freecodecamp.org/news/common-pitfalls-to-avoid-when-analyzing-and-modeling-data/ [source]
[source]

Cross-hub map — where every data-analytics topic lives

This family is split across these hubs. If a task's deep material is not in this hub's Sub-skill [source]
routing table, it is a reference file under a sibling hub below - **activate that hub or Read its [source]
references/<name>.md directly**. Every former standalone skill in this family is now a reference under one [source]
of these hubs (nothing was deleted). [source]

Children

← the whole tree · 3D view· how to read this page