Text Analytics and NLP for Analysts

Parent: data analysis · researched 2026-05-30T22:45:25.815Z· 28 sources · 11 concepts · skill da-36-text-analytics-nlp

Applied NLP for turning unstructured text into measurable signal. The audience is an analyst, not an ML engineer: the goal is defensible insight from reviews, support tickets, survey open-ends, call t

Text Analytics & NLP for Analysts

Applied NLP for turning unstructured text into measurable signal. The audience is an analyst, not an ML engineer: the goal is defensible insight from reviews, support tickets, survey open-ends, call transcripts, social posts, and documents - not training novel models. Prefer the simplest method that answers the question; escalate to embeddings/transformers/LLMs only when classical methods fall short. [source]

Decision guide (start here)

Rule of thumb: establish a cheap, interpretable baseline (TF-IDF + linear model, or VADER) before any transformer or LLM. Report the baseline number even when you ship the fancier model - it is your sanity check and your cost/latency benchmark. [source]

1. Preprocessing, tokenization, normalization

The decisions here silently determine every downstream result. [source]
Tokenization: splitting into tokens (words, subwords, sentences). spaCy and NLTK do linguistic word/sentence tokenization; transformers use subword tokenizers (WordPiece/BPE/SentencePiece) - never lowercase or strip punctuation before a transformer tokenizer, it expects raw text. [source]
Normalization: lowercasing, Unicode normalization (NFC/NFKC), accent folding, contraction expansion, whitespace cleanup. [source]
Stopword removal & stemming/lemmatization: helps bag-of-words / topic models; harmful for embeddings and transformers, which need full context. Lemmatization (spaCy) is preferred over crude stemming (Porter/Snowball) when you keep tokens human-readable. [source]
What to keep: for sentiment, keep negations, emojis, intensifiers, ALL-CAPS - VADER and good models use them as signal. [source]
Pitfall: applying the bag-of-words preprocessing recipe (lowercase, de-punct, stem, strip stopwords) before embedding/transformer steps destroys signal. [source]

2. Bag-of-words, TF-IDF, n-grams

BoW / Count: document = vector of token counts. Simple, sparse, ignores order. [source]
TF-IDF: down-weights terms common across the corpus, up-weights distinctive terms. TfidfVectorizer in scikit-learn is the analyst's workhorse for distinctive-term and baseline-classifier work. [source]
n-grams: capture short phrases ("not good", "customer service") that unigrams miss. Set ngram_range=(1,2) (sometimes (1,3)); watch dimensionality blow-up - use min_df/max_df and max_features. [source]
Still relevant in 2026: cheap, transparent, fast, and a strong classification baseline. Use it before anything heavier. [source]

3. Topic modeling: LDA, NMF, BERTopic

LDA (Latent Dirichlet Allocation, Gensim/scikit-learn): probabilistic, each doc = mixture of topics; needs num_topics chosen up front; works best on longer, cleaned text; tune with coherence (c_v / u_mass). [source]
NMF (scikit-learn, on TF-IDF): deterministic-ish matrix factorization; often crisper topics than LDA on short text; also needs k. [source]
BERTopic (MaartenGr): modular pipeline - embeddings (sentence-transformers) → UMAP (dim reduction) → HDBSCAN (density clustering, auto-detects topic count + outliers) → c-TF-IDF (per-cluster distinctive terms) → optional representation tuning (KeyBERT/MMR/LLM labels). Each stage is swappable. Default choice for modern, especially short/noisy, text because it does not force a topic count and handles outliers. Use reduce_outliers, nr_topics to merge, and pass a fixed random_state to UMAP for reproducibility. [source]
Choosing: short/messy/lots of docs → BERTopic; small clean corpus or need probabilistic doc-topic mixtures → LDA/NMF; need speed/transparency → NMF on TF-IDF. [source]

4. Sentiment: lexicon (VADER) vs transformer

VADER (Hutto & Gilbert, 2014): lexicon + rule-based, tuned for social media; handles negation, intensifiers, punctuation, emoji, caps. Returns compound in [-1, 1] (thresholds ≈ ±0.05). No training, instant, fully transparent, free - ideal first pass on tweets/reviews/short informal text. [source]
Transformer (e.g. fine-tuned BERT/RoBERTa via Hugging Face pipeline("sentiment-analysis")): context-aware, handles sarcasm/domain nuance better, but heavier, slower, and can carry positivity/domain bias. Studies consistently rank transformer/Flair > BERT > VADER on correlation with human ratings, with VADER competitive on short informal text and far cheaper. [source]
Aspect-based sentiment (ABSA): sentiment per aspect ("battery good, screen bad") - use when one polarity per doc is too coarse. [source]
Analyst guidance: VADER for fast directional reads and big volumes; transformer when domain/sarcasm matters and you can validate on a labeled sample. Always sanity-check both against a hand-labeled set - domain mismatch (finance, clinical) breaks generic models. [source]

5. Named-entity recognition (NER)

spaCy: statistical transition-based NER labeling non-overlapping spans (PERSON, ORG, GPE, DATE, PRODUCT, MONEY, …). en_core_web_sm/md/lg for speed, en_core_web_trf for accuracy. Add an EntityRuler for deterministic patterns (SKUs, ticket IDs) before/after the statistical NER. [source]
LLM / spacy-llm: zero/few-shot extraction of custom entity types without training data; good for niche schemas, but validate and watch for hallucinated spans. [source]
Analyst uses: aggregate mentions of competitors/products/locations, redact PII, link entities for rollups. [source]

6. Text classification

Baseline: TF-IDF + linear model (LogisticRegression / LinearSVC) or Naive Bayes. Fast, interpretable, hard to beat on small/medium labeled data. [source]
spaCy textcat (textcat mutually-exclusive, textcat_multilabel for overlapping labels) for an integrated trainable pipeline. [source]
Transformers (Hugging Face): fine-tune or use zero-shot (facebook/bart-large-mnli) when labels are scarce or semantics are subtle. [source]
LLM zero/few-shot: fastest path to a working classifier with no training data; pin the label set, give examples, and measure against a gold sample. [source]
Mind class imbalance (use class weights, stratified splits, macro-F1) and label leakage. [source]

7. Keyword & keyphrase extraction

RAKE: fast, unsupervised, co-occurrence based; great for speed, weak on stopword handling and precision. [source]
YAKE: statistical, single-document, language-agnostic, no corpus needed; can emit near-duplicates. [source]
KeyBERT (MaartenGr): embeds doc and candidate phrases (sentence-transformers) and ranks by cosine similarity; most accurate/contextual; use MMR / diversity to reduce redundancy. Slower (needs a model). [source]
Choose: RAKE/YAKE for speed and no dependencies; KeyBERT for quality and contextual relevance. Combine with c-TF-IDF for per-cluster keywords inside BERTopic. [source]

8. Embeddings & semantic clustering

word2vec / GloVe / fastText (Gensim): static word vectors; useful for analogy, vocabulary exploration, lightweight similarity; no context sensitivity. [source]
sentence-transformers (SBERT): contextual sentence/paragraph embeddings - all-MiniLM-L6-v2 (fast, 384-d, default) vs all-mpnet-base-v2 (best quality, slower); 2024+ multilingual & matryoshka/gte/bge options. Compute with model.encode(); compare with model.similarity() (cosine). [source]
Semantic clustering: embed → reduce (UMAP/PCA) → cluster (HDBSCAN for variable density + outliers, KMeans when you want fixed k). This is the engine under BERTopic and the analyst's go-to for "group these open-ends by meaning." [source]
Document similarity / dedup / near-search: cosine over embeddings; paraphrase mining for dedup; add a cross-encoder reranker when top-k precision matters. [source]

9. LLM-assisted qualitative coding & extraction

Deductive coding (codebook exists): prompt the LLM with the codebook + definitions + few-shot examples; LLMs give a systematic, reliable platform for code identification at scale - but human validation is mandatory; measure agreement (Cohen's/Krippendorff's) against human coders on a sample. [source]
Inductive/exploratory: embed + cluster first, then have the LLM label/summarize clusters (reflexive thematic analysis); multi-agent (coder/aggregator/reviewer) roles can improve rigor. [source]
Structured extraction: constrain output to JSON schema; extract fields, entities, ratings from free text reliably. [source]
Risks: hallucination, prompt-injection from the text itself (treat corpus content as untrusted data, not instructions), drift across runs (pin model + temperature=0), cost/latency at corpus scale. Performance varies by construct - validate per code. [source]

10. Evaluation

Classification/sentiment: accuracy, precision/recall, macro-F1 (for imbalance), confusion matrix; always hold out a labeled test set. [source]
Topic models: coherence (c_v, NPMI), topic diversity, and human inspection of top terms/exemplars - coherence alone is not enough. [source]
Clustering: silhouette, but mostly qualitative inspection of exemplars; report % outliers (HDBSCAN -1). [source]
Retrieval/similarity: precision@k, recall@k, MRR, nDCG. [source]
Coding/extraction: inter-rater agreement vs human gold (Cohen's κ, Krippendorff's α). [source]
Always: a labeled sample (even 100–300 items) is the cheapest insurance against shipping a wrong conclusion. [source]

Tools / frameworks

spaCy - production NLP: tokenization, lemmatization, POS, NER, textcat, EntityRuler, transformer pipelines, spacy-llm. Analyst default for linguistic preprocessing + entities. [source]
NLTK - teaching/classic toolkit: tokenizers, stopwords, stemmers, and VADER (nltk.sentiment.vader). [source]
scikit-learn - TfidfVectorizer/CountVectorizer, LDA, NMF, LogisticRegression/SVM, metrics, pipelines. The classical-NLP backbone. [source]
Gensim - LDA, word2vec/fastText, phrase detection, similarity indexes; good for large streaming corpora. [source]
Hugging Face Transformers / Hub - fine-tuned/zero-shot sentiment, classification, NER, embeddings; pipeline() API. [source]
sentence-transformers (SBERT) - embeddings for similarity, clustering, semantic search, paraphrase mining; cross-encoder rerankers. [source]
BERTopic - modular transformer topic modeling (embeddings → UMAP → HDBSCAN → c-TF-IDF → representation). [source]
KeyBERT / YAKE / RAKE - keyphrase extraction across the speed↔accuracy spectrum. [source]

Methodology (analyst workflow)

Frame the question in measurable terms ("what share of tickets mention billing AND are negative?"). [source]
Profile the corpus: length distribution, language(s), duplicates, source noise (HTML, emoji, boilerplate). [source]
Preprocess deliberately - match the recipe to the method (heavy for BoW/LDA, light for embeddings/transformers). [source]
Baseline first: TF-IDF distinctive terms, VADER, or TF-IDF + linear classifier. Record the number. [source]
Escalate only if needed: embeddings/BERTopic/transformer/LLM, justified by a baseline gap. [source]
Validate on a labeled sample before trusting aggregates. [source]
Aggregate and visualize with uncertainty (counts, %, CIs); never report a single sentiment number without volume + a sample of exemplars. [source]
Document preprocessing, model versions, thresholds, and seeds for reproducibility. [source]

Practical patterns

Distinctive-terms report: TF-IDF with ngram_range=(1,2), top terms per segment - cheap, persuasive, interpretable. [source]
Open-end theming: SBERT all-MiniLM-L6-v2 → BERTopic with fixed UMAP random_state → label topics with KeyBERT/LLM → reduce outliers. [source]
Directional sentiment at scale: VADER pass for volume; transformer on a sampled/uncertain subset; reconcile. [source]
PII scrub / entity rollup: spaCy NER + EntityRuler; aggregate ORG/PRODUCT mentions. [source]
Codebook coding: LLM deductive pass with few-shot codebook → human reviews a random 10–20% → report κ. [source]
Dedup near-identical feedback: SBERT embeddings + cosine threshold or paraphrase mining. [source]

Anti-patterns

Reporting one global sentiment score with no volume, no time trend, no exemplars. [source]
Stemming/stopword-stripping before embeddings or transformer tokenizers (destroys signal). [source]
Picking LDA num_topics by gut and never checking coherence or exemplars. [source]
Trusting LLM/transformer output on a new domain without a labeled validation sample. [source]
Treating corpus text as trusted instructions to the LLM (prompt-injection exposure). [source]
Using accuracy on imbalanced labels instead of macro-F1. [source]
Jumping to transformers/LLMs before a TF-IDF/VADER baseline (cost, latency, and no benchmark). [source]
Non-reproducible runs: unpinned model versions, no seeds, non-zero temperature for extraction. [source]

Troubleshooting

Topics are garbage / one giant topic → too few/noisy docs, bad preprocessing, or HDBSCAN merging; tune min_topic_size, embeddings, or use NMF; check % outliers. [source]
Everything looks "neutral" in VADER → wrong domain (formal/clinical/financial), translation needed, or compound thresholds wrong; try a domain transformer. [source]
Embeddings cluster by length/language not meaning → mixed languages or huge length variance; segment by language, chunk long docs. [source]
KeyBERT returns near-duplicates → enable MMR / raise diversity. [source]
Classifier great in CV, bad live → leakage, distribution shift, or imbalance; re-check splits and macro-F1. [source]
LLM coding disagrees with humans → tighten codebook definitions, add few-shot exemplars, lower temperature, measure κ per code. [source]
Slow at corpus scale → batch encode(), use MiniLM not mpnet, cache embeddings, sample for exploration. [source]

References

BERTopic docs & algorithm - modular embeddings/UMAP/HDBSCAN/c-TF-IDF pipeline (MaartenGr, 2022–2025): https://maartengr.github.io/BERTopic/ , https://maartengr.github.io/BERTopic/algorithm/algorithm.html , https://github.com/MaartenGr/BERTopic [source]
Sentence-Transformers (SBERT) docs - semantic similarity, pretrained models, computing embeddings (2024–2025): https://www.sbert.net/docs/quickstart.html , https://www.sbert.net/docs/sentence_transformer/pretrained_models.html , https://www.sbert.net/docs/sentence_transformer/usage/semantic_textual_similarity.html [source]
VADER: Hutto & Gilbert, "A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text," ICWSM 2014: https://ojs.aaai.org/index.php/ICWSM/article/view/14550 ; repo: https://github.com/cjhutto/vaderSentiment ; docs: https://vadersentiment.readthedocs.io/ [source]
Transformer vs lexicon sentiment comparisons (2024–2025): https://link.springer.com/article/10.1007/s10115-024-02214-3 ; healthcare VADER/BERT/Flair: https://pmc.ncbi.nlm.nih.gov/articles/PMC12382424/ [source]
spaCy docs - NER, textcat, training pipelines, LLM integration (v3, 2024–2025): https://spacy.io/usage/spacy-101 , https://spacy.io/api/entityrecognizer , https://spacy.io/usage/training , https://spacy.io/usage/large-language-models , https://github.com/explosion/spacy-llm [source]
KeyBERT (MaartenGr) + keyword-extraction benchmarks (RAKE/YAKE/KeyBERT, 2024): https://github.com/MaartenGr/KeyBERT , https://arxiv.org/pdf/2409.10640 , https://towardsdatascience.com/keyword-extraction-a-benchmark-of-7-algorithms-in-python-8a905326d93f/ [source]
scikit-learn text feature extraction (TF-IDF, n-grams) & decomposition (LDA/NMF): https://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction , https://scikit-learn.org/stable/modules/decomposition.html [source]
Gensim - LDA, word2vec, similarity: https://radimrehurek.com/gensim/ [source]
Hugging Face Transformers - pipelines (sentiment, zero-shot classification, NER): https://huggingface.co/docs/transformers/main_classes/pipelines [source]
LLM-assisted qualitative coding / thematic analysis (2024–2026): https://journals.sagepub.com/doi/10.1177/16094069241231168 , https://journals.sagepub.com/doi/10.1177/16094069261426100 , https://arxiv.org/html/2510.18456v1 [source]

Children

Text preprocessing, tokenization, normalization (frontier)
Bag-of-words, TF-IDF, n-grams (frontier)
Topic modeling (LDA, NMF, BERTopic) (frontier)
Sentiment analysis (lexicon VADER vs transformer) (frontier)
Named-entity recognition (frontier)
Text classification (frontier)
Keyword and keyphrase extraction (RAKE, YAKE, KeyBERT) (frontier)
Embeddings and semantic clustering (word2vec, sentence-transformers) (frontier)
Document similarity (frontier)
LLM-assisted qualitative coding and structured extraction (frontier)
Evaluation of NLP outputs (frontier)

Frontier under this node: Bag-of-words, TF-IDF, n-grams, Document similarity, Embeddings and semantic clustering (word2vec, sentence-transformers), Evaluation of NLP outputs, Keyword and keyphrase extraction (RAKE, YAKE, KeyBERT), LLM-assisted qualitative coding and structured extraction, Named-entity recognition, Sentiment analysis (lexicon VADER vs transformer), Text classification, Text preprocessing, tokenization, normalization, Topic modeling (LDA, NMF, BERTopic)

← the whole tree · 3D view· how to read this page