Text as Data

Bag of Words (BoW) is the simplest idea: throw all the words of a document into a "bag," shake it up (forget the order), and just count how many times each word appears. The result is a big table (called a Document-Term Matrix) where each row is a document and each column is a word.

TF-IDF improves on raw word counts by asking: "Is this word special to THIS document, or does it appear everywhere?" Words that appear in almost every document (like "the" or "is") get downweighted, while words unique to a few documents get boosted. This helps surface the words that truly distinguish one document from another.

词袋模型 (BoW) 是最简单的思路：把一篇文档的所有词丢进一个"袋子"，摇一摇（忘掉顺序），然后数每个词出现了几次。结果是一张大表格（叫文档-词项矩阵），每一行是一篇文档，每一列是一个词。

TF-IDF 在原始词频的基础上多问了一个问题："这个词是这篇文档独有的，还是到处都有？"几乎每篇文档都出现的词（如"的""是""了"）被降权，而只在少数文档中出现的词被升权。这样就能找出真正让一篇文档区别于其他文档的关键词。

BoW treats every word as completely unrelated — but we know "happy" and "joyful" are similar! Word embeddings fix this by representing each word as a list of numbers (a "vector") where similar words have similar numbers. The core idea is beautifully simple: "you shall know a word by the company it keeps" (Firth, 1957). If "cat" and "dog" often appear near the same words ("pet," "feed," "cute"), they must mean similar things.

Word2Vec (Mikolov et al., 2013) learns these vectors by training a simple neural network to predict either a word from its neighbors (CBOW) or neighbors from a word (Skip-gram). The famous result: vec("king") − vec("man") + vec("woman") ≈ vec("queen") — the model learned gender relationships just from reading text!

GloVe (Pennington et al., 2014) takes a different approach: instead of looking at local word windows, it analyzes the overall co-occurrence statistics of the entire corpus.

词袋模型把每个词都当作完全无关的——但我们知道"开心"和"高兴"是近义词！词嵌入解决了这个问题：把每个词表示为一串数字（"向量"），意思相近的词，数字也相近。核心思想非常优美："通过一个词的伙伴来了解它"（Firth, 1957）。如果"猫"和"狗"经常出现在相同的词旁边（"宠物""喂""可爱"），它们的意思一定很接近。

Word2Vec（Mikolov et al., 2013）通过训练一个简单的神经网络来学习这些向量：要么从邻居词预测中心词（CBOW），要么从中心词预测邻居词（Skip-gram）。著名的结果：vec("king") − vec("man") + vec("woman") ≈ vec("queen")——模型仅仅通过阅读文本就学会了性别关系！

GloVe（Pennington et al., 2014）采用不同的方法：不是看局部词窗口，而是分析整个语料库的全局共现统计。

The simplest text analysis you can do: make a word list, then count. For example, a sentiment dictionary might say "happy" = +1, "terrible" = -1, "okay" = 0. To score a movie review, just add up the scores of all its words. That's it!

Popular dictionaries include LIWC (psychology), AFINN (sentiment), and Hu & Liu (opinion mining). Strengths: Transparent (you can see exactly why a document got its score), reproducible, and requires zero training data. Limitations: Can't handle context — it thinks "not good" is positive (it sees "good"!), misses sarcasm entirely, and the same word can mean different things in different domains ("sick" is negative in healthcare but positive in slang).

最简单的文本分析：建个词表，然后数数。比如，一个情感词典可能规定"开心"= +1，"糟糕"= -1，"一般"= 0。给一条影评打分，就是把所有词的分数加起来。就这么简单！

常见词典包括 LIWC（心理学）、AFINN（情感）和 Hu & Liu（意见挖掘）。优势：透明（你能清楚看到为什么一篇文档得了这个分数）、可复现、完全不需要训练数据。局限：不理解上下文——它会认为"不好"是正面的（因为它看到了"好"！），完全忽略讽刺，而且同一个词在不同领域含义不同（"有毒"在化学里是危险的，在网络用语里可能是"太搞笑了"）。

"What are people talking about?" — that's the question topic models answer. Latent Dirichlet Allocation (LDA) (Blei et al., 2003) imagines that every document was written by first picking a few topics, then picking words from those topics. LDA works backwards: given the words we observe, it figures out what the hidden topics must be.

The researcher must choose how many topics K to look for — there's no single right answer, just like there's no single right number of folders to organize your files. You try different values and see which gives the most interpretable results. Common tools: held-out likelihood, topic coherence scores, and good old-fashioned "do these topics make sense to a human?"

"大家都在讨论什么？"——这就是主题模型要回答的问题。潜在狄利克雷分配（LDA）（Blei et al., 2003）想象每篇文档是这样写出来的：先选几个主题，再从这些主题里选词。LDA 做的是反向推理：给定我们看到的词语，推断出背后隐藏的主题是什么。

研究者需要选择寻找多少个主题 K——没有唯一正确的答案，就像整理文件时没有唯一正确的文件夹数量一样。你试不同的值，看哪个给出最容易解读的结果。常用工具：留出似然、主题连贯性分数，以及最经典的"这些主题对人类来说有道理吗？"

Regular LDA discovers topics but can't tell you why topics vary across documents. STM (Roberts et al., 2014) is LDA's smarter cousin: it lets you plug in extra information about each document — like who wrote it, when, or where — and then the model can tell you how topics differ across those categories.

普通的 LDA 能发现主题，但没法告诉你主题为什么在不同文档间变化。STM（Roberts et al., 2014）是 LDA 的"升级版"：它允许你加入每篇文档的额外信息——比如谁写的、什么时候写的、在哪里发表的——然后模型就能告诉你主题是如何随这些因素变化的。

Sometimes you don't want topics — you want to know where someone stands. Text scaling places documents (or their authors) on a spectrum, like a political left–right scale, based purely on word choices.

Wordscores (Laver et al., 2003) needs reference texts — you give it a few documents with known positions (e.g., a clearly left-wing and a clearly right-wing manifesto), and it figures out where new documents fall by looking at which reference they use more similar words to.

Wordfish (Slapin & Proksch, 2008) doesn't need references at all — it discovers the latent dimension purely from how word frequencies differ across documents. Think of it as "let the data tell me who's on which end."

有时候你关心的不是主题，而是想知道某人站在哪里。文本量表化根据用词选择把文档（或作者）放在一条光谱上，比如政治左右轴。

Wordscores（Laver et al., 2003）需要参考文本——你给它几篇已知立场的文档（比如一篇明确左翼和一篇明确右翼的宣言），它通过看新文档与哪个参考更像来确定位置。

Wordfish（Slapin & Proksch, 2008）完全不需要参考——它纯粹从词频在文档间的差异中发现潜在维度。可以理解为"让数据告诉我谁在哪一端"。

Imagine asking 100 people to classify a document, but each person only sees a random portion of the words. Each person makes an imperfect judgment, but when you take a majority vote, the crowd is surprisingly accurate. That's Random Forest: it builds hundreds of decision trees, each looking at a random subset of features, then averages their predictions.

A bonus: Random Forest tells you which words matter most for the prediction (called feature importance), so you can peek inside and understand why the model classifies things the way it does.

想象让 100 个人给一篇文档分类，但每个人只看到随机一部分词。每个人的判断都不完美，但投票表决后，群体的准确率出奇地高。这就是随机森林：它构建几百棵决策树，每棵树只看随机一部分特征，然后综合它们的预测。

额外好处：随机森林会告诉你哪些词对预测最重要（叫特征重要性），让你能窥探模型内部，理解它为什么这样分类。

Neural networks don't need you to tell them what features to look for — they figure it out themselves from the data. For text, three architectures matter:

CNNs — Like a sliding magnifying glass that scans text for useful local patterns. Good at spotting short phrases like "not good" or "highly recommended."

RNNs / LSTMs — Read text word by word, remembering what came before. Like a person reading a book from start to finish, keeping track of the plot. LSTMs are the improved version that can remember things from much earlier in the text.

Transformers — The breakthrough architecture behind ChatGPT. Instead of reading word by word, transformers look at ALL words at once and figure out which words are important for understanding each other. (More details on the LLM & NLP page.)

神经网络不需要你告诉它该关注什么特征——它自己从数据中摸索出来。对于文本，有三种重要的架构：

CNN——像一个滑动的放大镜，扫描文本寻找有用的局部模式。擅长发现短语，如"not good"或"强烈推荐"。

RNN / LSTM——逐词阅读文本，记住之前读过的内容。就像一个人从头到尾读一本书，一直跟踪剧情。LSTM 是改进版，能记住文本中更早出现的信息。

Transformer——ChatGPT 背后的突破性架构。不是逐词阅读，而是同时看所有词，搞清楚哪些词对理解彼此最重要。（详见 LLM & NLP 页面。）

LLMs have read billions of web pages during pre-training, so they already "understand" language before you give them any task-specific data. Two key players:

BERT — Reads text in both directions (like reading a sentence forwards AND backwards at once). Give it a few hundred labeled examples and it quickly learns to classify your specific task. The "fine-tuning" approach.

GPT (ChatGPT, GPT-4) — A text generator trained to predict the next word. The magic: you can simply describe what you want in natural language (a "prompt") and it does it — no training data needed. Give it zero examples (zero-shot), a few examples (few-shot), or step-by-step instructions (chain-of-thought).

LLM 在预训练阶段已经读过了数十亿网页，所以在你给它任何特定任务数据之前，它就已经"理解"语言了。两个关键角色：

BERT——双向阅读文本（像同时正着和倒着读一句话）。给它几百个标注样本，它就能快速学会你的特定分类任务。这是"微调"路线。

GPT（ChatGPT、GPT-4）——一个文本生成器，训练来预测下一个词。神奇之处：你只需用自然语言描述你想要什么（一个"提示词"），它就能做到——不需要训练数据。给零个例子（零样本）、几个例子（少样本），或者逐步说明（思维链）都行。