Before I Trust the Model, I Audit the Data | Life, The Universe, and Everything

type

status

date

slug

summary

1. Define what the label is supposed to mean

My first question is not “Which aggregation algorithm should I use?” It is “What are we asking annotators to decide?”

Consider a request that quotes a dangerous instruction in order to criticize it. One annotator may flag the words. Another may judge the speaker's intent. A third may evaluate whether a particular response would enable harm.

They are answering different questions. Voting does not fix that.

In my notes, I separated several explanations for disagreement: an accidental mistake, unclear instructions, a consistent rater bias, and a legitimate difference in perspective. The distinction matters because each explanation calls for a different response. Retraining an annotator might help with misunderstanding. Rewriting the rubric might help with ambiguity. Neither necessarily resolves a genuine difference in what people value.

I find it useful to separate observation from judgment. First record the evidence: what was requested, what context was available, and what happened. Then apply a named rubric. “Contains a threat” and “violates policy version 3” should not silently become the same field.

A hard label can still be necessary for a decision. But I would preserve the underlying votes, evidence, disagreement, and rubric version. A single operational decision does not require pretending that the evidence was unanimous.

What I take from the MIT lecture

MIT's Dataset Creation and Curation lecture organizes the problem around task definition, example collection, and label collection. It covers selection bias, deployment-relevant validation, and learning curves that vary training-set size while holding validation fixed.

For multiple annotators, it distinguishes three outputs: a consensus label, confidence in that label, and an estimate of annotator quality. Its main comparisons are majority vote, Dawid–Skene, and CROWDLAB. MACE is an extension from my related learning questions, not one of those three main comparisons.

The lesson I draw is broader than label cleaning: a dataset can have consistent labels and still answer the wrong question.

2. Agreement is useful. It is not truth.

One of my early practice solutions scored annotators by agreement with the majority, then used those scores for weighted voting. It is an understandable baseline. It is also easy to overinterpret.

Suppose five annotators all use the same mistaken interpretation. They agree perfectly. That does not make the interpretation correct.

There is also a smaller circularity: when I score a worker against a majority that includes their own vote, that worker helps construct the reference used to judge them. Leaving their vote out reduces that direct self-influence. It does not remove shared bias.

This is why I would call the resulting number an agreement score, not annotator accuracy. If a worker labeled only a few items, I would also retain the sample count and shrink extreme estimates. If different workers received different kinds of tasks, I would compare their task mix before ranking them.

Majority voting remains a good starting point. In an idealized binary setting with independent workers, each better than chance, adding votes can reduce error. Copy the same worker instead, and the additional votes add no independent evidence. The distinction is easy to demonstrate without a large model.

In the first notebook, a deliberately misspecified independent-vote model receives twenty copies of the same incorrect vote. With an assumed 80% reliability for each “source,” it becomes almost certain of the wrong answer. Count that source once, and its posterior is only 80% under the same balanced-prior assumptions.

The number of votes is not the number of independent reasons to believe something.

3. Choose a label model by the errors it can represent

I think of aggregation methods as different explanations for how the observed labels were produced—not a ladder where each more elaborate method is automatically better.

From one weight to a confusion matrix

Weighted voting gives each annotator a scalar weight. That cannot fully express a worker who is good at recognizing safe content but often misses unsafe content.

Dawid–Skene instead models a class-conditional confusion matrix for each worker. The entry

has the target on the row and the observed vote on the column. Keeping that direction straight matters. It is not the probability that the target is correct given the worker's answer. The original paper and Comparing Bayesian Models of Annotation develop this family of models.

The two updates that initially felt circular to me are:

E-step: given the current worker matrices and class prior, estimate a probability distribution over each item's target.

M-step: use those soft target estimates to update the worker matrices and class prior.

For an item with votes , the E-step is:

In plain language: start with the base rate, ask how likely these votes would be under each possible target, and normalize. The product assumes conditional independence between workers. The notebook implements the calculation in log space and adds smoothing to avoid zero probabilities.

A concrete example helps. Assume one worker is 95% accurate on both classes, while two others are each 60% accurate. They vote [0, 1, 1]. With a balanced prior, the likelihoods are:

The posterior probability of target 0 is about 89.4%. The reliable minority wins.

But those reliabilities were assumed for this example. The three votes do not establish them. In real data, estimating worker matrices requires enough observations and overlap, and the result depends on the model's assumptions. A global prior also describes a particular population; I would not casually reuse it across populations with different class frequencies.

MACE asks a different question

MACE models whether a worker follows the target or generates a response from a worker-specific guessing distribution. That is more constrained than a full confusion matrix.

A useful detail: its competence parameter is not simply accuracy. A guess can accidentally be correct. A worker's probability of answering correctly therefore depends on both competence and guessing behavior.

GLAD adds another distinction by modeling worker expertise and item difficulty. A difficult assignment should not automatically make its annotator look incompetent.

My practical comparison is:

Method	What it adds	What I would challenge
Majority vote	Equal treatment of observed votes	Are the workers making independent errors?
Weighted vote	A reliability weight per worker	Where did the weights come from?
Dawid–Skene	Class-specific worker error patterns	Is there enough overlap to estimate them?
MACE	Competence and guessing behavior	Does that explanation fit these mistakes?
GLAD	Worker expertise and item difficulty	Does a difficulty model capture the ambiguity?
CROWDLAB	Classifier probabilities alongside votes	Is the classifier providing useful new evidence?

None of these methods can guarantee semantic truth from shared misunderstanding alone.

4. Use the model to find questions, not to declare answers

Another question in my learning was whether a simple classifier could correct labels whenever it was more confident than the majority vote.

The appealing rule is: two of three annotators choose A, but the model assigns B a probability of 0.98, so trust the model.

That comparison is not justified. The fraction 2/3 measures observed agreement. The value 0.98 is a model's estimated probability. They are not interchangeable measures of correctness. Even a probabilistic classifier needs calibration checks before its probabilities can be interpreted that way.

I would start with a narrower use of the model: prioritize examples for inspection.

Train a simple baseline. Produce out-of-fold predictions, so each training item is scored by a model that did not train on its label. Rank confident disagreements. Send a fixed budget to independent review. Preserve the original label and record why it changed. Put preprocessing inside the cross-validation pipeline, and exclude related items together when they share a user, document, or duplicate family. This follows the same separation principle as scikit-learn's leakage guidance.

Out-of-fold prediction reduces direct memorization of the inspected label. It does not make the classifier an independent source of truth. The classifier can still learn a bias shared across the training data.

CROWDLAB is not training-dynamics analysis

My notes put these ideas close together, but they deserve separate names.

CROWDLAB combines classifier probabilities with multiple annotators' labels to estimate consensus and quality. The classifier can contribute information from the item's features, especially where annotations are sparse. Unlike Dawid–Skene's product of worker likelihoods, CROWDLAB uses a weighted combination of classifier and annotation-derived probability vectors. The question becomes how much to trust each source, rather than simply which source sounds most confident.

Dataset Cartography instead examines how confidence in the supplied label and its variability evolve during training. It distinguishes easy-to-learn, ambiguous, and hard-to-learn regions.

Both can help direct an audit. Neither makes “hard to learn” synonymous with “mislabeled.” A rare but important example may be difficult precisely because the model has not learned the right behavior yet.

5. Evaluate the cleaning process, not just the cleaned model

Here is the trap I most want to avoid: infer new labels, train a model to predict them, and announce success because the model matches those labels better.

That might measure improved imitation of the labeling procedure. It does not establish improved correctness.

I separate three targets. Annotation prediction asks whether I can predict what a worker will say. Consensus prediction asks whether I can reproduce an aggregation rule. Task correctness asks whether the result satisfies an independently specified target. These can be useful measurements, but they support different claims.

My process notes make the experimental boundary explicit: preserve the split; check that it is by item rather than annotation row; fit annotator parameters on training data; freeze them; choose models and thresholds on validation; reserve the final test for evaluation.

If the test reference is itself inferred by a label model, I would still call it a proxy reference, not gold. Freezing the inference model prevents one kind of leakage. It does not upgrade the reference into observed truth.

I also distinguish trusted development examples from a locked test reference. Development examples may help refine instructions or estimate worker quality. Once they influence those decisions, they are not an untouched test. A “gold” set is a carefully constructed measurement instrument, not something made infallible by its name.

A small curation experiment

The second notebook fixes a simple classifier and a review budget of 120 training items. It compares no changes, confident automatic relabeling, targeted review, and random review. Simulated reviewers reveal the generator's target only for selected training items. The classifier is evaluated on separate items using features alone.

In this particular synthetic run, 69.2% of the targeted review queue contained wrong labels, compared with 19.0% on average across five random-review seeds. Those are reproducible outputs in the notebook, not an expectation for real datasets. The simulated reviewer is also unrealistically perfect.

The important design choice is measuring two outcomes separately: whether the labels improved, and whether the downstream classifier improved. I would add review time and important-slice performance in a real workflow.

A targeted queue also cannot estimate the overall label-error rate without accounting for its selection mechanism. Finding many errors in a queue designed to find errors does not imply that most of the dataset is wrong.

6. Decide what evidence to collect next

My active-learning notes use three criteria: uncertainty, expected usefulness, and diversity. I still think that is a better starting point than “label whatever the model is least certain about.”

One hundred nearly identical uncertain examples can consume a budget without adding much coverage. My simple proposed experiment is to take an uncertain candidate pool, cluster it, select representatives, and compare against random sampling at the same budget. The second notebook demonstrates the selection step.

BADGE develops a more specific approach using gradient embeddings to combine uncertainty and diversity. It is not merely “take the largest gradient.” For softmax cross-entropy, the last-layer gradient involves probabilities minus the target indicator, not logits minus the label.

My notes also explored synthetic labels, augmentation, and semi-supervised learning. I would keep their roles separate. LLM-generated labels expand supervision but do not certify it. A paraphrase is useful only when it preserves the relevant meaning. A noisy label does not necessarily make the input worthless.

That last distinction motivates DivideMix, which uses estimated clean and noisy subsets within a semi-supervised procedure. It is more than a rule for deleting high-loss samples. My takeaway is to ask which information is untrustworthy—the label, the input, the task definition, or all three—before discarding the example.

For collection, I would maintain both a representative sample and a targeted challenge set. The first estimates ordinary performance. The second exposes specific weaknesses. They should not silently share the same interpretation.

7. Inspect the denominator before trusting the score

The arithmetic can be correct while the conclusion is wrong.

Consider a hypothetical safety detector screening 10,000 requests. Exactly 100 are unsafe. At 90% recall and a 1% false-positive rate, it produces:

	No alert	Alert
Actually safe	9,801	99
Actually unsafe	10	90

It catches 90 unsafe requests, but only 90 / 189 = 47.6% of its alerts are true positives. A system that marks everything safe gets 99% accuracy and catches nothing.

The 1% false-positive rate divides by safe requests. The 52.4% false-alert fraction divides by alerts. Confusing those denominators hides the operational burden.

With prevalence , recall , and false-positive rate , precision is:

A balanced evaluation set can therefore produce very different precision from deployment. This calculation holds the class-conditional error rates fixed; actual distribution shifts may change those too. I would report the evaluation population, not just the metric.

Metric names also need a contract

Average precision and trapezoidal precision–recall area are different calculations. Scikit-learn documents the distinction. In the third notebook's six-example case, they are approximately 0.7222 and 0.6778. The direction of the difference is not a universal rule.

Ranking metrics answer different questions too. Recall@K concerns which relevant items were retrieved; NDCG also depends on their ordering and relevance grades. Neither repairs incomplete or inconsistent relevance judgments.

No observed failures is not a guarantee

For zero failures in independent, representative Bernoulli trials, the exact one-sided 95% upper confidence bound on the failure probability is:

Zero failures in 1,000 trials gives an upper bound of about 0.30%, not zero. The notebook checks the calculation with SciPy's exact binomial interval.

The assumptions are as important as the formula. Repeated variants of one prompt are not necessarily independent. A hand-picked attack suite is not a random sample of real traffic. And recall's uncertainty depends on the number of positive cases, not the total number of screened requests.

8. An LLM judge is another annotator

The same questions return when the labeler is an LLM. What instructions does it follow? Which mistakes does it make? Does its agreement with humans hold on hard cases, or mostly on easy ones?

I would use a clear rubric, compare model judgments with independently reviewed references, and inspect false approvals and false rejections by failure type. I would keep examples used to tune the judge separate from examples used to assess it.

For agents, there is a further distinction: the transcript is not the outcome. An agent can say it completed an edit while changing the wrong record. Anthropic's agent-evaluation guide makes this distinction explicit and discusses code-based, model-based, and human graders.

The third notebook constructs a deliberately weak transcript-only judge. It approves every completion message. State checks find that only 60% of the toy tasks succeeded. No real LLM is being measured; the example isolates the flaw in the grading rule.

My proposed rule is simple: use state checks for actions, evidence checks for factual claims, and calibrated judgment for questions that require interpretation. Do not ask one fluent-sounding score to stand in for all three.

9. Rebuild the ideas in notebooks

The accompanying notebooks contain executed outputs and assertions. They run on CPU without network calls once the dependencies are installed.

Draft attachment note: The three companion notebooks and data_quality_blog_and_notebooks.zip remain attached to the originating ChatGPT conversation. Their upload to this Notion page did not succeed. The filenames below identify the files in that download package; they are not Notion download links.

Notebook	Main experiment
01 — Who should we trust? `01_who_should_we_trust.ipynb`	Majority and weighted votes; Dawid–Skene from scratch; a four-item EM exercise; copied-vote failure; MACE's observation model.
02 — Audit before relabeling `02_audit_before_relabeling.ipynb`	Item-level splitting; out-of-fold label audit; fixed-budget review versus random review; learning curves; diverse uncertainty sampling.
03 — Evaluation is not just a score `03_evaluation_not_just_a_score.ipynb`	Base rates; AP versus trapezoidal area; rare-failure bounds; ranking metrics; simulated judge versus actual state.

These are new teaching notebooks, not reproductions of every paper. MACE is illustrated rather than fully fitted; CROWDLAB, GLAD, BADGE, and DivideMix are explained or linked, not reimplemented.

For the course's own exercise, use the MIT dataset-curation lab. Its environment and execution are separate from the notebooks accompanying this post.

The hardest distinction

The hardest step for me is not memorizing the E-step and M-step. It is keeping two ideas separate:

Inferring what is probably true under a model is not the same as independently checking what is true for the task.

That distinction transfers from crowdsourced labels to synthetic data, retrieval judgments, reward models, and agent evaluations. In each case, I want to know what generated the signal, what assumptions connect it to the target, and what evidence could show that those assumptions failed.

My aim is not to make every example look clean. It is to know which parts of the evidence are reliable, which remain uncertain, and whether an improvement survives a test that was not built to agree with it.