Every annotation vendor says its quality is high. The useful question is how you would know if it slipped. Three tools answer it, and each catches a problem the others miss.
What are the three ways to measure label quality?
| Method | What it tells you | What it misses |
|---|---|---|
| Gold set | Whether labels match the answer you consider correct | Only covers the items in the set |
| Inter-annotator agreement | Whether the guidelines are clear enough for people to agree | Two people can agree and both be wrong |
| Random audit | Whether a delivery as a whole meets the bar | Depends on the auditor’s own accuracy |
Used together, they show you both the error rate and its cause.
How do you build a gold set?
A gold set is a few hundred items labeled by your own team, or by your best experts, and checked twice. Mix it into the vendor’s work without marking which items belong to it. Each time an annotator labels a gold item, you get a score you can trust, because you set the answer.
Some practical rules:
- Cover the hard cases. A gold set of easy items will report 99% accuracy while the real work drifts. Weight it toward the edge cases in your guidelines.
- Refresh it. Annotators learn which items repeat. Rotate in new gold items every few weeks.
- Score per annotator, not just overall. One weak annotator can hide inside a good average.
- Keep it out of the training data. It is a measuring stick, and it loses value once a model has seen it.
For shapes, accuracy usually means overlap: the intersection over union (IoU) between the vendor’s box or mask and yours, above a threshold you set, often 0.5 for loose detection work and higher for tight segmentation. For transcription, it means word error rate. For classification, it is plain accuracy or a per-class F1 score.
What does inter-annotator agreement measure?
Agreement scores ask a different question: when two or more annotators label the same item, how often do they match beyond what chance alone would produce?
- Cohen’s kappa compares two annotators on categorical labels. It was introduced by Jacob Cohen in 1960 and corrects raw agreement for the agreement you would expect by chance.
- Fleiss’ kappa extends the idea to more than two annotators, each item labeled by several people.
- Krippendorff’s alpha handles any number of annotators, missing labels, and ordinal or interval scales, which makes it a good fit for rating tasks such as scoring LLM answers from 1 to 5.
A low score rarely means the annotators are careless. Much more often it means the guideline leaves room for two readings. When agreement drops on one class, read the items where people disagreed; the fix is usually a sentence in the guide and two more examples.
There is no universal “good” number. Simple, well-defined labels can reach very high agreement; subjective tasks such as tone or helpfulness will sit lower even when the work is careful. Set the target after the pilot, from what your best annotators achieve, rather than from a figure in a vendor’s deck.
How should audits work?
An audit is a reviewer checking a random sample of a finished delivery. The two choices that matter are sample size and who audits.
On size, a fixed share (for example 5% of items) is easy to explain, but small deliveries then get very few checks. A minimum count per delivery, such as 200 items or the whole delivery if smaller, gives you a steadier read.
On who audits, compare three setups:
- The vendor’s own reviewers. Cheapest and fastest, but they mark their own homework.
- Your team. Most reliable, but it costs your people’s time.
- A second vendor or a separate review crew. Independent, and useful on large programs, at extra cost.
Many teams accept the vendor’s internal review for routine work and keep a smaller audit of their own on top. Whatever you choose, agree in advance what happens when a delivery fails: free rework, a credit, or both.
How do you write quality into the contract?
A quality clause that holds up has four parts:
- The metric. Accuracy on gold items, IoU threshold, word error rate, or agreement score.
- The target. A number, set from the pilot.
- The sample. How many items are checked per delivery, and who picks them.
- The remedy. Rework at no charge within a set number of days, plus a credit if a second attempt fails.
Ask vendors during the RFP how they report each of these. Some publish details of their review layers, consensus labeling or gold-standard checks; others only say they have “multi-step QA”. Our methodology explains how we treat each when ranking the best data annotation companies.
Which warning signs should you watch for?
- Accuracy on gold items is high but agreement between annotators is falling: the gold set may be too easy.
- Agreement is high but gold accuracy is low: the team has settled on a shared, wrong reading of the guideline.
- Quality dips every time the guideline changes: the vendor is not retraining the crew before the change goes live.
- Reports arrive late or only as averages: ask for per-annotator and per-class figures.
Quality you can measure is quality you can negotiate; everything else is a promise.