Blog
Practical notes for ML leads and product managers buying annotation: writing the brief, reading a price, checking quality and choosing between a managed service and a platform.
- Quality
November 7, 2026How to measure annotation quality: agreement, audits and gold sets
Measure annotation quality three ways at once. Seed a hidden gold set you labeled yourself to score accuracy, track agreement between annotators (Cohen's kappa, Fleiss' kappa or Krippendorff's alpha) to find unclear guidelines, and audit a random sample of every delivery. Put the metric, the target and the consequence of a miss in the contract.
- Buying
November 7, 2026How to write a data annotation RFP
A good data annotation RFP describes the task with real edge cases, states volume and weekly pace, names the quality metric and target, lists the security evidence you need, and asks every vendor to price in the same format. End it with a paid pilot on your own data, scored against a gold set you labeled yourself.
- Buying
November 7, 2026Managed annotation service or labeling platform?
Buy a managed service when you need the annotators as well as the tool, the dataset is large and one-off, and you have no spare people. Buy a platform when your own team or experts will label, the data must stay in your environment, or the work is ongoing. Many teams start managed and move routine work onto a platform later.
- Pricing
November 7, 2026Per-hour vs per-task annotation pricing
Per-task pricing fixes the cost of each label, so it suits stable, well-defined work where you can count items in advance. Per-hour pricing pays for time, so it suits exploratory, variable or expert work. To compare the two, ask every vendor for expected items per hour, and convert.
Get a quote
A ZIP code, a reply address, then a few taps describing the work. We match the outline against what each vendor publishes and write back with names; no vendor hears about you until you pick one.
Keep data, samples, label guides and logins out of this form. The outline on its own is enough for us to work from.
How this site earns: as an introduction service, it may be paid by the annotation company you are introduced to.
Received. Next steps from our side:
- Your outline is set against each vendor's published modalities, minimums, security reports and pricing.
- We write to you naming the vendors we think fit, and why.
- Nothing goes to any vendor until you reply to say which introductions you want.