How we rank annotation vendors

Each vendor is scored out of 16 on what it publishes on its own website: pricing, the modalities it labels, the security reports it lists, and whether it sells both a platform and services. We have not tested any vendor, and the score does not measure label quality.

Last verified November 7, 2026. All 18 vendors were read on that date.

What is scored, and how?

Criterion	Points	How points are given
Published pricing	0, 4 or 6	6 if dollar prices are published; 4 if plan structure, units or a free tier are published without dollar rates; 0 if quote only.
Modalities	0 to 6	One point for each of image, video, text, audio, 3D or sensor data, and code that the vendor says it covers.
Security reports	0 to 3	SOC 2 Type II counts 2; SOC 2 of unstated type, ISO 27001, ISO 27701, ISO 9001 and TISAX count 1 each; capped at 3. Self-declared "compliant" statements do not count.
Platform and services	0 or 1	One point when a vendor sells both a tool and the people to use it, which gives buyers room to change model later.
Quality controls	Not scored this round	We looked for published, specific quality controls (review layers, agreement targets, gold-set checks). None of the pages we read on November 7, 2026 gave detail we could verify against a source, so this criterion did not separate vendors this round.

What rules decide the order?

Ties are broken by security points, then pricing points, then name.
Only the vendor's own pages count. Press coverage, review sites and comparison sites are not used as evidence for any column.
Gaps are printed as gaps. If a price or certificate is not on the pages we read, the table says "not published" or "none found".
Fees do not move anyone. A vendor may reward us after a team we introduced hires it. Scores, ranks and sentences ignore that entirely.
Worker brands are not listed. Sites that recruit annotators for a listed vendor are not separate vendors and are left out.

The current scores run from 2 to 11. The full table is on the ranking page.

Terms changelog

Dated changes to what the vendors publish, newest first. We re-read every listed page on each check.

November 7, 2026Baseline: all 18 vendor pages in the table read and scored for the first time. Quality controls left unscored this round.
October 2026Appen revised its post on reinforcement learning with verifiable rewards; the post describes 500 coding tasks, every one shipped with its own automatic checker (Appen blog, read Nov 7, 2026).
September 22, 2026Snorkel AI issued a press release on a $350M Series E round, with a stated valuation of $3.5B (Snorkel AI press release, read Nov 7, 2026).

Where these facts come from

Questions about the method

Why rank on published information instead of testing?

Testing 18 vendors fairly would mean running the same paid project with each, which we have not done. Published information is something every buyer can check, and it shows how much a vendor lets you see before a sales call. We say plainly that it is not a measure of label quality.

Why does pricing count double?

Because it is the gap buyers complain about most. A vendor that publishes prices lets you budget and compare before you commit time to calls; most annotation vendors do not.

Can a vendor ask to be re-scored?

Yes. If a vendor publishes new prices or certifications on its own site, we re-check the page and update the table and the changelog. We do not accept private documents as evidence for the public score.

Why are some well-known names missing?

A company is listed only when we could read its own pages on the check date and confirm the facts in each column. Companies we could not verify are left out until the next check rather than listed with guesses.

How we rank annotation vendors

What is scored, and how?

What rules decide the order?

Terms changelog

Questions about the method

Read next

Get a quote

Received. Next steps from our side: