AI respondents missed by 12 points. Human answers still have a price.
Pew tested AI stand-ins for survey takers and found large errors. Meanwhile AI labs pay billions for human judgement. What both stories say about the value of real answers.

Two things are true about AI and opinion data in 2026. First, AI models are not good substitutes for people when you want to know what people think: Pew Research Center tested it and found errors big enough to change conclusions. Second, the companies building AI are paying more than ever for human judgement, because their models need it to improve. Both point the same way. Real answers from real people are getting scarcer and more valuable at the same time.
Can AI stand in for survey takers?
The idea is attractive. Instead of paying 1,000 people, prompt a language model to answer as 1,000 different people (“synthetic respondents” or “silicon samples”) and read off the results in minutes.
Pew Research Center tried it properly and published the result on 30 September 2026. “Across nearly 300 individual survey questions,” the AI respondents’ estimates differed from real people’s “by an average of 12 percentage points,” and “the average difference exceeded 15 points on around 28% of questions we asked.”1
The misses were not small or random:
- The model said “97% of Hispanic adults are at least somewhat likely to follow the World Cup (while the actual share is 43%).”1
- It “overstated the share of Americans who approved of President Donald Trump’s job performance at the time.”1
- Where a “not sure” option was offered, “human panelists were around four times as likely as the AI model to choose it.”1
That last finding is the deepest problem. Real people are uncertain, inconsistent and sometimes uninterested. A model trained to be helpful produces confident, tidy opinions. Pew used Anthropic’s Claude Opus 4.6 for most results,1 a frontier model, so this is not a story about weak technology. Pew’s own position is plain: it “has no current or future plans to use AI models to generate survey results.”1
Market researchers found something similar. A 2026 study in the NIM Marketing Intelligence Review reported that “synthetic data consistently overestimated positive attitudes” and “showed significantly less variation across respondents compared with real participants,” even though synthetic participants “matched real participants’ choices about 79% of the time.” The authors’ verdict on the best method they tested: “an improvement, but not a replacement for real data.”2
Why do AI labs pay so much for human data?
While researchers test whether AI can replace people, AI labs are spending heavily to hire people. Models learn what a good answer looks like from humans who label, rank and judge outputs.
- In June 2025, Meta made “a $14.3 billion investment in artificial intelligence company Scale” for a 49% stake. AP described Scale’s pitch as supplying “the human labor needed to improve AI systems.”3
- Surge AI, a rival in the same business, had “grown its annual revenue to more than $1 billion,” according to reports cited by SiliconANGLE in July 2025, which put Scale’s sales the previous year at $870 million.4
- Mercor, another company in the human-data market, announced in October 2025 a round that “values Mercor at $10 billion.”5
Regulation adds pressure on where training data comes from. Under the EU AI Act, rules for general-purpose AI models took effect in August 2025, and the Commission’s training-data template “requires providers to give an overview of the data used to train their models.”6
What makes a human answer worth paying for?
Not every human answer is worth the same. Buyers of human data, whether a market researcher or an AI lab, pay for three things:
- It is a real person. The bot problem we covered in a separate post applies to AI training data too. Answers produced by another model are worse than useless for teaching a model about people.
- It is a different person each time. One person with ten accounts is a sample of one.
- The person agreed to the use. Consent to answer a survey is not consent to have your answers train a model. Good platforms ask separately.
Pi Network is moving into the same space. At its Ecosystem Summit on 18 November 2026, after this post first went up, it previewed Pi DataLab, “a new human-in-the-loop platform designed to connect businesses, AI developers” with identity-verified Pi users, for work “requiring human input, including data labeling, ranking, evaluation.”7 It is an early announcement, but it shows where the market is going: identity-checked people, paid for judgement.
Where does Sibr stand?
Sibr Earn already pays verified people for research. Paid studies go only to Pi-verified accounts,8 respondents see “the pay, the estimated time and the best pay per minute first,” and people who do not fit a study are “paid the screen-out fee” with their answers not kept.9
Earn studies are for verified humans, and Sibr’s guide on verification says “AI training is always a separate, unticked choice in the consent step.”10 Human data is valuable because it comes from people; treating those people as a resource to mine would destroy the thing that makes it valuable.
Sources
-
Pew Research Center, report on AI models as survey respondents, 30 September 2026. ↑ ↑ ↑ ↑ ↑ ↑
-
Kaiser et al., NIM Marketing Intelligence Review 18(1), 2026, study of synthetic respondents in market research ↑
-
Associated Press via WJXT, report on Meta’s investment in Scale AI, 12 June 2025. ↑
-
SiliconANGLE, report on Surge AI’s first capital raise, 1 July 2025. ↑
-
Mercor, Series C announcement, 27 October 2025. ↑
-
European Commission, AI Act: regulatory framework (updated 19 October 2026). ↑
-
Pi Core Team, Ecosystem Summit, day 2, 18 November 2026. ↑
-
Sibr, sibr.online. ↑
-
Sibr, Verified human respondents: why they matter for research and AI. ↑
Sibr
Have a question for real people?
Make a poll or a survey on Sibr in a few minutes, share one link, and watch the answers come in as charts. It is free to create and free to answer.
Open Sibr