An AI agent passed 99.8% of survey attention checks. Now what?

A Dartmouth study showed AI agents can answer online surveys undetected, and Pew found nearly half of one opt-in sample failed a screening check. What still works.

A grid of soft violet pebble shapes with a few hard-edged black squares hidden among them, and a magnifying lens over one corner.

Online surveys have always had fake respondents: people clicking through for a reward, people taking the same survey twice, simple scripts. What changed in the last year is that the fakes got good. A 2025 study from Dartmouth showed that an AI agent can pass almost every attention check researchers use, and in 2026 Pew Research Center reported that no single cleaning method reliably removes bad respondents from opt-in online polls.

The practical conclusion is uncomfortable for anyone who buys or runs online research: checks inside the survey are no longer enough. You have to know who is answering before they start.

What did the Dartmouth study find?

Sean Westwood, a political scientist at Dartmouth, built an AI tool that takes surveys like a person would. According to Dartmouth’s announcement, “In 43,000 tests, the AI tool passed 99.8% of attention checks designed to detect automated responses.”1 He reports that every detection method in use failed: “Westwood tested every AI detection method currently in use and found that all failed to identify his tool.”2

Two numbers from the same release explain why this matters beyond academia:

  • Cost. Human respondents typically earn about $1.50 per survey, while the bot’s answers cost about five cents each.1
  • Leverage. For seven major national polls before the 2024 US election, the release says that adding as few as 10 to 52 fake AI responses, at five cents each, would have flipped the predicted outcome.1

The agents also “work even when programmed in Russian, Mandarin, or Korean, yet produce flawless English answers,”1 so a survey’s language is no barrier. The paper, published in PNAS, went on to win the journal’s 2025 Cozzarelli Prize.2

How common are bogus respondents already?

Pew Research Center has measured this twice, six years apart.

In 2020, Pew found that opt-in online polls “still had 3% to 7% of interviewing coming from bogus respondents,” while panels recruited by mail had “only trace levels.”3 Bogus respondents were not random noise: the share claiming to be Hispanic “was three times higher (30%) among cases flagged for bogus behavior,”3 which skews exactly the subgroups researchers most want to report on.

In August 2026, Pew published a larger test. It fielded an opt-in survey of 11,114 US adults in November 2024. “A total of 1,963 cases (18%) answered ‘Yes’ to one or more of these trap questions and were flagged as bogus.” A commercial pre-screening tool flagged even more: 5,369 cases, nearly half of all completes, failed at least one of its checks.4

Pew’s conclusion is careful: “Overall, purging bogus cases lowers error on most metrics, but a surefire solution remains elusive.”4 One fix even backfired: “Matching an opt-in sample to voter files slightly increased error by removing mostly good respondents.”4

Do attention checks still work?

Not well enough. Prolific, which runs a large research panel, tested several ways to catch AI agents in February 2026. “Traditional attention checks achieved just 32% sensitivity and 86% specificity.” A newer idea, questions designed to trip up machines, “performed barely above chance.”5 Prolific says its own check reached perfect scores in that test, on a small sample (125 humans and 5 agents run 25 times each), and it is honest about the limits: “The paper didn’t examine or prove that agents are currently flooding sample providers.”5

That last line is worth keeping in mind. Nobody has shown that agents already dominate commercial panels. What has been shown is that they could, cheaply, and that the usual checks would not notice.

What still works?

Every defence that holds up moves the check earlier, from “is this answer plausible?” to “is this a real, unique person?”.

Defence What it catches Weakness
Attention checks and trap questions Careless humans, simple bots AI agents pass them1
Behaviour analysis during the survey Speeders, copy-paste, odd timing An arms race with better agents
Address-based recruitment (mail) Almost everything Slow and expensive; Pew found 1% bogus3
Identity check before joining Duplicates, bots, fake demographics Needs a trusted identity provider

Identity checks are spreading outside research too. Decrypt reported in April 2026 that the network of World (formerly Worldcoin) had reached “18 million verified humans across 160 countries.”6 Human Passport, formerly Gitcoin Passport, reported “over 34.5M credentials for 2.1M+ users” in early 2025.7 Pi Network says its own KYC has verified more than 18 million people.8

What this means for Sibr

This is why paid studies on Sibr go only to Pi-verified accounts.9 Pi runs KYC on its members and keeps a one account per person policy, so the check happens before anyone can earn.10 It does not make bad answers impossible: “A real person can still answer carelessly.”10 But it removes the cheapest attack, which is one operator running thousands of agents, and it lets us spend the in-survey checks on quality instead of on proving humanity.

Open polls and surveys on Sibr stay open to anyone with a link, because not every question needs a verified sample. When the answer will drive a decision or a payment, ask who is answering before you ask anything else.

Sources

  1. Dartmouth College, news release on EurekAlert!, 17 November 2025. ↑ ↑ ↑ ↑ ↑

  2. Dartmouth Faculty of Arts and Sciences, news item on the PNAS Cozzarelli Prize, 28 April 2026. ↑ ↑

  3. Pew Research Center, report on bogus respondents in online polls, 18 February 2020. ↑ ↑ ↑

  4. Pew Research Center, report on bogus respondents in opt-in polls, 27 August 2026. ↑ ↑ ↑

  5. Prolific, test of methods for detecting AI agents, 4 February 2026. ↑ ↑

  6. Decrypt, report on World’s verification partners, 17 April 2026. ↑

  7. human.tech, announcement of the Gitcoin Passport acquisition, 10 February 2025. ↑

  8. Pi Core Team, Pi2Day 2026, 28 June 2026. ↑

  9. Sibr, sibr.online. ↑

  10. Sibr, Verified human respondents: why they matter for research and AI. ↑ ↑

Sibr

Have a question for real people?

Make a poll or a survey on Sibr in a few minutes, share one link, and watch the answers come in as charts. It is free to create and free to answer.

Open Sibr