Key insights
What we've learned so far from 51 senate questions and 436 House districts asked across 14 AI surfaces — consumer web apps captured in real browsers, and official APIs. Every response is stored, graded against time-versioned ground truth, and human-reviewable; browser captures carry full-page screenshots.
1 · Accuracy is high — and the errors are structured, not random
On the "who represents me" basics, every major surface answers correctly the large majority of the time (ChatGPT API GPT-5: 86% · ChatGPT API GPT-5 + web search: 96% · Claude API Opus 4.8: 55% · Claude API Opus 4.8 + thinking: 90% · Claude API Opus 4.8 + web search: 96% · Gemini API 3.1 Pro Preview: 78% · Gemini API 3.1 Pro Preview + search: 96% · Grok API Grok 4: 71% · ChatGPT web Plus: 92% · ChatGPT web anonymous: 100% · ChatGPT web free account: 90% · Claude web Pro: 84% · Claude web free account: 88% · Google Search AI Overview: 88% on senate questions). The failures that remain cluster into a few repeatable classes below — which matters more for civic information than a raw error rate, because structured failures hit the same kinds of voters the same way every time.
2 · Breaking news beats ground truth — chatbots were ahead of official sources
When Sen. Lindsey Graham died on July 11, 2026 and his sister Darline Graham Nordone was appointed days later, web-searching chatbots reported the succession correctly — with citations — while senate.gov still listed a vacancy and our own multi-source ground truth lagged. Verdict systems that grade against stale references will mislabel fresh, correct answers as "hallucinated." Time-versioned ground truth plus human review is the only defensible design. (Notably: the free ChatGPT tier garbled the new senator's name where paid tiers got it right.)
3 · Location bias turns ambiguity into wrong answers
Asked "Who are my U.S. senators in Washington?", four of five web surfaces answered with the senators for the asker's IP-geolocated state — Massachusetts — reading "Washington" as Washington, D.C. and falling back to "where you are." Consumer AI answers civic questions relative to where it thinks you are; for movers, travelers, VPN users, or ambiguous place names, that silently produces the wrong officials.
4 · Google's AI Overviews confuse federal and state offices
In repeated cases (Minnesota and Oklahoma senate questions, across independent runs), Google's AI Overview answered a U.S. Senate question with state legislature resources. No chatbot made this error. Overviews also simply don't appear for a small share of civic queries — the citizen gets ten blue links and no direct answer.
5 · Access is rationed: free tiers hit walls fast
Free Claude allowed roughly a dozen civic questions per 5-hour window before demanding an upgrade; even paid tiers throttle sustained use (ChatGPT Plus: "Too many requests"; Claude Pro: session limits). For an individual voter this rarely bites — but it shapes what any civic-information service built on consumer AI can deliver, and it means the "free" civic answer machine is materially rate-limited infrastructure.
6 · The judge needs the human: false "hallucinations" are a real class
Our LLM judge proposes verdicts; humans confirm. That layer earns its keep: responses that correctly reported post-cutoff news (Texas primary results, the South Carolina succession) were initially graded "hallucinated" because the ground truth hadn't caught up. Roughly speaking, most flagged "hallucinations" in these runs were the judge being behind reality, not the chatbot inventing it — the reverse of the usual assumption.
Method, in one paragraph
Questions are templated and fanned across jurisdictions; consumer web apps are exercised by real, headed Chrome sessions on dedicated research accounts (fresh conversation per question, memory/personalization disabled, human-paced with jitter), APIs are called server-side; every capture is graded by an LLM judge against multi-source, time-versioned ground truth (senate.gov, clerk.house.gov, ballotinfo, candidata, caucus-ai, FEC, Wikipedia), with human review as the final authority. Browser captures store full-page screenshots and HTML keyed to job ids.