AccuraCite Catches 99.4% of Fake Citations — Here's the Leak-Free Benchmark

How we tested it

We built a large test set of real citations from real published papers, mixed in citations engineered to be genuinely difficult — a fake title paired with a real author's actual name, a real citation with its DOI quietly swapped for an unrelated paper — and ran the whole set through AccuraCite to see what actually gets caught.

What's in the test set

627 of the 933 citations are real references from real published papers, spanning 8 academic fields:

And a wide range of publication years, not clustered in any one era:

How the fake citations were built

The remaining 306 citations are synthetic, constructed in five categories rather than as one undifferentiated pile of "fake":

• DOI-swapped — a real citation's title, authors, and year kept exactly intact, with its DOI replaced by one belonging to a real but unrelated paper. This is the case described above with the Nature and JAMA Network Open papers.
• Fully fabricated — a plausible-sounding title, author list, and DOI with no real paper behind any of it, written to match the phrasing conventions of the field it claims to belong to.
• Blended — title from one real paper, author list or venue from a different real paper, combined into a citation that matches neither source individually.
• Field-level mismatch — a real citation with one detail deliberately altered: the year shifted, a co-author dropped or swapped, or the venue changed to a plausible-but-wrong journal.
• Author-name edge cases — real citations with the kind of legitimate variation that a naive exact-match check would wrongly flag as broken: alternate transliterations, working-group authorship instead of named individuals, and diacritic differences across sources.

That last category isn't a "fake" in the sense of being wrong — it's in the benchmark specifically to test whether correct handling of a real citation gets penalized as a false positive, which is part of what the precision and false-positive-rate numbers below are actually measuring.

How a result was scored

Each citation in the set carries a ground-truth label — verified, mismatch, or hallucinated — assigned from the real paper's own published record (for the real citations) or from how it was constructed (for the synthetic ones), independent of what any lookup provider returns. A run counts as a "catch" only when AccuraCite's output label matches that ground truth; a fabricated citation that gets flagged as a mismatch instead of hallucinated, or vice versa, counts against the relevant metric even though it wasn't waved through as clean. Recall on the fabricated subset is what feeds the 99.4% headline number; precision and the false-positive rate are computed from how the real, verifiable subset was handled — specifically, how often a real citation was wrongly flagged instead of confirmed.

The numbers

Catching fabricated citations is the number that matters most — it's the failure mode that actually damages a paper, grant application, or legal brief if it slips through — and it's the one we optimized hardest for.

Built for the hard cases, not just the easy ones

Here's what we actually tested against, with real examples from the benchmark:

A real DOI attached to the wrong paper. Aravin et al.'s 2006 Nature paper, "A novel class of small RNAs bind to MILI protein in mouse testes", is real and well-cited. One test case kept its title, author list, and year exactly intact and swapped in a DOI belonging to a completely unrelated 2021 JAMA Network Open article. AccuraCite confirms the DOI actually points to the paper being cited, not just that it goes somewhere.

Fabricated papers written to sound like real research. One test citation claimed to be "Cognitive Behavioral Therapy for Treatment-Resistant Depression: A Randomized Controlled Trial Using Digital Therapeutics," attributed to a fabricated author, with a professional-looking but fake DOI — phrased exactly the way a real clinical-trial title reads. No such trial exists. AccuraCite requires the title, author, venue, and year to all agree before calling something a match.

Author names, handled correctly. A real citation to "Recommendations for the surgical treatment of endometriosis" lists its actual authorship as a medical-society working group, not a named person. And a real citation by Kozłowski & Konarzewski needs "Kozłowski" and "Kozlowski" recognized as the same author, even though different sources spell it differently. Both are common in real academic literature, and both are cases AccuraCite handles correctly.

The full dataset, every citation and result, is public: github.com/AccuraCite/accuracy-benchmark. Run compute_metrics.py yourself and check the numbers directly.

What this benchmark doesn't cover

A few things worth being direct about, since a benchmark is only as useful as its stated scope:

• English-language papers only. The test set doesn't include non-English publications, so we can't speak to accuracy on citations to papers indexed primarily in other languages.
• No books, and no grey literature. Every citation in the set is a journal article or conference paper with a DOI-bearing or otherwise indexable record. Books, theses, and non-indexed web sources go through a different fallback path in AccuraCite (a direct URL check, then a general web search) that this benchmark doesn't score separately.
• A snapshot, not a moving target. Provider databases change — new papers get indexed, some registrations get corrected or retracted — so a re-run of the same dataset a year from now wouldn't necessarily reproduce the exact same numbers, even with no changes to AccuraCite itself.

None of this changes the 99.4%/89.0% headline numbers, but it's the kind of context we'd want if we were reading someone else's benchmark instead of publishing our own.

Try it yourself

Verify your own bibliography free — no card required — and see what AccuraCite catches.

FAQ

What does "leak-free" mean? Our test citations came from real papers' own reference lists, not from the same lookup tools AccuraCite uses to verify citations. That way, the test can't be graded generously just because the same source is being used to both ask and answer the question. The full dataset is public on GitHub if you want to check this yourself.

What makes a fake citation "hard to catch"? Some of our fake test citations paired a real researcher's real name with a made-up, realistic-sounding paper title — built specifically to be harder to spot than an obviously nonsense reference. That's a meaningfully tougher test than checking whether a citation looks wrong at a glance.

Why does catching fakes matter more than avoiding false alarms? A real citation mistakenly flagged costs you a few seconds double-checking it. A fake citation that slips through undetected is the one that actually damages a paper, grant, or brief. We built and tuned AccuraCite with that in mind.

How were the 627 real citations selected? They're drawn directly from the reference lists of real published papers across the eight fields and publication-year ranges shown above, rather than hand-picked one at a time — the goal was a spread that looks like an actual bibliography, not a curated set of easy or convenient examples.

Will this benchmark be re-run as AccuraCite changes? We'd expect to re-run it against any change likely to affect verification accuracy, and to publish updated numbers rather than leave a stale result standing. The dataset and scoring script are public specifically so a re-run isn't something you have to take on faith.