How to evaluate an AI tool for evidence synthesis
You want to know how well an AI tool performs in systematic reviews, perhaps well enough to publish the answer. This guide takes you through an evaluation in four steps, whether you test the tool during a review you are running or on reviews that are already finished.
How this works
An evaluation asks one question: when the AI and human reviewers decide on the same records, how often is the AI right? You answer it by having both decide independently and comparing the two afterwards.
If you evaluate during a review of your own, most of the work is work the review needs anyway: a protocol, a pilot, two reviewers and a way to resolve disagreements. If you evaluate on finished reviews, the reviewers’ decisions already exist and you run the AI on the same records. Either way, the evaluation adds a few rules. The most important one is that you stop adjusting the AI before you measure it.
| Step | When | What you have at the end |
|---|---|---|
| 1. Plan | Before you start | A short protocol that says what you will measure |
| 2. Calibrate and freeze | On a small sample first | Settings and criteria that no longer change |
| 3. Run | On the remaining records | AI and reviewer decisions on the same records, disagreements resolved |
| 4. Report | When all decisions are in | A result with its uncertainty, ready to write up |
What are you planning?
Your answer decides how many of the 18 checklist items you need. The page shows only the ones that apply.
For a trial, keep it simple. Run the AI alongside your blinded reviewers in one live or recent review. Measure sensitivity with a 95% confidence interval, and count unclear AI output as “include”. One page of protocol is enough, and you need not register it. If the tool version changes during the trial, start again or note the change.
A trial tells your team whether to use the tool. It is not a validation you can publish, so if you may want to publish later, choose the other option now.
The examples describe screening within a live review, because that is the most common case. Where finished reviews work differently, the step says so. Search, data extraction and risk of bias follow the same four steps with a different measure, which step 1 helps you choose.
Plan what you will measure
You make four decisions here and write them down. Writing them down first is what lets a reader trust the result later.
Decide where you will test
Within a live review. The AI works on a review while it is being done. This is the strongest design, because the records, criteria and reviewers are real. Cochrane’s CESAR protocol is a worked example.
On finished reviews. The AI repeats decisions that were already made, which lets you cover many reviews quickly. Use reviews published after the model’s training cutoff, or unpublished ones, for your main result, because the model may have seen older reviews during training. Report older public benchmarks such as CLEF TAR 2019 or SYNERGY separately. A published review rarely lists every record it excluded, so check what its decisions cover.
A third option, a randomised comparison of two workflows, is worth the effort only if your question is time saved or the effect on the review’s conclusions.
Decide what role the AI will have
Evaluate the AI in the role you intend to give it. NICE describes three roles, and each one needs a different comparison.
| Role | What the AI does | What you compare |
|---|---|---|
| Support | Suggests; a reviewer decides every record. | Reviewers with AI support against reviewers without. |
| Semi-automation | Replaces one of two reviewers; a human resolves disagreements. | One reviewer plus AI against two reviewers. |
| Full automation | Decides alone. | The AI alone against your reference standard. |
NICE does not recommend full automation for full-text screening, data extraction or risk of bias, and allows only support use for risk of bias. Whichever role you choose, also report what the AI achieves on its own.
Decide what counts as correct
You need a set of decisions to measure the AI against. This is your reference standard. Build it from two independent reviewers. Reviewers make mistakes too: a single reviewer gets about one in ten abstract decisions wrong. So in step 3 a third person looks again at every record where the AI and the reviewers disagree.
Make sure the decisions belong to the stage you are testing. For title and abstract screening, compare with your title and abstract decisions, not with the studies you finally include.
Choose your measure
Pick one main measure for the stage you are evaluating, and decide now what you will compare it with.
| Stage | Main measure | Watch out for |
|---|---|---|
| Search | Recall: the share of your included studies the AI search found. | Check that the tool covers the same databases. It cannot find what your own search missed. |
| Title and abstract screening | Sensitivity: the share of relevant records the AI kept. | Accuracy looks high even when the AI misses studies, because most records are excludes. Do not use it as your headline. |
| Full-text screening | Sensitivity, on records that passed title and abstract screening. | Report what text the AI read. If it fell back to the abstract, it did a different task. |
| Data extraction | Concordance: the share of data items that match. | Decide before you start what counts as a match. Count values the AI invented as errors. |
| Risk of bias | Agreement per domain (percent and weighted κ). | Reviewers often disagree with each other here. Measure their agreement too. |
Do not set a pass mark such as “95% recall”; NICE advises against it. Compare the AI with how your own reviewers did in the same study, or with published human error rates. Also decide how you will count records where the AI gave no clear answer. For screening, count them as “include”.
- 1Name the role of the AI (support, semi-automation or full automation) and the review stage.NICE ECD20
- 2Choose the design: within a live review, on finished reviews the model has not seen, or a randomised comparison of workflows.PublicationRAISE 1, rec. 3.1.3CESAR protocol
- 3Define the reference standard: two independent reviewers, with decisions that match the stage.PublicationNICE ECD20RAISE 2
- 4Fix the main measure, the comparison that decides, how unclear AI output counts, and the sample.PublicationNICE ECD20RAISE 2
- 5Write the protocol. Include your interests and any help from the developer you plan to use.NICE ECD20RAISE 1, rec. 3.7
If you plan to publish, send us your draft protocol. We comment on the role, the settings, the criteria wording and the pilot design, and you decide what to adopt. We also cover the AI usage in your protocol free of charge. See How Puraite supports you.
How many records do you need?
What matters is the number of relevant records, not the total you screen. The table shows how certain you can be if the AI finds 95% of the relevant records in one review.
| Relevant records | Found by the AI | 95% interval for sensitivity |
|---|---|---|
| 20 | 19 | 0.76 to 0.99 |
| 40 | 38 | 0.83 to 0.99 |
| 60 | 57 | 0.86 to 0.98 |
| 100 | 95 | 0.89 to 0.98 |
| 200 | 190 | 0.91 to 0.97 |
One review with few includes gives you a case study. That is still worth publishing if you show the wide interval. For a general claim about a tool you need about 100 to 120 relevant records spread over at least 15 to 20 reviews.
Mistakes that have distorted published evaluations
- Adjusting the AI on the records used for the final result. In Oami 2024, the planned prompt reached a sensitivity of 0.75; prompts written after seeing results reached 0.89 to 0.91. Step 2 protects you from this.
- A test set with an unrealistic share of includes. In Sanghera 2025, precision was 0.93 on a balanced set and 0.004 to 0.096 on the real one. Testing within your own review avoids it.
- An unchecked reference standard. This usually makes the AI look worse than it is.
- Different input for the AI and the reviewers. In Konet 2024, the same model scored 68.8% reading a PDF through a plug-in and 100% given the text.
Calibrate, then freeze
You are allowed to adjust the AI to the review. You just have to finish adjusting before you measure.
Take a small sample of records and set it aside as your calibration sample. Use it the way you would use a pilot: your reviewers screen it, you look at where the AI differs, and you sharpen the wording of your eligibility criteria or change settings. Repeat if you need to. Note the size of the sample, the number of rounds and each change you make. If you test on finished reviews, take the calibration sample from reviews you will not use for the result.
When you are satisfied, freeze: from now on the settings, the criteria wording and the tool version stay as they are. All remaining records are your evaluation set, and the calibration sample is not part of it.
The reason is simple. If you change something after seeing how the AI did on the evaluation set, you no longer know whether the result would hold on records it has not seen.
- 6Calibrate settings and criteria wording on a calibration sample kept out of the evaluation set. Record each round and change.RAISE 2
- 7Check the configuration against your protocol line by line. Agree how reviewers mark records they keep for other reasons, such as reviews kept for their reference lists.PublicationOur recommendation
- 8Freeze settings, criteria wording and tool version. Advice from the developer ends here.RAISE 2
- 9Register the protocol together with the frozen configuration.PublicationRAISE 1, rec. 3.3
Use a pilot round for the calibration sample. It draws a random sample, your reviewers screen it, and you see how the AI’s decisions compare with their consensus after each change to the criteria. We can help you set it up.
Before you freeze, note what the “AI version” section of your project settings shows (release, model and effort), and export the audit trail to record your criteria wording.
Where to register
OSF Registries works for any evaluation. If the evaluation is part of a review, the SWAR Store is made for studies within a review, and you can mention it in the review’s own protocol. PROSPERO registers systematic reviews only.
What if you find a problem after the freeze?
It happens, and it does not ruin your evaluation as long as you report it. Three cases:
- The tool has a defect. It does not behave as documented: a run fails or records are skipped. Pause, let the developer fix it, re-run the affected records and report it.
- Your setup does not match your protocol. For example, a criterion ended up in the wrong list. Correct it against the protocol, never against the results. Re-run the affected records, and report the earlier results as a sensitivity analysis.
- You disagree with the AI’s decisions. That is a result, not a problem to fix. Rewording criteria now would be calibration, so it has to wait until after the evaluation.
Run the AI on the same records as the reviewers
In a live review, your reviewers screen as they normally would. The AI screens the same records, and neither sees the other’s decisions.
On finished reviews, the reviewers’ decisions already exist. Run the AI on the same records, and make sure nothing it receives reveals those decisions.
Two things keep the comparison fair. Your reviewers must not see the AI’s output while they decide, or their decisions stop being independent. And the AI should get the same material as your reviewers: the same abstracts, the same full texts and the same version of the criteria. Where that is not possible, say so in your report.
When all decisions are in, collect every record where the AI and your reviewers disagree. A third person decides each one against the written protocol, without knowing which decision came from the AI. Some of these will turn out to be reviewer errors. That is expected, and you report it.
- 10Run the AI on the evaluation set, with reviewers blinded to its output and the same material for both.NICE ECD20Konet 2024
- 11Keep a log of every deviation: pauses, fixes, version changes and contact with the developer.PublicationRAISE 2The contact log is our recommendation
- 12Run the AI again on a random subset, at least twice, and report how often the runs agree.PublicationNICE ECD20RAISE 2
- 13Have every disagreement between the AI and the reference standard decided by someone who does not know its source.NICE ECD20
In a live review, add the AI to your project in the Shadow role. It screens every record, and your reviewers do not see its decisions until the stage is finished. For finished reviews, see what Puraite cannot do yet.
For item 12, do not re-run the AI in the same project, because that overwrites its earlier decisions. Copy your random subset with the project setup into two new projects straight after the main run, run the AI once in each, check that the “AI version” matches, and compare the three exports.
After the freeze we answer technical questions and fix defects, and we send you a log of every contact so that you can report it.
Why run the AI more than once?
A language model can give different answers to the same record. If you run a subset two more times and the answers rarely change, your readers know the result is stable. If they change often, that is an important finding in itself.
Report what you found
Report enough that someone else could judge your result and repeat your study.
- Your main measure with a 95% confidence interval, and the full table of counts behind it. The interval shows how much the result could differ in another review.
- How your reviewers did, measured the same way. A missed study means something different if single reviewers miss as many.
- The records the AI wrongly excluded. Say what they were and, if you can tell, why the AI missed them. This is often the most useful part for other reviewers.
- What exactly you tested: tool, version, model, run dates and settings, plus how you calibrated and anything that deviated from the plan.
- Your interests and any involvement of the developer.
There is no agreed reporting checklist for this kind of study yet. Use the reporting framework in RAISE 2 and TRIPOD-LLM. In the review itself, describe the tool under PRISMA 2020 items 8 and 9.
Publish whatever the result is. A study that shows where a tool fails helps other review teams as much as one that shows where it works.
- 14Report the main measure with a 95% confidence interval, the full table and the human error rate.NICE ECD20
- 15Analyse the wrongly excluded records.PublicationRAISE 2
- 16Report tool, version, model, run dates, settings, calibration history and deviations.RAISE 1, rec. 3.5TRIPOD-LLM
- 17Disclose your interests and any involvement of the developer.PublicationRAISE 1, rec. 3.7RAISE 2
- 18Publish whatever the result, share your record-level decisions and code, and send the published report to the developer.PublicationRAISE 1, recs. 3.4–3.6 and 3.8
The screening decision export gives you each reviewer’s and the AI’s decision, reasoning and timestamp per record, with the model and the run dates. The Screening statistics page shows sensitivity, specificity and the table of counts with 95% intervals; see Screening statistics in the Help Center (sign-in required).
For item 16, report the release, model and effort from the “AI version” section, and the sampling settings (temperature 0.2 and seed 42 as of 10 November 2026). For full-text screening, report the share of AI decisions made from the abstract, which the Text Basis column of the export shows.
For the reference and a methods sentence, see How to cite Puraite. If you like, we check the statements about how Puraite works in your manuscript. We do not comment on results or conclusions.
How to word the disclosure
Help from a developer is allowed if it ended before the freeze and you declare it. If you have a financial interest in the tool, declare it and do not call the study independent (RAISE 1, rec. 3.7). Free access is not a financial interest, but mention it. You can adapt this:
Puraite (version X) was used under [free evaluation access / a paid licence]. Puraite staff [reviewed the protocol before the main run / checked factual statements about the tool]. Puraite had no role in data collection, adjudication, analysis, interpretation or the decision to publish. [Author] declares [interest / none].
Sharing your data
Share record-level decisions with identifiers such as DOIs or PMIDs. Do not share abstracts or full texts, which are usually under copyright.
How Puraite supports you
If you plan to publish an evaluation of Puraite, we help you set it up well, and then we step back.
You do not need to be a customer. This is what you can ask for, and when:
| When | What we do |
|---|---|
| Step 1 | Free evaluation access, covering the AI usage in your protocol until publication, plus three months for revisions. Protocol review: we comment, you decide. |
| Step 2 | Help with the pilot round on your calibration sample, and a line-by-line check of your configuration against your protocol. |
| Step 3 | Our advice ends at the freeze. We only answer technical questions and fix defects, and we log every contact for you. |
| Step 4 | A factual check of your manuscript, if you want one, within 10 working days. Afterwards we list your published evaluation on our site, whatever its result. |
We do not run the AI on your data, do the analysis, decide disagreements or write any part of your manuscript.
What we never do
- We do not ask for your results before publication. If you share them, we do not ask for changes.
- We never approve, veto or delay publication, and never select which results get reported.
- We never ask for conclusions to be changed, and never edit them.
- We never change your configuration or your reference standard.
- We never require an NDA, and never offer payment or other compensation beyond evaluation access.
- We do not co-author a study that is presented as independent.
There is no separate contract. Our normal terms of service and data-processing agreement apply. We confirm these terms and the support you asked for by email, and you can cite that email in your disclosure.
If your evaluation shows that Puraite needs to improve, we want to see it. We file a product issue for each confirmed weakness.
If Puraite changes while your evaluation is running
We always tell you about a version or model change that affects your project. You then choose: restart the main run on the new version, or finish and report the change as a deviation with results split by version. We never make that choice for you. If a defect blocks your evaluation, we fix it for all customers, not tuned to your data, and tell you the release and what changed.
What Puraite cannot do yet
As of 10 November 2026. We update this list as things change.
- Prompt version. It is not yet shown or frozen. Report the release and the run dates.
- Repeat runs. There is no built-in feature. Use the copy method described in step 3.
- Finished reviews. Testing against existing decisions needs an import, which we switch on for you on request. It reads Rayyan exports only.
- Full texts. The AI reads text extracted from the PDF, while your reviewers read the PDF itself. State this in your report.
- Data extraction. There is no built-in comparison. Run blinded human passes and an AI pass, export the values with their origin, and score them yourself.
- Risk of bias. A blinded comparison of AI and reviewers is not possible yet.
Plan an evaluation with us
Tell us which stage you want to evaluate, the role of the AI, your planned design and timeline, and roughly how many records you expect. We reply within 5 working days.
Sources
- NICE ECD20. AI methods for evidence: framework. Published 9 November 2026.
- RAISE 1 to 3. Responsible AI in Evidence Synthesis, version 4, consultation draft of 25 September 2026.
- Joint position statement of Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence, 2025.
- PRISMA 2020. PRISMA-AIM is registered and under development.
- TRIPOD-LLM. Reporting guideline for studies using large language models.
- CESAR. Cochrane’s protocol for evaluating AI screening within reviews.
- Studies: Oami 2024; Sanghera 2025; Konet 2024.
Changelog
We correct and update this guide as guidelines and Puraite change. Cite the version you used.
| Version | Date | What changed |
|---|---|---|
| 1.0 | 10 November 2026 | First published. |