How to Evaluate Radiology AI Vendor Evidence: An 8-Question Scorecard

Every radiology AI vendor arrives with evidence. It might be a sensitivity figure, a reader study, a conference poster, or sometimes a peer-reviewed trial. The problem is that these are not equal, and the slide deck rarely tells you which one you are looking at. A retrospective test on a hand-picked image set and a randomized trial inside a national screening program can both be described as “clinically validated.”
This guide gives imaging leaders eight questions to ask of any AI study. It then applies them to four published mammography AI studies, so you can see what strong and weak evidence actually look like. The point is not to rank products. It grades how much a given study can tell you about how a tool will behave in your practice.
Why AI vendor evidence is hard to read
Three traps catch most buyers.
Enriched test sets inflate performance. Many AI studies test on image sets deliberately loaded with cancers, so they can get statistically meaningful results with a few hundred cases. That is a reasonable research choice, but it is not your screening population. In one widely cited mammography reader study, the test set was about 50% cancers, while real screening populations run at roughly 0.5% to 1%. Readers and algorithms behave differently when almost every other image is positive.
“Validated” does not mean what it sounds like. In AI research, validation often refers to a step during model development where researchers check performance and tune parameters. Physicians tend to hear it as clinical validation. The CLAIM 2024 reporting guideline from RSNA now discourages the term for this reason and recommends terms such as testing or evaluation instead. When a vendor says a model was validated, ask what was done, on whose data, and in what setting.
Summaries round up. Press releases compress nuanced results into headlines. The final results of the MASAI randomized trial are a good example:
- The result: AI-supported screening was noninferior to standard double reading on the trial’s primary endpoint, interval cancers. The AI arm had 1.55 interval cancers per 1,000 women, versus 1.76 in the control arm.
- The headline: that is a 12% lower rate, and a vendor summary described it as a 12% reduction. Both statements are accurate.
- The nuance: the confidence interval around that difference included no difference at all, and the vendor itself later described the reduction as descriptive.
Knowing how to read that distinction is the difference between “proven to reduce missed cancers” and “proven not to be worse, with a promising trend.”
The 8-question evidence scorecard

Use these questions on any study a vendor shares. Rate each one Strong, Partial, or Weak. Do not add the ratings up into a single score; the pattern matters more than a total. A study can be excellent in design and weak in fit to your setting, and both matter.
1. What was the study design?
Randomized controlled trials sit at the top, followed by prospective studies inside real clinical workflows, then retrospective analyses and reader studies. Retrospective work can provide an early signal, but it cannot show how radiologists behave when the AI is live.
Strong: randomized, or prospective within routine practice. Red flag: only retrospective or simulation results presented as real-world performance.
2. Was the population real-world?
Check whether the study used a consecutive screening or clinical population, or a curated case set. Look at disease prevalence and how cases were selected.
Strong: consecutive patients at realistic prevalence. Red flag: enriched case sets with no real-world follow-up study.
3. Was it independent of the developer’s own data?
The minimum bar is performance on data the model never saw, from sites that did not contribute training data. More sites and more countries are better.
Strong: external testing across multiple sites, with a statement that trial data were not used in development. Red flag: results only from the institutions that built the model.
4. Does it match your setting?
Check scanner manufacturers, modalities, and patient groups. Exclusions matter as much as inclusions: if the study excludes a group, the evidence doesn’t cover it.
Strong: multiple scanner vendors, and exclusions stated with reasons. Red flag: a single scanner vendor that is not yours, or unexplained exclusions.
5. Did it measure outcomes that matter?
Area under the curve and standalone sensitivity are starting points. For screening, the outcomes that matter are cancer detection rate, recall rate, and ideally interval cancers, the cancers that appear between screening rounds after a negative result.
Strong: clinical outcomes with adequate follow-up. Red flag: only AUC, or only detection without recalls.
6. Is the human and AI workflow described?
The same algorithm can help or hurt depending on how its output reaches the reader, when it appears, and who decides. Reporting guidelines differ on this:
- The CONSORT-AI guideline for AI trials requires researchers to describe how they used AI output.
- The CLAIM 2024 and TRIPOD+AI guidelines do not specifically address human-AI interaction, so check this yourself.
Strong: the reading workflow, timing of AI display, and decision authority are explicit. Red flag: “radiologists used the AI” with no further detail.
7. Who funded it, and who wrote it?
Vendor funding does not invalidate a study, but it raises the bar for independent confirmation. Check the funding statement and the competing interests section, including authors employed by the vendor.
Strong: public or independent funding with disclosures. Red flag: vendor-funded, vendor-authored, and no independent study exists.
8. Is it final, preregistered, and transparently reported?
Interim and preliminary analyses can change. Preregistration on a public registry shows the endpoints were fixed in advance. Reporting guidelines such as CLAIM 2024 for imaging AI and STARD-AI for diagnostic accuracy studies set the standard for what should be disclosed.
Strong: final results, preregistered endpoints, and data or protocols available. Red flag: preliminary results presented as conclusive.
Download the one-page Radiology AI Evidence Scorecard, an AI vendor evaluation checklist you can fill in during a vendor meeting.
The scorecard applied: four published mammography AI studies
To show the scorecard in use, we applied it to four published studies of two commercial breast AI products: Lunit INSIGHT MMG and ScreenPoint Medical’s Transpara. For each product, there is an earlier or narrower study and a stronger one. All four are respected publications. The point is that evidence strength varies even within the same product, and that no study answers every question.
| Question | Kim et al. 2020, Lunit | AI-STREAM 2025, Lunit | AITIC 2026, Transpara | MASAI 2023 to 2026, Transpara |
|---|---|---|---|---|
| 1. Design | Weak. Retrospective, enriched, multi-reader, multi-case reader study. | Partial. Prospective multicenter cohort; observational, not randomized. | Partial. Prospective paired noninferiority trial; every exam read both ways. | Strong. Randomized controlled noninferiority trial. |
| 2. Population | Weak. 320-mammogram reader set, half cancers. | Partial. 24,543 women in Korea’s national screening program, at academic hospitals with higher than expected prevalence. | Strong. 31,301 women in Spain’s Andalusian screening program. | Strong. About 106,000 women in Sweden’s national screening program. |
| 3. Independence and sites | Partial. Model tested on data from five institutions in three countries; reader set and readers from separate institutions. | Strong. Six academic hospitals; readers had no prior experience with the AI. | Partial. Single site, but trial data were not used in algorithm development and thresholds were fixed in advance. | Partial. Four screening sites in one Swedish region. |
| 4. Setting match | Partial. Training data from several manufacturers; reader set from Korean screening only. | Strong. GE and Hologic systems; women with prior breast cancer or breast surgery excluded because the AI was not validated for them. | Weak. Four devices from a single vendor; women with visible implants excluded. | Weak. Single mammography vendor and single AI system, per published commentary. |
| 5. Outcomes | Weak. AUC and reader performance on a test set. | Partial. Detection and recall at one year; interval cancers pending registry linkage. | Partial. Workload, detection, recall, PPV; interval cancers cannot be measured in a paired design. | Strong. Interval cancer rate as primary endpoint, plus detection, recall, and workload. |
| 6. Human and AI workflow | Partial. Unaided versus aided reading in a study setting. | Strong. Readers recorded a decision without AI, then with AI; the first decision was locked. | Strong. Triage rules fully specified; radiologists decided all recalls for flagged exams. | Strong. AI score routed exams to single or double reading and added detection marks. |
| 7. Funding and authors | Weak. Funded by the vendor; vendor authors. | Strong. Government grant; no competing interests declared. | Partial. Not industry funded; one author employed by the vendor. | Strong. Public funding from Swedish cancer and research bodies. |
| 8. Final and transparent | Partial. Final publication; registration not applicable to a retrospective study. | Weak. Preliminary analysis, no correction for multiple comparisons; preregistered. | Strong. Final, preregistered, de-identified dataset public; note the paper lists two different registry numbers. | Strong. Final results published; preregistered. |
Scores reflect what each paper reports, reviewed by [Radiologist name]. They grade the evidence, not the products. Read the original papers before relying on any single cell.
What the four studies teach
The same product can have very different evidence behind it. Lunit’s evidence shows the range:
- The 2020 reader study was vendor-funded and tested on an enriched image set, the normal profile of early-stage evidence.
- AI-STREAM, five years later, prospectively tested the same product family at six hospitals within a national screening program, with government funding.
Both are real evidence, but only the second tells you much about routine screening. Ask a vendor for its strongest study, not its most quoted one.
Even the best study leaves questions about your setting. MASAI is the strongest evidence in this set: a publicly funded randomized trial with interval cancers as its primary endpoint. It was also run in one Swedish region, with one mammography vendor and one AI system, in a program with its own reading conventions. If your scanners, population, or reading model differ, the result is encouraging but not automatically transferable.
Good studies state their limits plainly, and that is a quality signal. AITIC reported three weaknesses openly:
- Its recall rate rose and did not meet its noninferiority target.
- The AI automatically cleared 11 cancers.
- Its paired design cannot measure interval cancers.
It also published its de-identified dataset. Transparency about weaknesses should increase your confidence in a study, not decrease it.
Watch for details that connect to your infrastructure. AI-STREAM notes that its radiologists could compare each exam with prior mammograms, while the AI itself had no ability to analyze earlier images. Radiologists rely heavily on prior comparison, and many AI tools still read each study in isolation. When you evaluate a tool, ask whether it uses prior exams at all, and if it does, whether your archive can deliver those priors to it reliably.
Questions to ask in the vendor meeting
Turn the scorecard into direct questions:
- What is your strongest study, prospective or randomized, and can we read the full paper?
- Was it run on our scanner manufacturer and a population like ours?
- Which patient groups is the product not validated for?
- Do your results include recall rates and interval cancers, or only detection and AUC?
- When your materials say validated, what exactly was done, and on whose data?
- How did radiologists see the AI output in the study, and who made final decisions?
- Who funded the study, and is there independent confirmation?
- Are these results final and preregistered?
Once the evidence holds up, the next step is to compare radiology AI vendors on product scope and integration fit.
After the evidence, check the infrastructure
Strong published evidence describes performance in someone else’s environment. Two of the studies above excluded participants for infrastructure reasons:
- AI-STREAM excluded participants because of data errors on a cloud server.
- AITIC excluded images it could not process, including some affected by erroneous PACS image transfer.
Before deployment, confirm three things: the AI can reliably receive complete studies, it can access prior exams when needed, and it can write results back to the worklist with an audit trail. Then monitor local performance after go-live.
Medicai’s role is the imaging layer underneath the AI: storing studies, routing them to AI tools through standard interfaces such as DICOMweb, and keeping results attached to the study. For how those layers fit together across PACS, VNA, RIS, and EHR, see our guide to enterprise imaging architecture.
Frequently asked questions
At minimum, a peer-reviewed study testing the product on data independent of its development, in a population similar to yours. The strongest evidence is a prospective or randomized study inside routine clinical workflow. It reports clinical outcomes such as detection and recall rates, discloses funding and author interests, uses preregistered endpoints, and presents final rather than interim results.
A reader study asks radiologists to interpret a fixed set of past images with and without AI, often enriched with extra cancers. It measures how readers perform in a controlled setting. A prospective trial applies the AI to real patients as they arrive, inside routine workflow, and follows what happens to them. Reader studies are useful early evidence; prospective trials show real-world effects.
CLAIM, the Checklist for Artificial Intelligence in Medical Imaging, is a reporting guideline published in RSNA’s journal Radiology: Artificial Intelligence and updated in 2024. It tells researchers what to report in imaging AI studies, so readers can judge and reproduce the work. Buyers can use it in reverse: a study missing what CLAIM requires is harder to trust.
Not necessarily. Most radiology AI tools reach the US market through FDA 510(k) clearance, which is based on showing substantial equivalence to a device already on the market. Clearance defines a cleared intended use, but it does not by itself mean that prospective clinical outcome studies exist. Check the cleared indication, and ask for the clinical evidence separately.
Sources
- Tejani AS, Klontzas ME, Gatti AA, et al. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update. Radiol Artif Intell. 2024;6(4):e240300. doi:10.1148/ryai.240300
- Kim HE, Kim HH, Han BK, et al. Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study. Lancet Digit Health. 2020;2:e138-e148. doi:10.1016/S2589-7500(20)30003-0
- Chang YW, Ryu JK, An JK, et al. Artificial intelligence for breast cancer screening in mammography (AI-STREAM): preliminary analysis of a prospective multicenter cohort study. Nat Commun. 2025;16:2248. doi:10.1038/s41467-025-57469-3 (ClinicalTrials.gov NCT05024591)
- Elías-Cabot E, Romero-Martín S, Raya-Povedano JL, Rodríguez-Ruiz A, Álvarez-Benito M. AI-based triage and decision support in mammography and digital tomosynthesis for breast cancer screening: a paired, noninferiority trial. Nat Med. 2026;32:1296-1305. doi:10.1038/s41591-026-04277-x
- Lång K, Josefsson V, Larsson AM, et al. MASAI clinical safety analysis. Lancet Oncol. 2023;24:936-944. doi:10.1016/S1470-2045(23)00298-X
- Hernström V, et al. MASAI screening performance. Lancet Digit Health. 2025;7:e175-e183. doi:10.1016/S2589-7500(24)00267-X
- Gommers J, Hernström V, Josefsson V, et al. Interval cancer, sensitivity, and specificity in the MASAI study. Lancet. 2026;407:505-514. Read the article (ClinicalTrials.gov NCT04838756)
- Sounderajah V, et al. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nat Med. 2025;31(10):3283-3289.
Related Articles



Lets get in touch!
Learn more about how Medicai can help you strengthen your practice and improve your patients’ experience. Ready to start your Journey?
Book A Free Demo