Case study · Healthcare research
Automating systematic reviews without losing the rigor
A medical evidence synthesis team was extracting outcomes from research papers by hand. Brainy Neurals built an AI paper analyzer to take on that load. The system extracts each outcome with its statistics and runs ten bias and certainty checks before a researcher approves it.
Nandni Barot, Brainy Neurals.
Reviewed by Mitesh Patel, Founder & Director and NVIDIA Certified AI Architect.
Published August 2026 · 11 min read
01
The engagement at a glance
What problem did this solve?
Manual systematic reviews forced researchers to pull outcomes, statistics and quality judgments from every paper by hand. Cycles ran long, and conclusions varied between reviewers.
What did Brainy Neurals build?
Brainy Neurals built an AI research paper analyzer that extracts text, tables and figures from medical PDFs. The system appraises evidence quality and drafts reports researchers approve in stages.
What changed after it went live?
Reviewers now go from raw PDF to structured, appraised evidence in one pipeline. Every outcome carries a human approval decision before it enters a report.
Who else could use this?
Any team that appraises dense technical documents against a fixed checklist can reuse the pattern. Legal, regulatory, patent, financial research and policy teams all face the same extraction problem.
02
Why manual evidence reviews kept stalling
Systematic reviews are how medicine decides what works, and they crawl. Across 195 registered reviews, the mean time from registration to publication was 67 weeks (Borah et al., BMJ Open 2017, DOI: 10.1136/bmjopen-2016-012545).
A medical evidence synthesis organization came to us with the reading problem, not the searching problem. Their reviewers were doing intelligent document processing by hand, pulling outcomes, statistics and quality judgments out of dense PDFs.
Paper after paper.
And the cost was not just hours. Divergent judgments meant the same trial could read as strong evidence to one reviewer and weak to another.
Several hours per paper was normal, and nobody had ever counted the running total.
Where it broke, in their words:
- Outcomes scattered across text, tables and figures
- Two reviewers, two different quality judgments
- Findings buried in personal spreadsheets
- Every new question meant rereading old papers
- Batch reviews stalled on coordination
03
What do review teams usually try first?
Most review teams try three familiar things before commissioning anything custom.
| Approach | Strength | Limit | Best for |
|---|---|---|---|
| Add more reviewers | Moves the backlog | Costs multiply, judgments drift | One-off funded reviews |
| Screening tools | Fast title and abstract triage | Full-text extraction stays manual | Search and screening stages |
| A general chatbot | Quick single-paper summaries | No structure, no audit trail | Orientation, not evidence work |
| A gated extraction pipeline | Structured outcomes, human-approved appraisal | Needs a build and workflow change | Teams reviewing continuously |
A 2025 comparison of AI extraction tools against human reviewers logged confabulations, values the papers never reported (Helms Andersen et al., DOI: 10.1002/cesm.70036). Their conclusion matched our design. Keep a human deciding.
04
How did we design the analyzer?
The analyzer is a staged pipeline, not a chat window: a paper goes in as a PDF. Structured, appraised evidence comes out the other end, with a researcher’s decision recorded at the gate that matters.
Two decisions shaped everything.
First, we rejected a single-pass summarizer, the obvious build. Clinical claims need provenance, so each stage writes typed, checkable fields that the next stage can verify. The model proposes, the researcher disposes, and nothing reaches the report without a human decision on record.
Second, we rejected spreadsheet-only output: analyzed papers land in a knowledge graph and a retrieval index instead. That lets retrieval augmented generation answer questions across every study the team has processed.
Fragmented findings were half the original pain, and flat spreadsheet files would simply have recreated them.
Brainy Neurals built the research paper analyzer for a medical evidence synthesis organization. The boundaries are deliberate: the system drafts, structures and remembers what it has seen.
People decide.
The model proposes, the researcher disposes, and nothing reaches the report without a human decision on record.
05
The technology stack we used
Every layer in the stack follows one rule: structure first, generation second. Large language models do the language work, and our LLM development choices keep it honest. Two appraisal standards are built into the flow: risk of bias and GRADE, the grading framework for evidence certainty.
| Layer | What we used | Why | Ruled out |
|---|---|---|---|
| 01Document parsing | Splits each PDF into text, tables, figures | Signal lives in all three | Flattening pages to text |
| 02Table handling | Structure-first rebuild, clean comma-separated output | Merged cells break naive extraction | Treating tables as text |
| 03Language models | The reading, drafting and summarizing engine | Best comprehension available at scale | Training from scratch |
| 04Evidence frameworks | Risk-of-bias and GRADE as staged checks | Consistency needs a checklist | One generic score |
| 05Knowledge layer | Graph database linking papers, outcomes, evidence | Cross-paper questions need relationships | Spreadsheet silos |
| 06Retrieval and chat | Vector retrieval behind a chat interface | Answers ground in approved analysis | Chatting raw PDFs |
| 07Review application | Approval queues and report export | The human gate is the product | Unreviewed automatic exports |
06
How does one paper move through?
Follow one paper through the pipeline below, from upload all the way to answerable evidence.
- 1A researcher uploads one PDF or a zipped batch, and the analyzer splits every paper into text, tables and figures.
- 2The analyzer proposes the clinical outcomes it finds in each paper, and the researcher approves or rejects each one.
- 3For every approved outcome, the system extracts effect sizes, confidence intervals, p-values and study arm comparisons.
- 4Staged checks then appraise five bias domains and five certainty factors, each judgment tied to quoted evidence.
- 5The system drafts a recommendation and a narrative summary, grounded only in what was extracted and approved.
- 6Results export to a spreadsheet report, load into the knowledge graph, and become answerable through chat.
That’s the whole loop: six repeatable steps, and one gate where a person decides. Batches follow the same six steps, with the approval queue collecting every outcome in one place.
Want this walked through for your setup?
07
The three problems that nearly stopped us
01
Misbehaving tables
Clinical tables refused to behave: they span pages, merge cells and nest subgroups, and our early extraction quietly mangled them.
Columns slid, and subgroup rows attached to the wrong study arms. We weren’t alone: a 2025 evaluation of an AI review platform found extraction weakest on tables (Cassell et al., DOI: 10.3389/frai.2025.1662202).
02
Fluent nonsense
Ask a language model for a bias judgment and it answers, whether or not the paper reported the data. Early drafts asserted statistics that did not exist, fluently and with confidence. Nothing looked wrong.
03
Outcome sprawl
And papers name the same outcome five different ways. Our first pass flagged every variant separately, and reviewers drowned in near-duplicate approval requests. The specialist engineers on the build spent two sprints on that queue alone.
The longest table in our test set ran four pages, and we still talk about it.
08
How we fixed each one
Because each failure was specific, each fix could be specific too.
FIX 01
A pipeline just for tables
Tables got their own pipeline: the system detects structure first, rebuilds the grid, and exports clean comma-separated files. Row and column counts are verified against the source before any model reads a cell.
FIX 02
Evidence or silence
Fluent nonsense met an evidence-or-silence rule: every judgment must cite a quoted span from the paper it came from. Missing statistics route to a synthesis-without-meta-analysis path and get labeled honestly, not guessed.
FIX 03
Cluster, then approve once
Outcome sprawl got a clustering layer that groups near-duplicate outcomes into one card, so a reviewer approves the cluster once.
None of these fixes was exotic, just sequencing, verification and honest labels shipped across two releases. That phased rhythm is exactly what our engagement models are built around.
09
What changed after go-live?
Here is what we can print about the results, and what we will not.
Before, one paper meant hours of reading and a private spreadsheet nobody else could query.
Now one upload returns extracted outcomes, appraised evidence and a structured report from a single pipeline.
Every appraisal now covers the same ground: five bias domains and five certainty factors per outcome, ten checks in all.
Zero outcomes reach a report without human approval. Every outcome the analyzer proposes passes a researcher’s decision before appraisal begins, on every paper processed.
We have not published a timing benchmark, though the client reports shorter review cycles. A number we have not measured is a number we will not print.
What the structure buys is consistency: two reviewers now start from the same fields and the same ten checks.
Disagreement still happens, but now it happens about evidence, in one place, on the record.
Tell us what slows your reviews.
No sales sequence. One reply, from the person who would architect it.
10
What is running today
The analyzer runs in production today on the client’s active review work. Papers go in one at a time or as zipped batches of studies. The team now queries analyzed studies through chat instead of reopening the source PDFs.
A second phase is underway for the client’s wider life sciences workflow. It adds role-based dashboards, reviewer assignment, conflict-of-interest checks, outcome voting and PRISMA 2020 reporting, the standard format for systematic reviews.
Nothing about the human gate changed at handover, because it was the part the client asked us to keep.
11
What we would do differently
01
Build the approval queue before the extractor ever gets clever.
We built it the other way around, and reviewers spent those first two sprints buried in flags. Sequencing was the mistake, not ambition.
02
Encode the checklist, not the impression.
Bias appraisal only became consistent once every domain became its own staged check with quoted evidence attached to the call.
03
Treat tables as first-class documents.
Half the clinical signal lives in tables, and generic text extraction reads right past it. We learned that the expensive way, in re-runs nobody had budgeted for.
04
So keep the graph in from day one.
Retrofitting relationships between papers, outcomes and evidence is far harder than writing them at ingest, and cross-paper questions arrive early. The graph earned its keep early.
Half the clinical signal lives in tables, and generic text extraction reads right past it.
12
Where else this pattern fits
Automated evidence appraisal turns dense documents into structured, quality-rated findings through machine extraction plus human approval, wherever consistent judgment matters.
- Pharmacovigilance teams, who monitor drug safety literature, face the same shape: adverse event mentions instead of outcomes.
- Legal teams appraise contracts against playbooks, so the extraction schema swaps to clauses and obligations.
- In banking and finance, the documents become filings and analyst reports, and the checklist becomes risk criteria.
- Patent teams run prior-art appraisal, which is the same extract-then-judge loop applied to claims and citations.
- Policy units grade evidence for government decisions, often against published standards of certainty and quality.
- And construction bid teams appraise tender and standards documents the same way, clause by clause.
Porting means swapping the extraction schema and the appraisal checklist, while the pipeline, gates and graph carry over unchanged.
13
Questions buyers usually ask us
Method and trust
Can AI actually do a systematic review?
Parts of one, with supervision. AI handles extraction, appraisal drafts and summaries well, and current research still finds errors and outright confabulations. So this build keeps a researcher’s approval on every outcome: the review stays yours, and the grunt work doesn’t.
How accurate is AI data extraction from papers?
Published studies report strong overall accuracy and persistent failure cases, especially tables and review-specific fields. Our answer was architectural rather than statistical: verified table rebuilds, quoted evidence behind every judgment, and a human gate. We do not publish our own accuracy figures.
What happens when a paper lacks full statistics?
Papers without complete statistics route automatically to a synthesis-without-meta-analysis path instead. The outcome is appraised with what exists, labeled for what is missing, and nothing gets invented to fill the gap.
Do we need patient data to start?
No: the system analyzes only published literature, so no patient records or protected health information enter the pipeline. Your own prior reviews help calibration, but published papers are the raw material.
Time and cost
How long would a build like this take?
A scoped pilot on your own papers is a matter of weeks, not quarters. The full platform, with collaboration, dashboards and reporting, is a phased build over several months. An AI readiness assessment is the honest first step when the document mix is unclear.
What drives the cost of a system like this?
Three things: document variety, the depth of the appraisal frameworks, and how many systems it must talk to. A narrow pilot keeps all three small, which is why we price the pilot first and the platform after. Integration is usually the quiet multiplier, because review teams already live in reference managers and reporting tools.
Your reviews do not have to crawl.
14
The services this was built from
Six capabilities carried this build, and each is a Brainy Neurals service on its own.
Document AI
The parsing and extraction layer that turned dense medical PDFs into structured, analyzable data.
RAG development
The retrieval layer that answers researchers’ questions from their own analyzed papers, with grounding.
Generative AI applications
The large language model work behind outcome finding, appraisal drafts and narrative summaries.
Conversational AI
The chat interface researchers use to explore outcomes, evidence and relationships across papers.
AI proof of concept
How this engagement started: a narrow pilot that proved the pipeline before the platform.
AI in healthcare
Where this case sits among our clinical and life sciences delivery work.
If you are earlier than a pilot, start with AI consulting or an AI readiness assessment. The industries hub shows who we help.
Barot, N. (2026). Automating systematic reviews without losing the rigor. Brainy Neurals.
https://brainyneurals.com/case-studies/ai-systematic-review-automation/
Published
August 2026.