Which Handwritten OCR Model Should You Train on Your Forms?

Gemini Flash ranks first of 72 on out-of-the-box accuracy in our ranking, and it still finishes sixth overall. TrOCR sits 23rd before anyone trains it and climbs to second once training on your own data counts. olmOCR 2 climbs further still, from 41st to tenth. Both cloud-only services in the top ten, Gemini Flash and Transkribus, rank first and fifth out of the box, then drop to sixth and ninth once trainability and forms handling count.

That reversal is the decision in front of anyone who owns a forms backlog. You can call a frontier model’s API and accept its accuracy as it ships, or you can train an open recogniser on your own forms and run it on hardware you control. A zero-shot leaderboard speaks to the first path and says little about the second.

This page is for the CTO or VP Engineering commissioning the build, and for the operations head who owns the backlog, from claim forms in banking and insurance to patient intake sheets in healthcare. It explains how we  scored 72 candidates from 477 graded sources, and it shows the top ten along with the four gaps the published evidence leaves open for forms like yours. Everything else sits behind a short form at the end.

Once fine-tuning counts, the top four candidates finish within four points of each other. At that spread your forms pick the winner, mostly through where the handwriting sits on the page and where the model has to run.

Should you train an open model or call a frontier API for handwritten forms?

Call a frontier API when your forms can leave your network and you have no labelled handwriting, and train an open model when the data must stay in-house. Accuracy that has to improve on your own writers and layouts also points to training. In our ranking, eight of the top ten candidates run on your own servers, and all ten can be trained in some form.

Gemini Flash makes the strongest case for the API route, since it ranks first of 72 on out-of-the-box accuracy. Its tuning runs only on Google’s Gemini Enterprise Agent Platform, formerly Vertex AI, and only for some Flash versions, while the sources we graded hold no published before-and-after result for it. That missing evidence costs it places once trainability counts. Transkribus follows the same path from fifth out of the box to ninth overall, with training available only inside its own platform, on free and paid plans.

The open side starts with Qwen3-VL, first overall on 80.6 and released under Apache 2.0. PyLaia sits fourth and runs inference on a CPU under MIT. The same split runs through any build versus buy decision for AI, and the licence column deserves a look before the accuracy column does.

Handwriting recognition is one stage of a forms system. The stages around it, from intake and classification to printed fields and validation, belong to document AI and intelligent OCR, and our pillar on document AI for handwritten forms maps how they connect.

If your forms can’t leave your network, shortlist from the eight open candidates and keep the two cloud services as a baseline to beat.

How did we score 72 handwritten OCR models?

We graded 477 sources on 72 candidates, drawing on papers, benchmark results, model cards and vendor documentation, plus fine-tuning reports from practitioners. Accuracy measured by an independent party counts for more than accuracy a model’s makers report about their own model, so a self-reported result can lift a candidate only so far. Results measured at different levels stay separate in the score. A character error rate (CER) on cropped lines, the usual way results on the IAM handwriting benchmark are reported, measures something different from a page-level result. Neither one tells you whether a date field on a form came out right, so each level counts as its own kind of evidence.

Two more factors go into the score, trainability and forms handling. Trainability asks whether you can fine-tune the model on your own data and whether anyone has published a before-and-after result. Forms handling asks whether the model can find handwriting scattered across a page by itself, or whether something has to crop each line before the model ever sees it. Licence and deployment appear in every row so you can filter on them. The zero-shot rank orders candidates by out-of-the-box accuracy alone, which is why it disagrees so often with the overall rank. The full 72-model ranking shows the evidence behind every row, and you can read any accuracy claim on it, ours included, by asking who measured it and at which level.

Which handwritten OCR models rank highest once you can fine-tune?

Qwen3-VL leads our ranking on 80.6, followed by TrOCR on 80.0, PaddleOCR on 79.0 and PyLaia on 77.0. The next six candidates sit five points or more below fourth place, from Qwen2.5-VL on 72.0 down to olmOCR 2 on 65.9.

Top 10 of 72 handwritten OCR candidates, as of October 2026

Scores weigh graded accuracy evidence first, then add trainability and forms handling, while zero-shot rank orders all 72 candidates by out-of-the-box accuracy alone. VLM means vision-language model, HTR means handwritten text recognition, and the full ranking holds the other 62 rows.

Why do four models finish within four points of each other?

Each of the four leads on a different strength, so their scores converge while their designs stay far apart. Qwen3-VL reads whole pages, TrOCR has a published training result on historical handwriting, PaddleOCR finds handwriting by itself, and PyLaia runs on a CPU.

Qwen3-VL, 80.6

This vision-language model reads a whole page or region in one pass, so nothing has to crop lines for it. It fine-tunes with low-rank adaptation (LoRA) or its quantized variant QLoRA as well as full training, and a public walkthrough fine-tunes the 8B with LoRA. Our ranking lists the 8B running on one 24 GB GPU under Apache 2.0. The Arabic fine-tune behind its table row, Mubsir OCR, was scored on lines that also sat in its training data, so the size of its training gain is still open. In the sources we graded, its boxes were unreliable, which matters when a reviewer’s screen or an audit trail needs coordinates and matters less when you only need the text back.

TrOCR, 80.0

Microsoft’s transformer recogniser reads one cropped line at a time and supports full fine-tuning, with the base model under MIT. It reached a CER of 1.93 after training on 3,603 lines of Latin manuscripts. Out of the box it sits 23rd, so training earns most of its place, and something else has to find and crop every line before TrOCR reads it.

PaddleOCR, 79.0

The PP-OCRv5 and PP-OCRv6 toolkit ships a detector and a recogniser you can retrain separately, under Apache 2.0 and light enough for edge devices. Its default PP-OCRv6 detector, the medium tier, scored 84.0 (Hmean) on handwritten English in PaddlePaddle’s in-house benchmark. That makes it the only clear yes in the table’s field-finding column. The fine-tuning result in the table, 37.5 to 50.0 percent exact match, came from Han-Nom character lines that the paper’s method section describes as synthetic, so it says little about handwriting.

PyLaia, 77.0

This handwriting recogniser, maintained by Teklia, trains from a command line with an optional KenLM language model, and it runs inference on a CPU under MIT. Teklia’s own research reports lower error rates when PyLaia decodes with a language model, which can help on fields that follow known patterns. Like TrOCR, it needs a segmenter to hand it lines.

Those four profiles reduce to two pipeline designs. A page-level vision-language model reads the form in one pass, while a detector-plus-recogniser pipeline finds each handwritten region first, a computer vision task, and hands it to a line recogniser such as TrOCR or PyLaia. On forms where handwriting lands wherever the writer found room, I’d test the detector stage first. Only one of the four leaders has a published detection result, while every recogniser in the table can be retrained on your own lines. For a scanning station or a branch server, edge AI deployment pulls PaddleOCR and PyLaia forward, since both run on modest hardware.

Shortlist one page-level model and one detector-plus-recogniser pipeline, then let a test on a sample of your own forms settle the tie between them.

How much does fine-tuning improve handwritten OCR accuracy?

Training helps in every result we could trace, though the evidence for how much is thin. The traced results come from historical manuscripts, Arabic handwriting, synthetic text and single Devanagari words, and none of them comes from a form. The two cleanest are below, so read them as evidence that training works on handwriting and keep them out of model-to-model comparisons.

• 1.93 CER for TrOCR-base after training on 3,603 lines of Latin manuscripts, scored on the validation split (Meoded, arXiv).
• 3.27 to 1.65 CER for Calamari as fine-tuning on medieval manuscripts grew from 2 pages to 32, starting from a pretrained model (Reul and colleagues, arXiv).

A third-party LoRA on olmOCR 2 reached a self-reported CER of 0.174 on single Devanagari words, about 17 wrong characters in every 100 (Enclave Labs model card). Its base model had written English descriptions instead of transcribing the script. The Arabic results behind the two Qwen rows, from Mubsir OCR and Ketaba-OCR, were scored on lines that also sat in training, so they show the models adapting without telling you by how much. Kohút and Hradiš found that fine-tuning helps even with small sets of lines, and that active learning cuts the labelling you need, which matters for a forms team labelling its first sample.

Budget for a labelled sample of your own forms, because every published gain above came from someone else’s handwriting.

What can't public handwriting benchmarks tell you about your forms?

The training results behind the top ten come from manuscripts, Arabic handwriting, synthetic text and single words, while results on the IAM handwriting benchmark are usually reported on cropped lines. None of that tells you how a model behaves on your forms. Four gaps sit between those results and a working forms system, so we turned them into four tests for your own sample and called the set the Four-Gap Forms Test.

Test 1 · Does the pipeline find every handwritten field?

Handwritten field detection is the thinnest column in the table. One of the ten rows carries a clear yes, PaddleOCR’s self-reported 84.0 on handwritten English. The table marks the Qwen models’ boxes as unreliable and Gemini Flash’s as approximate, and the line recognisers need a segmenter. Mark every handwritten region on your sample, then count the fields each pipeline misses. Do this before you look at a single character error rate.

Test 2 · Does training on your forms move accuracy?

Apart from the two Qwen rows, which share one Arabic dataset, the training results we traced each come from a different dataset. Gemini Flash and InternVL have no fine-tuning data at all. Train every shortlisted model on the same labelled split of your forms, then score all of them on the same held-out forms so the comparison stays fair.

Test 3 · Are whole fields right, or only most of the characters?

Seven of the ten rows report training gains as character error rate, and one reports exact match on synthetic lines. A forms system succeeds or fails field by field, because one wrong digit in an account number makes the whole value wrong while barely moving the CER. Score exact match per field, and log which fields you would route to a reviewer.

Test 4 · Does it cope where print and handwriting share a field?

Handwriting on forms crosses printed box lines and runs into pre-filled text. None of the training results we traced was measured on fields where print and handwriting overlap, so put those fields in your sample on purpose and score them separately.

These four tests fit inside a fixed-scope proof of concept, run before any production commitment, and each one can remove a candidate that the leaderboard kept.

When does this ranking point you the wrong way?

The ranking assumes you can label data and run open models under their licences. Five conditions break that assumption, and in each one a different candidate, or a different kind of tool, should lead your shortlist.

• You can’t label any of your own data.
  Zero-shot rank then matters more than the overall score, which puts Gemini Flash (first) and Transkribus (fifth) at the front, provided your forms may leave your network.
• Your forms are not allowed to leave your network.
  Both cloud services drop out whatever their accuracy, and the eight open candidates become your whole field.
• You plan to ship the model inside a product.
  Calamari ships under GPL 3.0, and two Qwen2.5-VL sizes carry Qwen’s own licences, with the 3B limited to research use. Clear those terms with counsel before you build, since the Apache 2.0 and MIT licences elsewhere in the top ten carry different obligations.
• Your forms are mostly print with a few handwritten marks.
  A parser tuned for printed fields may serve you better there, with only the handwritten fields routed to one of the four leaders.
• A vision-language model adds text that was never on the form.
  The authors of OmniHandwritingOCR, a diagnostic handwriting benchmark, found that several generative models “hallucinate plausible but visually unsupported corrections”. Compare each output with the length and format the field expects, and send mismatches to a reviewer.

If you bring in a partner for the test, our guide to choosing an AI development company lists the questions to settle before you share a single form.

What should you do with your shortlist next?

Take two designs from the top four into a test on your own forms, one page-level model such as Qwen3-VL and one PaddleOCR-led pipeline with TrOCR or PyLaia. Run the Four-Gap Forms Test on both. It shows whether your forms need a detector, how much labelling buys you, which hardware the winner needs and which fields still need a reviewer. Those four answers decide the build, and only your own forms can supply them. The wider pipeline around recognition sits in our pillar on document AI for handwritten forms, and when you want every row and every source behind this page, get the full 72-model ranking.