LLMs Changed Everything We Thought We Knew About Document Processing | AI PM Portfolio

LLMs Changed Everything We Thought We Knew About Document Processing

April 10, 2023 · 15 min read · Technical Deep Dive

For years, document processing followed a rigid pipeline: OCR to extract text, regex to find patterns, rules to validate, and manual review to catch the rest. Then LLMs arrived, and we could skip half the steps. But the transition was not as simple as "just use GPT-4." At a YC-backed tax-tech startup, we rebuilt our document processing architecture around LLMs and discovered what they actually replaced, what they made worse, and the hybrid approach that outperformed both pure-traditional and pure-LLM systems by a wide margin.

What did document processing look like before LLMs?

Before LLMs became practically usable in production -- which for us meant January 2023 when GPT-4 access became reliable -- our document processing pipeline was a classic five-stage architecture that had barely changed in a decade. According to a 2022 IDC survey, 83% of enterprise document processing systems used some variant of this same pipeline.

The pre-LLM pipeline worked like this:

1. Ingestion: Accept uploaded documents (PDF, image, scan). Normalize format, resolution, and orientation.
2. OCR: Extract raw text from images using optical character recognition. Handle multi-column layouts, tables, and handwritten text separately.
3. Pattern matching: Apply regex patterns and keyword rules to identify document type and locate specific fields (e.g., "Box 1" on a W-2).
4. Validation: Cross-reference extracted values against known constraints (e.g., state tax withheld should not exceed federal tax withheld for standard cases).
5. Manual review: Route low-confidence extractions to human reviewers who corrected errors and confirmed ambiguous values.

This pipeline worked. Over 18 months, we achieved 91.4% end-to-end accuracy across 37 document types. But it had three fundamental problems that no amount of optimization could solve.

First, it was brittle. Every new document type required writing new regex patterns, new field-location rules, and new validation logic. Adding support for a single new form type took 2-3 weeks of engineering time. According to Gartner's 2022 document processing report, the average enterprise spends 340 engineering hours per year maintaining regex-based extraction rules. We were spending closer to 500 hours because financial documents have more variants than typical business documents.

Second, it failed catastrophically on non-standard documents. When a user uploaded a W-2 from an unusual payroll provider with a non-standard layout, the regex patterns found nothing. The entire extraction returned blank. Zero partial credit. According to our logs, 8.7% of documents failed in this total-failure mode pre-LLM.

Third, it could not reason. When Box 1 on a W-2 was partially obscured and the OCR read "$4,523" but Box 2 (federal tax withheld) read "$12,847," a human would immediately notice that the withholding exceeds the reported wages and infer that Box 1 is probably "$54,523" or "$45,230." The pre-LLM pipeline could flag the inconsistency but could not make the inference. [LINK:post-18]

How did the architecture change with LLMs?

Here is the before-and-after comparison of our document processing architecture:

The net effect: our end-to-end accuracy went from 91.4% to 98.7%, our manual review rate dropped from 12% to 4.1%, and adding support for a new document type went from 2-3 weeks to 2-3 hours. According to a 2023 Forrester analysis of AI document processing platforms, the industry average accuracy improvement from LLM integration was 4-7 percentage points. Our 7.3-point improvement was at the high end, largely because our pre-LLM baseline was already strong.

What did LLMs make obsolete in document processing?

Not everything changed. But three capabilities that consumed enormous engineering effort became largely unnecessary:

• Per-document-type regex libraries: We had maintained 847 regex patterns across 37 document types. With LLM-based extraction, we retired 680 of them (80%). The remaining 167 handle deterministic cases where regex is faster and more reliable than an LLM call. According to our measurements, regex extraction takes 0.3ms per field versus 200ms per field for LLM extraction. For fields with perfectly predictable formats (like EIN numbers), regex wins.
• Layout-specific parsing rules: Pre-LLM, we wrote custom parsers for every W-2 layout variant we encountered. We had 23 different W-2 parsers. With GPT-4 Vision, we send the image and a schema describing what fields to extract. The LLM handles layout variation naturally. We went from 23 parsers to one prompt.
• Low-confidence fallback workflows: Previously, any extraction with confidence below 0.7 went to human review. With LLM-based error correction, the system attempts to resolve low-confidence extractions by re-examining the document with a different prompt strategy. This resolved 62% of previously-human-bound cases automatically. According to our cost analysis, this saved approximately $0.47 per document in human review costs, totaling $8,200 over the season.

What are LLMs still bad at in document processing?

The hype around LLMs in 2023 made it tempting to go all-in. We did not, and the restraint paid off. Four areas where LLMs consistently underperformed traditional approaches:

1. Mathematical validation: LLMs are notoriously unreliable at arithmetic. When we need to verify that Box 1 minus Box 12a equals the expected value, we use a deterministic calculator. In our testing, GPT-4 made arithmetic errors on 3.2% of validation checks. Our rule-based validator made zero arithmetic errors. According to a 2023 Stanford study on LLM mathematical reasoning, GPT-4 achieves only 92% accuracy on multi-step arithmetic involving numbers with more than 4 digits. For financial documents where every digit matters, 92% is unacceptable.
2. Exact number transcription: OCR reads "$54,231.87" character by character. An LLM processing the same image might round, truncate, or transpose digits. We found that GPT-4 Vision transposed digits in 1.8% of numeric field extractions -- a rate that would be invisible in most applications but catastrophic for tax returns. We always use OCR for numeric extraction and LLMs for semantic extraction.
3. Deterministic consistency: The same document sent to GPT-4 twice might produce slightly different extractions. For a tax platform, deterministic behavior is non-negotiable. A user who uploads the same document twice and gets different results will immediately lose trust. We solved this by using temperature=0, structured output schemas, and caching, but the fundamental non-determinism of LLMs requires active mitigation.
4. Processing speed for simple documents: For a standard, cleanly-scanned W-2, our regex-based pipeline extracted all fields in 12ms. The LLM-based pipeline took 1,800ms. For the 75% of documents that are straightforward, the traditional pipeline is 150x faster. Speed matters because users upload documents in batches, and batch processing at 1.8 seconds per document creates noticeable wait times. [LINK:post-16]

How does the hybrid approach work in practice?

The architecture that outperformed both pure-traditional and pure-LLM approaches was what we called "traditional-first, LLM-escalation." The principle was simple: use the fastest, cheapest, most reliable method first. Escalate to more powerful (and expensive) methods only when simpler methods fail.

Here is the decision logic for each document:

1. Attempt structural classification and OCR extraction. If all fields are extracted with confidence above 0.85 and pass validation rules: done. Cost: $0.001. Latency: 160ms. This path handles 58% of documents.
2. If OCR extraction fails on any field: Send the failed fields (not the entire document) to GPT-4 for targeted extraction. Cost: $0.008. Latency: 400ms. This path handles 24% of documents.
3. If the document type is unrecognized or the layout is non-standard: Send the full document to GPT-4 Vision with a comprehensive extraction prompt. Cost: $0.025. Latency: 2,100ms. This path handles 14% of documents.
4. If the LLM extraction produces results that fail validation: Route to human review with the LLM's extraction as a pre-filled draft. Cost: $1.20 (human time). Latency: varies. This path handles 4.1% of documents.

The weighted average cost across all paths was $0.043 per document. A pure-LLM approach would have cost $0.025 per document but with lower accuracy on standard forms. A pure-traditional approach would have cost $0.003 per document but with a 12% human review rate that added $0.14 per document in human costs. The hybrid approach found the optimal balance.

Key insight: The hybrid approach is not a compromise between old and new. It is architecturally superior to either approach alone because it routes each document to the method best suited for that specific document's characteristics. According to a 2023 MIT CSAIL paper on hybrid AI architectures, systems that combine rule-based and neural approaches outperform pure neural systems by 8-15% on structured document tasks while using 60% fewer compute resources.

What does this mean for the future of document processing?

After rebuilding our pipeline around LLMs, I believe the industry is heading toward three convergence points:

• Multimodal models will replace OCR for complex documents. GPT-4 Vision already outperforms traditional OCR on handwritten text, rotated images, and low-quality scans. As multimodal models get faster and cheaper, the OCR step will shrink to handling only the simplest, highest-volume cases. According to OpenAI's published benchmarks, GPT-4 Vision achieves 94% accuracy on handwritten text versus 71% for Tesseract OCR.
• Schema-driven extraction will replace template-driven extraction. Instead of building a template for each document type, you will define a schema (field names, types, constraints) and let the model figure out where those fields appear. This makes document processing systems dramatically more adaptable. [LINK:post-20]
• Human review will shift from correction to validation. Pre-LLM, human reviewers filled in missing fields. Post-LLM, human reviewers confirm that auto-filled fields are correct. The cognitive task changed from production to quality assurance, and it requires different skills and different interfaces.

The biggest mistake I see teams making right now is treating the LLM transition as binary: either you go all-in on LLMs or you stick with the traditional pipeline. The right answer is the hybrid architecture that uses each tool where it excels. That is not a temporary compromise. It is the long-term architecture. Even as LLMs get faster and cheaper, there will always be cases where a deterministic rule is more reliable than a probabilistic model. The art is in knowing which cases fall where.

Frequently Asked Questions

How much does it cost to process documents with LLMs versus traditional OCR?

In our system, the per-document cost breakdown was: pure OCR/regex at $0.001-0.003, targeted LLM extraction (single field) at $0.005-0.01, full LLM extraction at $0.02-0.04, and human review at $1.00-1.50. The hybrid approach averaged $0.043 per document across all paths. The critical variable is your human review rate. If LLMs reduce your human review rate by 8 percentage points (from 12% to 4%), the LLM cost is more than offset by human review savings. According to our ROI analysis, the breakeven point was at a 3-point reduction in human review rate.

Can you use open-source LLMs instead of GPT-4 for document processing?

We tested Llama 2 (70B) and Mistral for extraction tasks in March 2023. Accuracy was 6-11 percentage points lower than GPT-4 on our benchmark. The gap was largest on non-standard document layouts and handwritten text. For classification (a simpler task), the gap narrowed to 2-4 points. Our recommendation: use GPT-4 (or Claude) for extraction where accuracy matters, and consider open-source models for classification where cost savings justify the accuracy tradeoff. As of April 2023, the accuracy gap is closing fast.

How do you handle PII and sensitive data when sending documents to LLM APIs?

This was a non-negotiable architectural constraint. Tax documents contain Social Security numbers, bank account numbers, and income data. We implemented three safeguards: first, PII fields were redacted before sending to external APIs and reconstructed from OCR output afterward. Second, we used API providers with SOC 2 compliance, data processing agreements, and zero-retention policies. Third, we built an on-premise fallback path for the most sensitive document types that never left our infrastructure. According to a 2023 IAPP survey, 61% of organizations cite PII handling as their top concern with LLM adoption. It should be.

What is the minimum viable team to build an LLM-powered document processing system?

We built ours with three engineers and one product manager over four months. The breakdown: one backend engineer focused on the pipeline architecture and API integration, one ML engineer focused on the classification model and confidence calibration, one full-stack engineer focused on the review interface and feedback loop, and me as PM defining requirements, managing the training data, and running accuracy benchmarks. A 2023 survey by Scale AI found that the median team size for production document AI systems was 4-6 people, which aligns with our experience.

Last updated: April 10, 2023