Quality
955 Tests, Zero Flaky Failures: Building Reliable AI Systems | AI PM Portfolio 955 Tests, Zero Flaky Failures: Building Reliable AI Systems August 15, 2025 · 15 min read · Engineering Deep Dive We run 955 tests on every push with a zero-flaky-failure rate. The secret is not discipline -- it is architecture.
Quality
Building a Self-Learning QA Loop: When Your Product Tests Itself | AI PM Portfolio Building a Self-Learning QA Loop: When Your Product Tests Itself March 10, 2025 · 16 min read · Engineering Deep Dive A self-learning QA loop is a testing system that automatically generates new tests from past bugs, validates system
AI Engineering
The AI Evaluation Playbook: Measuring What Matters in Production LLM Systems | AI PM Portfolio The AI Evaluation Playbook: Measuring What Matters in Production LLM Systems May 15, 2024 · 18 min read · Definitive Guide Most AI evaluation is done on benchmarks that do not reflect production reality. After 18 months of
AI Engineering
Building an AI Extraction Platform: 4 Providers, 60 Analyzers, 510 Tests | AI PM Portfolio Building an AI Extraction Platform: 4 Providers, 60 Analyzers, 510 Tests February 22, 2024 · 15 min read · Platform Architecture Case Study We built a document extraction platform that uses 4 AI providers, 60 specialized analyzers, and
AI Engineering
The 10% Accuracy Improvement That Saved Millions | AI PM Portfolio The 10% Accuracy Improvement That Saved Millions September 30, 2022 · 15 min read · Case Study / Deep-Dive A 10% accuracy improvement on 50,000 AI-processed tax returns does not sound dramatic until you calculate the downstream impact: approximately 5,000 fewer
AI Engineering
50,000 AI-Processed Tax Returns: What We Measured and What Surprised Us | AI PM Portfolio 50,000 AI-Processed Tax Returns: What We Measured and What Surprised Us August 22, 2022 · 16 min read · Case Study / Metrics Deep-Dive After processing 50,000 tax returns with AI at a national tax services
AI Engineering
The Human-in-the-Loop Imperative: Why Our AI Tax System Needed 3 Layers of Review | AI PM Portfolio The Human-in-the-Loop Imperative: Why Our AI Tax System Needed 3 Layers of Review March 15, 2022 · 15 min read · Case Study / Framework When building AI systems that make consequential decisions, human-in-the-loop review is not