Measuring Automation ROI When the Baseline Is 'We Don't Know' | Dinesh Sanikommu

Last updated: July 18, 2021

Measuring Automation ROI When the Baseline Is "We Don't Know"

To measure automation ROI without baseline data, use a three-phase approach: first, instrument the manual process for 2-4 weeks to establish proxy baselines; second, run automation in shadow mode alongside manual processes to build a comparison dataset; third, measure the delta after cutover. At an enterprise logistics platform, this framework helped us prove a 45% efficiency improvement even though nobody had measured the manual process before we arrived.

The hardest question in enterprise automation is not "what should we automate?" It is "how much better is the automated version?" That question is impossible to answer when nobody measured the manual process in the first place, which is almost always the case.

At an enterprise logistics platform serving 14 clients across 400+ cities, we automated dispatch, route planning, and SLA monitoring. The executive team wanted to know the ROI. The honest answer was: we do not know, because nobody tracked how long these tasks took manually. According to a 2020 McKinsey study, only 23% of companies have reliable baseline metrics for the processes they are trying to automate. The other 77% are guessing, or worse, making up numbers that sound good in a board presentation.

This is the framework I built to measure automation ROI from zero data. It is not perfect, but it produces defensible numbers that survive scrutiny from CFOs and board members.

Why Is Measuring Automation ROI So Difficult?

Four structural problems make automation measurement harder than it should be:

• The baseline paradox: The processes most worth automating are the ones that are most chaotic and least measured. If someone had already instrumented the process, they would have optimized it before you arrived.
• The human variability problem: Manual processes depend on who is doing them. One operations manager dispatches 40 orders per hour. Another does 25. The "baseline" depends entirely on which person and which shift you measure.
• The apples-to-oranges problem: Automation does not just do the same thing faster. It often changes the process itself. Automated dispatch considers 15 variables simultaneously; manual dispatch considered 4-5. How do you compare the output of two fundamentally different processes?
• The Hawthorne effect: When you tell people you are measuring their manual process as a baseline, they perform better than usual. Your baseline becomes artificially high, making your automation improvement look worse. A 2019 meta-analysis published in the Journal of Applied Psychology found that the Hawthorne effect inflates measured performance by 12-20% in workplace studies.

What Is the Three-Phase ROI Measurement Framework?

Here is the framework, phase by phase, with specific instructions for each:

Phase 1: Instrument the manual process (weeks 1-4)

Before building any automation, spend 2-4 weeks collecting data on the manual process. The goal is not perfect measurement. It is directional accuracy: enough data to establish reasonable proxy baselines.

1. Time-stamp key milestones. Do not try to measure everything. Pick 3-5 milestones in the process and record when they happen. For logistics dispatch: order received, dispatcher starts planning, route assigned, driver notified. The time between these milestones is your baseline.
2. Use system logs, not self-reports. People are terrible at estimating their own time. Instead of asking "how long does dispatch take?", look at email timestamps, system login times, and order status change logs. These are objective, even if incomplete. We found that system log timestamps were available for 70% of the milestones we wanted to track, even in processes that had never been formally measured.
3. Sample, do not census. You do not need to measure every order for four weeks. A representative sample of 100-200 orders across different days, shifts, and operators gives you a statistically defensible baseline. According to standard sampling methodology, a sample of 150 with 2+ weeks of variation provides a 95% confidence interval narrow enough for business decisions.
4. Document the outliers. Record the fastest and slowest instances, and why they were fast or slow. These outliers tell you what the process could achieve at its best (your automation target) and what goes wrong at its worst (your risk mitigation target).

Phase 2: Shadow mode (weeks 5-8)

Run the automation alongside the manual process without replacing it. Both systems process the same inputs. Only the manual output is used operationally. The automated output is logged for comparison.

1. Process the same orders through both paths. Every order that a human dispatcher plans should also be planned by the automated system in the background. This produces a direct comparison on identical inputs.
2. Compare on three dimensions: speed (how long each path takes), quality (route efficiency, SLA adherence, constraint satisfaction), and throughput (how many orders each path can handle per unit time).
3. Identify disagreements. When the automated system and the human dispatcher produce different outputs for the same input, investigate why. Sometimes the human is right (they know something the system does not). Sometimes the system is right (it considered variables the human overlooked). These disagreements are the most valuable data you will collect. In our case, the automated system produced better routes in 68% of disagreements and the human produced better routes in 32%, usually because of local knowledge the system lacked.
4. Build stakeholder confidence. Shadow mode is as much about organizational readiness as measurement. Operations managers who watch the automated system produce good outputs for 4 weeks are far more willing to trust it when you cut over. This is the adoption lesson from our predictive alerts experience applied proactively. [LINK:post-03]

Phase 3: Cutover and measurement (weeks 9-12+)

1. Phase the cutover. Do not switch everything at once. Start with one city, one shift, or one client. Measure the delta between automated performance and the Phase 1 baseline for that segment. Then expand.
2. Track the same milestones. Use exactly the same milestones from Phase 1. Order received to route assigned. Route assigned to driver notified. Same timestamps, same measurement methodology. This ensures the comparison is valid.
3. Measure for at least 4 weeks post-cutover. The first week will show artificially low performance (learning curve, configuration adjustments). Weeks 2-4 show steady-state performance. Use weeks 2-4 for your ROI calculation.
4. Calculate the composite ROI. Here is the formula we used:

Automation ROI = [(Baseline Cost - Automated Cost) + Revenue from Increased Throughput] / Total Automation Investment

Where:

• Baseline Cost = (Average manual time per order x hourly labor cost x order volume) from Phase 1
• Automated Cost = (Compute costs + monitoring costs + reduced-headcount labor cost) from Phase 3
• Revenue from Increased Throughput = additional orders processed that were previously constrained by manual capacity
• Total Automation Investment = engineering time + infrastructure + ongoing maintenance

How Do You Handle the Proxy Metrics Problem?

Sometimes you cannot measure the thing you actually care about, so you measure something correlated with it. Here is our proxy metrics map:

The key rule with proxy metrics: always disclose the proxy. When presenting to leadership, say "we measured order-to-route-assignment time as a proxy for dispatch efficiency, with a strong historical correlation." This is more credible than claiming you measured something you did not. A 2021 Deloitte survey found that 78% of executives prefer honest proxy metrics over precise-sounding numbers they cannot verify.

What Were the Actual Results?

After running this framework across our logistics platform:

• Phase 1 baseline: Average manual dispatch took 8.3 minutes per order. Best operators averaged 5.1 minutes. Worst averaged 14.2 minutes. Total manual throughput was constrained to approximately 180 orders per dispatcher per shift.
• Phase 2 shadow mode: Automated dispatch averaged 0.4 seconds per order (effectively instantaneous from the operator's perspective). Route quality was higher in 68% of cases. Constraint violations (wrong vehicle type, weight limit exceeded) dropped to near zero.
• Phase 3 cutover results: End-to-end dispatch cycle time decreased by 45%. Throughput per dispatcher increased by 3.2x because dispatchers shifted from planning to exception handling. SLA adherence improved by 6.2% as a secondary effect of better route quality.

The 45% efficiency improvement became the number we reported externally. But the number I am most proud of is the methodology. Because we had instrumented Phase 1, run shadow mode in Phase 2, and measured the same milestones in Phase 3, nobody could credibly challenge the result. The CFO signed off on the first review. [LINK:post-01]

What Are the Common Mistakes in Automation ROI Measurement?

Five mistakes I have seen teams make, including some I made myself before developing this framework:

1. Using estimates instead of measurements. "We think manual dispatch takes about 10 minutes" is not a baseline. It is a guess. And guesses always skew toward the number that makes the automation look better. Invest in Phase 1 measurement even if it delays the project by 3-4 weeks.
2. Measuring only speed. Speed is the easiest metric but rarely the most important. Quality (fewer errors), consistency (lower variance), and throughput (more volume) often contribute more to ROI than raw speed. Our 45% efficiency improvement was a composite of speed, quality, and throughput gains.
3. Ignoring the transition cost. The first month after cutover is always worse than steady state. Teams that measure ROI in week 1 get discouraged. Teams that wait until week 4-6 see the real picture. According to change management research, organizational performance typically dips 15-20% during the first 2-3 weeks of a major process change before recovering and exceeding the baseline.
4. Forgetting maintenance costs. Automation is not a one-time investment. It requires monitoring, updates, and engineering support. Include at least 12 months of projected maintenance costs in your ROI calculation, or your number will not survive annual budget reviews.
5. Comparing peak manual to average automated. Honest comparison means average-to-average or median-to-median. Cherry-picking the worst manual performance and comparing it to average automated performance produces impressive but indefensible numbers. [LINK:post-04]

How Do You Present Automation ROI to Different Audiences?

The same 45% number needs different framing for different stakeholders:

• For the CFO: Lead with cost reduction and payback period. "Automation reduced dispatch cost per order by 45%, yielding a payback period of 7 months on the total investment."
• For the COO: Lead with throughput and quality. "Automation increased dispatch throughput by 3.2x and reduced constraint violations to near zero, enabling us to scale operations without proportional headcount growth."
• For the CTO: Lead with methodology and architecture. "We instrumented the baseline over 4 weeks, ran shadow mode for 4 weeks, and measured the same milestones post-cutover. The architecture supports horizontal scaling across all client instances." [LINK:post-02]
• For the board: Lead with the strategic implication. "Automated dispatch is now a platform capability that every new client gets on day one, reducing our marginal cost of onboarding and increasing our gross margin per client."

Frequently Asked Questions

How long does the full three-phase measurement process take?

Typically 10-14 weeks: 2-4 weeks for Phase 1 (baseline instrumentation), 3-4 weeks for Phase 2 (shadow mode), and 4-6 weeks for Phase 3 (cutover and steady-state measurement). This adds time to the project, but it produces ROI numbers that withstand scrutiny. The alternative is shipping automation with no defensible ROI, which makes every future budget request harder.

What if stakeholders are not willing to wait for Phase 1 measurement?

Run Phase 1 in parallel with development. While engineers build the automation, instrument the manual process. This does not add calendar time. It does require a PM or analyst dedicated to the measurement work alongside the engineering work. In our case, I personally ran Phase 1 data collection while the engineering team built the automation system.

Can this framework work for non-logistics automation (e.g., back-office, finance)?

Yes. The framework is domain-agnostic. The milestones will differ (invoice received, invoice processed, payment scheduled instead of order received, route assigned, driver notified), but the three-phase structure applies to any process automation. I have seen variations of this framework used in healthcare claims processing, insurance underwriting, and financial reconciliation. The proxy metrics table will need to be rebuilt for each domain.

What is the minimum sample size for a defensible Phase 1 baseline?

For most operational processes, 100-200 observations across at least 2 weeks provides a 95% confidence interval narrow enough for executive reporting. Ensure the sample includes variation across time of day, day of week, operator, and any other relevant dimension. If your process has high variance (coefficient of variation above 0.5), increase the sample to 250-300. [LINK:post-03]