Predictive Alerts That Nobody Used: A PM's Guide to Adoption Failure | Dinesh Sanikommu

Last updated: April 20, 2021

Predictive Alerts That Nobody Used: A PM's Guide to Adoption Failure

Product adoption fails when teams optimize for technical accuracy instead of workflow integration. We built a predictive alert system at an enterprise logistics platform that could improve SLA adherence by 8%, but adoption stalled below 15% because the alerts interrupted rather than augmented existing workflows. The fix was not better ML. It was embedding predictions inside the tools operators already used, with actions attached to every alert.

I want to tell you about the best-performing feature I ever shipped that nobody used. At an enterprise logistics platform serving 14 clients across 400+ cities, my team built a predictive alert system that could forecast SLA breaches 45 minutes before they happened. The ML model was strong: 82% precision at 78% recall. The UX was clean. The potential impact was an 8% improvement in SLA adherence, worth hundreds of thousands of dollars annually across our client base.

Six weeks after launch, daily active usage was below 15%. Three months in, two clients had asked us to turn it off. This is the story of what went wrong and what I learned about the gap between "useful" and "used."

What Did We Build and Why?

The problem was real. Logistics operations managers were firefighting SLA breaches reactively. A delivery would miss its window, the client's customer would complain, and the operations team would scramble to figure out what happened. By then, the damage was done.

Our hypothesis: if we could predict breaches before they happened, operations managers could intervene proactively. Reroute a driver, reassign a delivery, or contact the customer in advance. According to a 2020 study from Aberdeen Group, proactive customer service reduces customer churn by 33% compared to reactive service. The business case was solid.

We spent 8 weeks building the system:

1. Data pipeline: Aggregated GPS data, traffic patterns, historical delivery times, and weather data into a feature store.
2. ML model: A random forest classifier trained on 6 months of delivery data to predict SLA breach probability for each in-flight delivery.
3. Alert system: A dedicated dashboard showing at-risk deliveries ranked by breach probability, with 45-minute advance warning.
4. Notification layer: Push notifications and email alerts to operations managers when high-risk deliveries were detected.

The engineering was solid. The model performed well in testing. We launched with confidence.

What Went Wrong With Adoption?

The core failure: We built a tool that was technically correct but operationally useless. We gave operations managers information without giving them action. We created a new workflow instead of enhancing the one they already had.

Here are the five specific failure modes, in the order I discovered them:

Failure 1: Alert fatigue from day one

Our model triggered alerts for any delivery with a breach probability above 30%. On a busy day across 400+ cities, that meant 200-400 alerts per operations manager per shift. The signal-to-noise ratio was catastrophic. By day three, managers were dismissing alerts without reading them. A 2019 study published in the Journal of the American Medical Informatics Association found that alert fatigue causes 49-96% of clinical alerts to be overridden in healthcare settings. The same dynamic applies to any alert-heavy system.

Failure 2: No clear action attached to each alert

An alert saying "Delivery #4521 has a 67% probability of missing its SLA" is information, not guidance. The operations manager still had to figure out what to do: check the driver's location, look up alternative routes, decide whether to reroute or contact the customer. We had given them a prediction without a recommended action. The cognitive load of interpreting and acting on each alert was higher than the cognitive load of just firefighting breaches as they happened.

Failure 3: A separate dashboard nobody opened

We built a beautiful dedicated dashboard for predictive alerts. The problem: operations managers lived in the dispatch console. They had four monitors showing live delivery maps, driver statuses, and client communications. Adding a fifth screen for predictions was not realistic. Our dashboard had an average session duration of 47 seconds. Managers would open it, see a wall of alerts, and go back to the tools they already trusted.

Failure 4: Confidence calibration mismatch

When we said "67% probability of SLA breach," managers interpreted that differently than we intended. Some treated 67% as a certainty and over-reacted. Others treated anything below 90% as noise and ignored it. We had not invested in helping users understand what probabilities meant in operational terms. The model was well-calibrated statistically, but the UX was not calibrated to human decision-making.

Failure 5: No feedback loop

When a manager did act on an alert and successfully prevented an SLA breach, there was no way for the system to know. The alert would age out, the delivery would arrive on time, and the system recorded that as a "false positive" because the breach never happened. This created a perverse incentive: the better managers used the system, the worse the model's accuracy appeared, which eroded trust further.

What Does the Adoption Curve Actually Look Like for Enterprise Features?

After this failure, I studied our adoption data more carefully. I identified five stages that enterprise features pass through, and where ours broke down:

We passed awareness and trial easily. The system broke at value recognition because users could never see the counterfactual, the breach that did not happen because they acted on an alert.

How Did We Fix It?

The fix took six weeks and involved no changes to the ML model. The model was fine. The product around the model was broken. Here is what we changed:

1. Killed the separate dashboard. We embedded predictions directly into the existing dispatch console as colored risk indicators on each delivery. Green, yellow, orange, red. No new screen to open. No context switch. Managers could see risk at a glance while doing their normal work.
2. Attached actions to every alert. Instead of "Delivery #4521 is at risk," we showed "Delivery #4521 is at risk. Recommended: reassign to Driver #78 (2.3 km closer, estimated 12 min faster)." One click to execute the recommendation. Operations managers went from interpreting alerts to approving recommendations.
3. Raised the threshold dramatically. We moved from 30% breach probability to 75%. This reduced alerts by 80%, but the remaining alerts had a much higher true positive rate. Managers started trusting the alerts because when the system flagged something, it was almost always right.
4. Built the feedback loop. When a manager acted on a recommendation, we tracked the outcome. If the delivery arrived on time after rerouting, we showed a "breach prevented" notification. We added a weekly summary: "Your team prevented 23 SLA breaches this week." This made the invisible visible.
5. Added progressive disclosure. The default view showed only the recommendation. Managers who wanted to understand the "why" could expand to see the underlying prediction, contributing factors, and confidence level. This served both the "just tell me what to do" managers and the "I need to understand before I act" managers.

What Were the Results After the Fix?

Four weeks after shipping the redesigned system:

• Daily active usage went from 15% to 73% of operations managers.
• SLA adherence improved by 6.2% across the client base (below our theoretical 8% ceiling, but real).
• The two clients who had asked us to turn off the system asked to turn it back on.
• Alert-to-action conversion rate went from 3% (on the old system) to 41% (on the new one).

None of those gains came from improving the model. They came from improving the product.

What Are the Broader Lessons About Product Adoption?

This failure taught me five principles I have carried into every product decision since:

1. Information is not value. Showing someone data does not help them unless the data comes with context and a recommended action. A prediction without a prescription is just noise. [LINK:post-01]
2. New workflows fail. Enhanced workflows succeed. If your feature requires users to open a new tool, check a new dashboard, or learn a new process, adoption will be low regardless of the feature's value. Embed into existing workflows or accept low adoption.
3. The counterfactual problem is real. Preventive features suffer from invisibility. If your feature prevents bad outcomes, you must make the prevention visible. Show users what they avoided, not just what they did. A 2020 study from Harvard Business School found that preventive actions are systematically undervalued compared to reactive ones because the benefit is invisible by definition.
4. Alert thresholds should start high and lower gradually. It is better to launch with too few alerts and have users ask "why didn't you warn me about X?" than to launch with too many and have users disable notifications. Trust is built by accuracy, not coverage.
5. Your best metric is not model accuracy. It is action rate. An 82% precision model with a 3% action rate delivers less value than a 70% precision model with a 40% action rate. Optimize for the metric that reflects real-world impact, not the one that looks good in a model evaluation. [LINK:post-05]

How Do You Know if Your Feature Has an Adoption Problem Before Launch?

If I could go back, here are the four pre-launch checks I would run:

• Workflow shadow test: Before building anything, sit with 3-5 users and observe their actual workflow for a full shift. Map every tool they use, every screen they check, every decision they make. If your feature does not fit into that existing flow, redesign it until it does.
• Action audit: For every piece of information your feature displays, ask "what will the user do with this?" If the answer is "they'll know something," that is not enough. The answer needs to be "they'll do something specific."
• Alert math: Calculate the expected volume of notifications per user per day. If it exceeds 15-20 actionable alerts, you need to raise your thresholds or batch your notifications. This number comes from research on cognitive load in high-attention environments.
• Counterfactual visualization: If your feature prevents bad outcomes, design the "you avoided this" experience before building the feature itself. If you cannot make the counterfactual visible, reconsider whether prevention is the right framing. [LINK:post-04]

Frequently Asked Questions

Is this failure common for ML-based features in enterprise products?

Extremely common. According to VentureBeat research from 2019, 87% of ML projects never make it to production, and of those that do, a significant percentage fail at adoption rather than accuracy. The ML community has invested heavily in model performance and relatively little in the product layer that makes models useful to end users.

How do you present a product failure like this to leadership without losing credibility?

Frame it as a learning investment, not a loss. We spent 8 weeks building V1 (low adoption) and 6 weeks fixing it (high adoption). The total 14 weeks produced a system that delivered measurable SLA improvements. Compare that to teams that never ship and never learn. Present the adoption metrics side by side: before fix vs after fix. Leadership respects the ability to diagnose and recover more than the ability to never fail.

Should you always embed features in existing tools rather than building standalone dashboards?

For features that support existing workflows, yes. For features that create entirely new workflows (like a new planning tool), standalone is fine. The key question is: does this feature augment a job the user already does, or does it create a new job? If augmenting, embed. If creating, standalone. Our predictive alerts were augmenting dispatch work, so they belonged in the dispatch console. [LINK:post-02]

What is the right precision-recall tradeoff for alert systems?

It depends on the cost of false positives versus false negatives. For our SLA alerts, a false positive meant a manager spent 2 minutes investigating an alert that was not real. A false negative meant a missed SLA and a client penalty. We optimized for high precision (fewer false alarms) because alert fatigue was the bigger risk to adoption. As a rule of thumb: if users can easily dismiss an alert, optimize for recall. If dismissing alerts has cognitive cost, optimize for precision.