The situation
A Delhi-based D2C brand — anonymized per our client NDA — running a mid-market beauty and personal care storefront on Shopify Plus with custom checkout, ~₹5 lakh/day in GMV, three warehouses on ShipRocket, three payment gateways, and a WhatsApp order support team of six.
When they signed with us, their incident response looked like this: something breaks. A customer complains on Instagram DM. Support team pings the developer on WhatsApp. Developer opens laptop from wherever they are, tries to figure out what's broken by tailing logs across three servers, eventually finds the issue, deploys a fix, comes back to update the team. Average time from customer report to full resolution: 4.2 hours.
At their GMV, that's roughly ₹87,000 in lost orders per incident. They were having 20-25 incidents per year that qualified as "user-visible" — a mix of checkout failures, payment gateway timeouts, ShipRocket sync issues, inventory desync bugs, and image CDN outages. Real total downtime cost: ~₹18 lakh/year, not counting the reputational cost of Instagram customers watching their orders fail in public.
Every e-commerce business has incidents. The problem here was that the business was learning about them from customers, not from monitoring. Alerts existed but they were noisy — Slack channel with 200+ alerts/day, most false positives, all ignored. That's alert fatigue, and it's the root cause of most long-MTTR shops.
Week 1-2: audit + the honest baseline
Before changing anything, we established a baseline nobody wanted to see. We pulled 90 days of incident data from their Slack alert channel, PagerDuty (barely used), Shopify webhook failures, payment gateway retry logs, and cross-referenced against customer support tickets that mentioned errors.
Findings:
- 62% of user-visible incidents were caught first by customers, not by monitoring
- Average time from incident start to first alert: 34 minutes (should be under 3 minutes)
- Average time from first alert to acknowledgment: 41 minutes (should be under 10 minutes)
- Average time from acknowledgment to resolution: 2 hours 27 minutes
- 68% of alerts in the Slack channel were false positives or duplicates
- No documented runbook existed for any of the top-10 recurring incidents
Presented this data to the client's founder. Their reaction: "This is worse than I thought. Fix it."
Week 3-4: alerting redesign (the quick win)
Before adding AI or anything sophisticated, we did unsexy fundamentals. This alone dropped MTTR to ~2 hours by end of Week 4.
Alert quality over alert quantity
We killed 60% of the existing alerts. Threshold-based CPU/memory alerts that never mattered. Duplicate alerts firing from three different sources for the same underlying issue. "Warning" level alerts that had never once corresponded to an actual problem.
What we kept + what we added:
- User-visible symptom alerts — checkout success rate drops below 95% in a 5-minute window, payment gateway error rate above 2%, product page 5xx above 1%. These fire only when actual users are being hit.
- Composite alerts — "checkout is failing AND payment gateway is slow" gets a single alert, not four separate ones.
- SLO-based alerts — burn-rate alerts on customer-facing service level objectives. If we're consuming error budget too fast, we get told.
- Alert routing by severity — P1 (customer-impacting) pages someone on PagerDuty. P2 (degraded) goes to a Slack channel with human acknowledgment required. P3 (informational) goes to a dashboard.
PagerDuty + on-call rotation
PagerDuty was already licensed but nobody was on rotation. We set up a proper primary + secondary rotation across three of their dev team + our Ops Pod on-call. Escalation: 5-minute ack window, escalates to secondary at 10 min, escalates to Ops Pod lead at 15 min.
Result: acknowledgment time dropped from 41 minutes to under 4 minutes within one rotation cycle.
Week 5-8: on-call playbooks + runbooks
Faster alerts + faster ack are worth nothing if the person who acks doesn't know what to do. So we spent 4 weeks writing runbooks for the top 15 recurring incident types.
Each runbook followed a strict template:
- Symptom — what triggered the alert, what the customer sees
- First 5 minutes — the exact commands to run + dashboards to open to confirm the incident
- Rollback / mitigation — the fastest way to stop customer harm even if the root cause isn't understood yet
- Root cause investigation — where to look for the actual bug once mitigation is in place
- Communication template — what to post in the customer support Slack channel
- Postmortem trigger — when this incident requires a full postmortem
Runbooks were tested via gameday drills — we simulated three of the top incidents on staging and timed the response. First drill: 45 minutes to resolution. By the fourth drill: 18 minutes.
Every incident that happens twice deserves a runbook. Every incident that happens three times deserves automation. Every incident that happens five times deserves a fundamental architectural fix. This progression alone eliminated 6 of the top-15 recurring incidents within 3 months.
Week 9-12: AI-driven anomaly detection
By Week 8, MTTR was already down to ~45 minutes. This is where AI enters the picture — the remaining gains came from catching incidents before they became customer-visible.
The anomaly detection layer
We deployed a simple but effective ML pipeline that watched three streams:
The order-rate anomaly detection alone caught 4 incidents in the first month that would previously have taken 30-60 minutes to detect. In one case, a Shopify script edit caused checkout to fail silently for one specific product SKU — the anomaly detection caught the drop in that SKU's order rate within 4 minutes. Traditional threshold-based monitoring would never have seen it.
Log novelty detection: the surprise winner
The most valuable component turned out to be the simplest — a Python job that embeds new error log messages and compares them against a corpus of known errors. When a completely new error pattern appears, someone investigates within minutes.
This caught: two new third-party API breaking changes, one CDN provider deprecation notice buried in error responses, and a slow-brewing memory leak in a background worker. All three would have grown into customer-visible incidents; all three were fixed before that happened.
The results, in numbers
Customer support ticket volume dropped 34% within 3 months. Not because customers were happier per se, but because they weren't opening tickets about broken orders any more — because the orders weren't breaking, and when they did, the incident was resolved before customers noticed. Fewer angry Instagram DMs. Higher NPS. Better repeat purchase rate.
The economics
- Client's monthly retainer with us: ₹1.2 lakh/month covering Ops Pod time, monitoring tooling (Datadog + PagerDuty + Loki), quarterly SRE reviews
- Annual retainer cost: ₹14.4 lakh
- Downtime saved (recovered orders): ~₹18 lakh/year
- Support ticket cost saved: ~₹4.2 lakh/year (34% reduction × cost per ticket × volume)
- Reputational value: harder to quantify but real
- Net annual ROI: ~₹8 lakh + reputational upside + founder sanity
- Payback period: under 8 weeks
What made this repeatable
The pattern here — audit, alert quality, on-call discipline, runbook-driven response, AI anomaly detection layered on top — works across most e-commerce businesses at this scale. The specific tools change based on the client's stack, but the sequence doesn't.
This is what we mean by managed application operations. Building AI on top of broken monitoring gets you nothing. Fix the operational foundation first, then use AI as a multiplier. That's the order.
What we'd do differently
- Start with runbooks in Week 1, not Week 5. Waiting until we had "good" alerts to write runbooks delayed the biggest MTTR gains by 4 weeks. Even runbooks for imperfect alerts would have helped.
- Deploy log novelty detection in Week 3, not Week 9. This single component caught 3 incidents that would've been slow-brewing disasters. Should have been the second thing we shipped after alert cleanup.
- Involve the customer support team in the transformation from Day 1. They have the highest-signal data — customer complaints — and we integrated their feedback loop only in Week 8. Would've saved 3-4 weeks of investigation.
Can we do this for your e-commerce operation?
Every mid-market e-commerce business we've worked with had a similar pattern — decent tooling, no operational discipline, MTTR way too long, teams burned out. The transformation is more about process than technology, though good technology accelerates it.
If your incident response looks anything like the "before" picture above, book a call. We'll do a 90-day audit + transformation plan the same way we did here, with concrete milestones and measured outcomes.