The First 90 Days of Stabilising a Shaky Production System

Your product goes down on a Tuesday. Support is flooded, sales is apologising, and your engineers spend Wednesday fixing it. Two weeks later, something different breaks. Nobody can tell you whether things are getting better or worse.

The instinct is to ask for more servers, a rewrite, or more engineers. Almost always, that is the wrong first move. A system that fails in ways nobody understands will fail in more expensive ways when it is bigger. This is the plan I would follow for the first 90 days: stabilise before you scale.

What "stable" means before you start

Agree on this with your team first, in plain words. Stable does not mean "never breaks". It means:

You find out about problems before customers tell you.
You can recover quickly, and you know how quickly.
The same failure does not happen twice.
Changes can ship without anyone holding their breath.

If you cannot say today which of these is true, that is your starting point, not a failure.

Days 1 to 14: stop the bleeding

Goal: make the system visible and recoverable. Do not fix root causes yet. You do not know what they are.

1. Get basic monitoring on what customers feel. Not forty dashboards. Four numbers: is the site up, how long do key pages and API calls take, what share of requests fail, and are background jobs running. Alerts go to a phone someone actually answers.

2. Prove your backups work. Having backups is not the same as being able to restore them. Pick a day in week one, restore the production database into a scratch environment, and time it. Founders are regularly surprised here, and it is far better to be surprised on a calm Tuesday.

3. Decide who is on call. Even with a team of four, write down who gets woken up, in what order, and who can make a call to roll back. An alert with no owner is just noise.

4. Start a simple incident log. Every outage, slowdown, or "weird thing" goes into one shared sheet: date, what customers saw, how long, what fixed it. This is your evidence base for the next phase.

5. Name the top three incident causes. After 10 to 14 days of logging, plus a look back through the last 3 months of support tickets, chat threads and deploys, patterns show up. Typical ones: a slow database query, a deploy that went out untested, a third-party API that times out, a server running out of memory or disk.

6. Freeze the risky stuff. Pause big refactors and new infrastructure. Keep shipping product, but in small changes.

What you hold at day 14: alerts that work, a restore you have actually timed, an on-call list, an incident log, and a short list of the three things that hurt most.

For your CTO or engineers

Use the four golden signals (latency, traffic, errors, saturation) per service. Alert on symptoms (error rate, p95 latency), not causes (CPU at 80%).
Turn on slow query logging and pg_stat_statements (or your database's equivalent). Capture the top queries by total time.
Add error tracking with release tags so you can link a spike to a deploy.
Confirm point-in-time recovery is enabled, and write down your real RTO and RPO from the restore test.
Check disk, connection count, and memory headroom on the database. These are the quiet killers.

Days 15 to 45: fix root causes in business-impact order

Now you have evidence. Rank every known problem by one question: what does this cost the business per month? Lost signups, support hours, refunds, churn risk, engineers' time spent firefighting.

Do not rank by what is technically interesting or what the team wants to rewrite.

Take the top three and fix them properly. For each one:

Write down the symptom, the cause, and how you will know it is fixed (a number, not a feeling).
Fix the smallest thing that removes the cause. A missing index beats a new database. A timeout and retry beats a new queue.
Ship it behind a safe release step (see below), watch the numbers for a week, then close it.

Most of the time, the biggest wins in this window are unglamorous: indexes and slow queries, a missing timeout on an external call, an unbounded background job, a memory leak, a deploy process with no rollback. If the database is your bottleneck, the order of moves in Postgres first is worth reading before anyone suggests a new datastore.

Also in this window: every fix gets a short post-mortem (template below). Not for blame. For learning, so the same cause is not hiding under a new symptom next month.

Example (illustrative)

A 15-person B2B SaaS startup. Hypothetical numbers, but a realistic shape. Over three months they log 11 incidents. Sorted by cause:

Cause	Incidents	Customer-facing downtime	Estimated cost
Slow report query locks up the database	4	about 6 hours	2 enterprise customers threatening to leave
Deploys with no rollback	3	about 3 hours	30 engineer-hours of firefighting
Payment provider timeouts hang requests	2	about 2 hours	failed checkouts, refund requests
Everything else	2	about 1 hour	low

The team had wanted to move to microservices. The data says otherwise. Three fixes cover 9 of 11 incidents: add an index and move reports to a read replica, add one-click rollback, and put a 5-second timeout with retry on the payment call. That is roughly four to six engineer-weeks, not a six-month rewrite. At day 45, incidents drop and, more importantly, the team can explain each one.

Days 46 to 90: guardrails, release safety, cost, ownership

The system is calmer. Now make sure it stays that way when people are tired, new engineers join, and traffic grows.

Release safety.

Every deploy can be rolled back in minutes, by anyone on call.
Small, frequent releases beat big monthly ones. Smaller change means smaller blast radius.
A short automated test set covering the flows that make you money: signup, login, checkout, the core feature.
Risky features go behind a switch you can turn off without a deploy.

Guardrails.

Set a target for reliability that reflects what customers need, for example "key pages respond in under 1 second for 99% of requests". Tie alerts to it. The post on SLOs and error budgets covers this in depth.
Capacity alerts at 70% (disk, database connections, memory), not 100%.
Rate limits and timeouts on everything that touches the outside world.

Cost. Stabilising often reveals you are overpaying for emergency capacity added during past incidents. Review it once things are calm. The AWS cost playbook gives the order of work. Do not cut capacity before you have the monitoring to see what happens.

Ownership. Every service, job and dashboard has a named owner. Write a one-page runbook for each of your top five failure modes: what it looks like, how to confirm it, how to fix it, who to call. Keep a short architecture note so the next hire is not dependent on one person's memory.

What you hold at day 90: fewer incidents, faster recovery, a safe release path, an owner for every part of the system, and a clear, evidence-based answer to "are we ready to scale?" For the next stage, see the scaling roadmap and the backend scaling checklist.

Incident post-mortem template

Use this for every meaningful incident. Keep it blameless: people did what made sense with the information they had. The aim is to fix the system, not to find a culprit. Aim to write it within 3 working days.

INCIDENT POST-MORTEM

Title:
Date / time started (with timezone):
Date / time resolved:
Duration:
Severity (1 = customers cannot use the product, 2 = major feature down, 3 = degraded):
Incident lead:
Author(s):

1. Summary
   Two or three plain sentences. What happened, who was affected.

2. Customer and business impact
   Number of customers/users affected, failed transactions,
   support tickets, revenue or contract risk.

3. Timeline (timestamps)
   - When did it actually start?
   - When was it detected, and by whom (alert, customer, engineer)?
   - Key decisions and actions
   - When was it resolved?

4. Root cause
   What was the underlying cause? Keep asking "why" until you reach
   something you can change (process, design, missing check),
   not "human error".

5. Contributing factors
   What made it worse or slower to fix? (no alert, unclear owner,
   no rollback, missing runbook)

6. What went well
   What helped us detect or recover?

7. Where we got lucky
   What could have made this much worse?

8. Action items
   | Action | Owner | Due date | Prevents / detects / mitigates | Done? |
   Limit to the few that matter. Every item has one owner and a date.

9. Follow-up
   Date we will check the actions are done and that the same
   failure has not recurred.

When NOT to do this

The product is pre-launch or has almost no users. Stabilisation is for systems customers depend on. If you are still finding product fit, keep it simple and fix things as they hurt.
The system is truly past saving. Rare, but if the core design cannot support the business model at all, a planned rebuild may be right. Even then, do days 1 to 14 first, so you are not rebuilding blind.
A single known bug is the whole problem. If one specific issue explains everything, just fix it. A 90-day plan is overkill.
You hit a hard growth wall this month. If a launch or a big customer lands in two weeks, do the capacity work you must, and run the rest of the plan afterwards.

Common mistakes

Scaling first. Bigger servers hide the problem for a month, then it returns at a higher bill.
Fixing the loudest incident, not the costliest. Rank by business impact, with the incident log as proof.
Never testing the restore. An untested backup is a hope.
Alert fatigue. Fifty alerts that nobody reads is worse than five that everyone does.
Post-mortems that blame a person. People then hide problems, and you lose the data you need.
Action items with no owner or date. They quietly die.
A big rewrite as the first response. It resets your reliability to zero while the old system still needs support.

Checklist

Four customer-facing signals monitored, alerts reach a phone
Backup restored in a test and timed
On-call list and rollback authority written down
Incident log running; top three causes named by day 14
Problems ranked by monthly business cost
Each fix has a measurable "done" number
Post-mortem written for every meaningful incident
One-click rollback for every deploy
Reliability target and capacity alerts in place
Named owner and short runbook for the top five failure modes
Day-90 review: ready to scale, yes or no, with evidence

Want a second pair of eyes on your first 14 days?

This is the work I do under Fix & Scale Existing Systems. I look at your monitoring, backups, incident history and database, and tell you which three problems to fix first. If you want to know where your system stands, book a free 30-minute technical review via /contact. More on this topic sits in the Scaling & Reliability guide, and you can see examples of past engagements on the work page.

FAQ

How long does it really take to stabilise a system?

For a typical startup, the first visible improvement (monitoring, a tested backup, a short list of top causes) lands in two weeks. Fixing the top causes and putting guardrails in place usually takes the rest of 90 days, depending on team size and how tangled the code is.

Do we need to stop building features during this?

No. Slow down large, risky changes and ship in small steps with rollback. Many teams keep most of their feature work going and set aside a fixed share of time for stabilisation.

Should we hire a full-time SRE or DevOps engineer?

Often not yet. A small team usually needs a clear plan, basic monitoring and a safe release process first. A senior outside engineer can set that up and train your team, and you can decide on a hire once you know what the work actually is.

How do we know it is working?

You see fewer incidents, shorter recovery times, and incidents you detect before customers do. Track those three numbers monthly from your incident log.