Keeping a learning platform up when the world moved online overnight

When universities closed in March 2020, students arrived faster than any forecast we had. Not a busy week. A step change that did not come back down.

We stayed up. That was not luck, and it was not heroics either. It was a set of unremarkable decisions taken before the load arrived, plus a few bad decisions taken during it that we spent the following year removing. Both halves are worth writing down, because the shape of the event is the same one a public body faces on a deadline day.

The order things broke in

Systems fail in a sequence, and knowing the sequence is most of the value of having lived through one.

Search went first. It was the most expensive operation per request, and unlike page views it could not be served from a cache, because every query was slightly different. Search degrading is also the worst failure to have, because a student who cannot find anything simply retries, which adds load.

The database went second, and not through raw query volume. It went through connections. Every new application instance we added to cope with traffic opened its own pool, and the pools collectively exhausted the server long before CPU became the constraint. Scaling out the thing in front of a database can be what kills the database.

Support went third, and it kept failing for weeks after the platform was stable. Tickets arrive after the incident, not during it, so a support queue is a lagging indicator. Anyone planning capacity should plan the human queue with the same seriousness as the technical one.

What did not break was interesting too. Static assets, video delivery and anything already at the edge were untroubled throughout. The lesson is that the parts of a system that hold under stress are the parts that had been moved off the critical path earlier, for unrelated reasons.

What held, and why it held before it was needed

Three things carried us, and none of them was clever.

Caching at the edge, so that the majority of requests never reached our own servers at all. The discipline behind it is deciding, per route, what is cacheable and for how long, and having a way to invalidate a single item without flushing everything.

Read replicas, with the application knowing which queries were allowed to read stale data. That distinction has to be made in the code long in advance. You cannot decide during an incident which of your two hundred queries tolerate a lag of a few seconds.

A queue in front of anything that wrote. Under sudden load the difference between a system that slows down and a system that falls over is whether writes are allowed to pile up in a buffer or are forced to happen synchronously. Queues convert an outage into a delay, and a delay is something you can explain to users.

The reason all three worked is that they had been rehearsed. We had run load tests, we had failed over a replica on purpose, and we had practised flipping the cache aggressively. A control that has never been exercised is not a control. It is an assumption.

The decisions we had to unwind afterwards

This is the part usually left out of stories like this.

To buy headroom quickly we turned off several things that mattered. Some non-critical writes were dropped rather than queued, which meant a few weeks of incomplete usage analytics that we could never reconstruct. We raised cache times far beyond what the content justified, so a small number of students saw a corrected answer explanation days later than they should have. We scaled the database vertically because it was the only move available in an afternoon, and we then carried that cost for months before the work to shard read traffic properly was finished.

We also added feature flags in a hurry, and hurried flags become permanent. A year later we were still finding conditionals guarding a thing that had been turned off in April 2020 and never turned back on. Every emergency switch needs an owner and a review date at the moment it is created, or it becomes part of the architecture by default.

The honest summary is that surviving a load event costs technical debt, and the bill arrives later. Plan for the cleanup as part of the response, not as something you will get to.

What it cost

Infrastructure spend rose sharply and did not fall back, because the traffic did not fall back. That is the easy number.

The harder costs were the roadmap and the people. Two months of planned product work did not happen. The engineers who carried the pager through that period were tired in a way that showed up in the following quarter, and that is a real cost that does not appear in any invoice. We now treat on-call rotation during a sustained incident as something to be actively managed rather than left to whoever is most willing.

And there was a trust cost even though we stayed up. Response times were worse than normal for several weeks. Users notice slow before they notice down.

Turning it into something a department can use

Afterwards we wrote down the numbers. Requests per second at each stage, the ratio of reads to writes, the point at which each component became the constraint, and how long each mitigation took to apply. Those figures are worth more than any general advice, because they let you argue about capacity with evidence.

Then we turned the rehearsals into a runbook, and that runbook now goes into government proposals. It contains, for each of the ten most likely failures, what the symptom looks like on a dashboard, what to check first, what to do, who to tell, and what the rollback is. It is written to be executed by a department employee at two in the morning, not by us.

The mapping to public-sector work is direct. A citizen portal on the day a tax deadline lands behaves exactly like a study platform the night before exams: a hard deadline, a population that cannot choose to come later, and a spike that is entirely predictable in date and largely predictable in size. Predictable spikes are the easiest kind to prepare for and the most embarrassing kind to fail.

The practical step we recommend is a rehearsal with the client before go-live. Load-test against the real production shape, take a replica out deliberately, have department staff work the runbook while we watch without intervening. Whatever they cannot do without us is a gap in the documentation, not a gap in them.

The counter-argument

Not every system should be built for peak, and we say so in proposals.

Engineering for a spike that is ten or a hundred times normal load carries a permanent cost in complexity. Caching introduces staleness, replicas introduce lag, queues introduce eventual consistency, and every one of those makes the system harder to reason about and harder for an inheriting team to operate. For an internal system used by four hundred staff on weekdays, that trade is a bad one. Buy a larger server.

The judgement is about the shape of the demand, not the size of the organisation. Ask whether your users can be asked to come back later. For an internal workflow tool, often yes. For a portal on the last day of a filing window, absolutely not, and that is when the complexity earns its place.

The other honest caveat is that no amount of preparation removes the need to decide in the moment. A runbook covers the failures you imagined. The one that takes you down will be the one you did not, and what saves you then is a team that understands why each control exists rather than one that can follow the steps.