Five years of on-premise GPU inference: what actually breaks

We have been running phone-use and seat-belt violation detection inside a client's own infrastructure in Saudi Arabia since 2018, for Tahakom. Five years is long enough to stop guessing about what goes wrong. The short version is that the model has been the most stable part of the system. Almost everything that has demanded attention sat around it, in the parts nobody writes papers about.

This matters for anyone evaluating an AI proposal, because vendor documents are almost entirely about the model. Accuracy figures, architecture diagrams, training data volumes. Those are the things a vendor controls in a laboratory. The things that determine whether the system is still working in year four are operational, and they are usually a single line in the proposal.

Storage is the thing that stops you

Detection systems produce evidence. Every event generates frames, and often a short clip, and the frames have to be kept because a violation can be contested. Nobody sizes this correctly at the start. The estimate is built from an expected event rate, and the real event rate is higher, and the retention period gets extended during operations because a legal question came up and nobody wanted to be the person who deleted the answer.

The failure is not gradual. A detection pipeline writing to a full disk does not slow down politely. It stops, or worse, it keeps accepting work and silently drops output, and you discover the gap days later when someone queries a date range and finds nothing. We have seen the shape of this often enough that the first alert we now install on any deployment is a disk projection alert. Not a threshold on current usage, a projection: at the current write rate, this volume fills in eleven days. That gives an operations team time to act rather than a notification that the incident has already started.

The policy side is harder than the engineering side. Retention has to be decided with the client, in writing, with a legal owner, and then enforced automatically. If deletion is a manual task on somebody's checklist, it will not happen, because nobody wants personal responsibility for destroying evidence. Automating it turns a career risk into a policy that was signed once.

Driver updates change your numbers

This is the lesson that surprises people. A GPU driver update, or a change in the inference runtime, can change numerical results. Not dramatically, but enough. Floating point operations get reordered, a kernel gets a faster implementation with slightly different rounding, a library switches its default precision. Outputs that sat just above a confidence threshold move just below it. The model has not changed. The behaviour has.

In a system that produces legal consequences for citizens, that is not an acceptable surprise. So versions are pinned. Driver, CUDA runtime, inference library, container image, all pinned, with the versions recorded in a document the client holds. Nothing gets upgraded because it is available. Upgrades happen deliberately, for a reason, and each one runs through a fixed regression suite before it reaches production.

The regression suite is worth describing, because it is simple and most teams do not have one. It is a frozen set of inputs with known expected outputs, held aside, covering the ordinary cases and a deliberately chosen set of hard ones: poor light, partial occlusion, the edges of the camera frame. After any change to any layer of the stack, the suite runs and the results are compared against the stored baseline. A difference is not automatically a failure, but it is automatically a conversation. Without this, a driver update is an uncontrolled experiment on live traffic.

Time, and why it is not a small problem

A server clock that drifts a few seconds sounds harmless. It is not, in a system where events are ordered and timestamped for evidentiary use. Drift across multiple machines means events recorded on one node appear to precede events that actually happened earlier on another. Correlating a detection with a camera feed becomes unreliable. A timestamp on a violation record becomes contestable, and once one is contestable, all of them are.

The fix is unglamorous. Time comes from one agreed source, every machine synchronises to it, and drift is monitored as a first-class metric with an alert. We treat a clock out of sync the same way we treat a service that is down, because in evidentiary terms it is. We also write the time zone and the offset explicitly into every record rather than relying on a server default, because server defaults change when a machine is rebuilt and nobody notices until a daylight boundary.

The people problem is the real one

Five years is longer than most people stay in a role. The engineers who built the original deployment are not all the engineers who run it now. On the client side, the staff who were trained at handover have moved, been promoted, or left. This is the failure mode that quietly kills long-running systems, and it is entirely predictable.

What we do about it is documentation the client owns, not documentation we own. There is a runbook that lives in the client's systems, in their language, covering what the system does, what each alert means, what the first three steps are for each one, how to restart each component safely, what must never be restarted during a working shift, and who to call. It is written to be usable by an engineer who has never met us, at two in the morning, under pressure.

The test of a runbook is not whether it exists. It is whether someone who did not write it can follow it. So we rehearse. Periodically, a client engineer who was not part of the original team works through a scenario using only the runbook, and we watch and write down every place they hesitate. Those hesitations become edits. A runbook that has never been tested by a stranger is a document, not an operational asset.

We also keep a named engineer on our side who has physically been inside the building. Remote support for an on-premise system has a ceiling. When the question is which cable goes where, or what that rack sounds like when it is wrong, someone needs to have stood there.

When on-premise is the wrong choice

We run this deployment on-premise because the data is sensitive, the volumes are high and the client requires it. But we would not recommend the pattern by default, and it is worth being honest about the cost.

On-premise means you own the hardware lifecycle. GPUs age, warranties expire, a failed card is a procurement cycle rather than an API call. It means capacity is fixed, so a surge in load is a purchase order and a wait. It means you need people who can be physically present. And it means every security patch, every operating system upgrade, every network change is yours to plan and yours to survive.

If the data can legally and politically leave the building, and the volumes are moderate, a hosted deployment will cost less over five years and will be easier to staff. The case for on-premise is made by data residency requirements, by latency needs that a link cannot meet, or by a volume of inference that makes per-call pricing absurd. If none of those apply, choosing on-premise to feel in control is buying a maintenance obligation with no corresponding benefit.

What to put in the contract

If you are procuring a system like this, the clauses that protect you in year four are not about accuracy. They are these. The vendor states the exact pinned versions of every layer of the stack and cannot change them without written approval and a regression run. A frozen regression dataset exists, is held by the buyer, and its results are reported after every change. Retention policy is written into the contract with a named legal owner and enforced by the system, not by staff. Runbooks are delivered in the buyer's language and validated by a buyer engineer who did not write them, as a condition of acceptance. And the acceptance test includes an operational scenario, not only an accuracy benchmark: fill a disk in a test environment and show what the system does.

That last one is the test we would run before signing anything. An accuracy number tells you how the system performs on a good day. How it behaves on a bad day is what you will actually live with.