Five years of on-premise GPU inference: what actually breaks
Not the models. Disks, drivers, clock drift and people leaving. The operational lessons from running detection inside a client's data centre since 2018.
· 6 min read · Cozyplus Technologies
After five years, the model has needed retraining a handful of times. Everything around it has needed more attention: storage filling with frames, driver updates that changed numerical results, and a server clock that drifted enough to misorder events.
The practices that came out of it
Retention policies decided with the client and enforced automatically. Pinned driver and runtime versions with a test suite that runs after any change. Time from a single source. A named engineer who has actually been inside the building.
Handover, again
Every one of these is documented in a runbook the client owns, because the people who built the system will not always be the people who run it.