Putting computer vision on a nation's roads, before it was fashionable

In 2018 there was nothing to buy. If you wanted a system that could tell whether a driver was holding a phone, or whether a seat belt was lying across a shoulder, no vendor could license you one. The published research was about pedestrians and number plates. The public datasets were American and European, shot in mild light through clean glass. None of them looked anything like a Gulf motorway at two in the afternoon in July.

We built that system for Tahakom. It has been detecting phone use and seat-belt violations in Saudi Arabia since 2018. Eight years on, the questions procurement committees ask us about computer vision are still mostly the wrong questions, so this is an account of what the work actually involved.

The hard part is the glass, not the phone

A windscreen is a filter that changes every few seconds. It reflects the sky. It carries dust, rain, wiper smear and a tinted film that is legal in some vehicles and not others. Sun glare at a low angle turns the whole upper half of the cabin white. At night the only light on the driver comes from the instrument cluster and whatever the camera itself provides. The object you are looking for is roughly the size of a hand, often partly behind that hand, and it is moving past the camera at motorway speed.

Seat belts are harder than phones in one way. A black belt against a black shirt at dusk is close to invisible to a sensor and to a person. Worse, a belt can be present and still wrong: routed behind the shoulder, clipped and then sat on, or worn under the arm. A model trained only on belt or no belt learns to answer a question the regulation does not ask.

So the first real engineering decision had nothing to do with machine learning. It was capture. Shutter speed short enough to freeze the cabin. Exposure metered on the cabin rather than the scene, because a correctly exposed road gives you a black windscreen. Polarising filters to cut sky reflection. Infrared illumination at night, chosen and aimed so it does not dazzle the driver. Camera height and angle fixed so the line of sight goes under the sun visor.

We will put this plainly, because it is the most expensive lesson in the field. You can spend a year improving a model to recover information the camera never captured, and you will fail. Fixing the capture costs a fraction of that. Any proposal that talks about model accuracy without first talking about optics and mounting is describing a laboratory, not a road.

Building a dataset that looks like the road

We trained on regional footage because there was no alternative. Collecting it was most of the first year.

The sampling has to mirror reality on the axes that change the image. Hour of the day, including the two hours around sunrise and sunset that produce the worst glare. Season, because summer haze and winter rain are different problems. Vehicle type, because the driver of a high cab sits at a different angle to the camera than the driver of a low saloon. Lane position. Weather. Get any of these wrong and the system works beautifully in the month you tested it and degrades quietly in the next one.

Then there is the imbalance. The overwhelming majority of drivers on any given day are doing nothing wrong. If you sample naively, a model can reach very high accuracy by saying no to everything. We built the negative set deliberately instead, out of the cases that actually confuse a detector: a hand resting against the ear, a cigarette, a water bottle raised to the mouth, a hand on the top of the wheel, sunglasses being adjusted, a passenger phone visible through the windscreen on the wrong side.

Labelling needs a written guideline before anyone labels anything, and the guideline needs to state what to do with the ambiguous cases rather than leaving each labeller to decide privately. We double-labelled a proportion of every batch and adjudicated disagreements. Where two trained people could not agree on a frame, that frame was not evidence, and it went into a separate bucket rather than into training as though it were clean.

The review loop is the product

The model was the smaller problem. The larger one was building something a human reviewer, a supervisor and eventually a court could trust.

Every detection is a case record, not a score. It carries the full frame and the cropped region, the timestamp, the camera identifier, the model version that produced it and the confidence at which it fired. A person confirms the case before it becomes a fine. That loop is the reason the system is still in service.

It also determines how you tune the model. The two errors are not symmetrical. A missed violation costs nothing visible: no one knows it happened, and the vehicle passes another camera later. A false fine sent to a citizen who did nothing costs public trust, staff time in appeals, and the political standing of the whole programme. So the operating threshold is set for precision, and the misses are accepted. The machine is a filter that removes the ninety-odd per cent of frames a human never needs to look at. It is not a judge.

Model versioning follows from the same logic. When a citizen contests a fine eighteen months later, somebody has to produce the exact frame and state which version of which model produced that decision, and ideally re-run it. That is not engineering hygiene. In an enforcement system it is a legal requirement, and it needs to be in the acceptance criteria.

What it actually costs

The visible costs are the easy ones: GPU servers sized for the camera count, the cameras themselves, and the civil works to mount and power them, which in our experience takes longer than the software.

The cost people forget is the standing human one. A detection system does not reduce headcount. It creates a permanent review workload proportional to traffic volume, and that team needs rotas, supervision, quality sampling and training. Budget it for the life of the system, not the life of the project.

Then the quiet ones. Models must be re-evaluated on fresh samples every month, which means fresh labelled data every month, which means the labelling function never shuts down. Cameras drift, get knocked, get dirty, and a camera that has silently gone out of focus produces a slow decline that nobody notices unless something is watching per-camera detection rates. And a bad first month costs political capital that takes a year to earn back, which is why the pilot phase should be run with the fines switched off.

When this is the wrong thing to build

We turn this work down more often than people expect.

If the enforcement policy is not settled, do not automate it. The definition of the offence has to be precise enough to be written into a labelling guideline. If the legal text says something a labeller cannot apply consistently to a photograph, the system will encode somebody’s private interpretation at national scale.

If there is no appeals process with a named owner, do not deploy. The appeal is the safety valve for every error you did not anticipate.

If the department cannot staff review, stop. An automated pipeline with no human confirmation is not a cheaper version of this system. It is a different and much riskier one.

And if the road estate has no cameras and no power at the gantries, the honest answer is that this is a civil-works programme with a software component, and it should be procured as one.

What to ask before you sign anything

Ask for per-class precision and recall at the operating threshold, not overall accuracy. Overall accuracy on an imbalanced problem is close to meaningless.

Ask for those numbers on data the vendor did not train on, collected on your roads, in your light. Insist the test set is held by you.

Ask what happens at three in the morning and what happens in July, and ask for the figures separately rather than as an annual average.

Ask to see the labelling guideline. It tells you more about how the vendor thinks than the architecture diagram does.

Ask how a case record is bound to a model version, and ask them to demonstrate re-running an old case on the version that produced it.

Then write an acceptance test into the tender: your own staff label a fresh sample of several thousand frames from your own cameras, the system is run against it blind, and the agreed precision threshold has to be met before acceptance. Run the same test again at six months. A vendor who has genuinely done this will welcome that clause, because it is the one that protects them too.