Migrating a paper archive: reconciliation is the project

We are working with the Land and Tax Department in Palestine on its digital transformation. The part of this work that consumes the most time is not the part anyone budgets for. Scanning is a solved problem. You can rent the machines and hire the operators. What takes the time is deciding what is true.

A single land record can exist in three forms at once. There is the bound ledger, handwritten, authoritative by tradition and often the only version with a signature. There is a spreadsheet someone in the department built in 2009 to make their own job possible, which has been maintained by a succession of people with different conventions and no schema. And there is a folder of scans from an earlier digitisation attempt, partial, unindexed, and named according to a logic that left with the contractor.

The three do not agree. They disagree about boundaries, about areas, about the spelling of names, about which transaction came last. A registry can only hold one version. So the project is not migration. It is adjudication, at scale, and then the recording of a defensible reason for every decision.

Why the contradictions exist

It helps to understand that the disagreements are not mostly errors. They are history.

A boundary description written in 1974 referenced a wall and a tree. The wall was rebuilt two metres over in 1991. Neither record is wrong; they describe different moments. An area figure changed because a strip was taken for a road, and the change was recorded in one place and not another. A name is spelled three ways because the person themselves spelled it three ways across forty years of documents, all of which are genuinely theirs.

There is also transcription drift. Someone copying a ledger into a spreadsheet in 2009 read a nine as a four. That single character has propagated into every downstream document since, and the spreadsheet has been treated as the working truth for fifteen years because it is the one you can search.

This distinction matters operationally. An error can be corrected. A historical difference has to be represented, because both facts were true at different times and a future dispute may turn on either. A registry that flattens history into a single current value destroys the evidence that the current value rests on.

How we run reconciliation

Every record that enters the system carries three things beyond its content: a source, a confidence and a reviewer.

The source says which physical or digital artefact this value came from, specifically enough to find it again. Ledger, volume, page, line. Not the name of a system.

The confidence is set by how the value was established. A value that appears identically in all three sources is high. A value taken from one source with no corroboration is medium. A value where sources conflict and a human chose between them is recorded as a decision, not a confidence, with the reasoning attached.

The reviewer is a person, named. Not a service account, not a batch job. If a record was adjudicated, someone in the department put their name to it.

Conflicts do not get resolved automatically. They go into a queue, and department staff work through the queue with the original documents in front of them. The interface shows all versions side by side, with the source of each, and asks for a decision and a reason. The reason is a required field. It cannot be skipped, and it is stored permanently with the record.

Nothing is silently merged. That is the rule we would defend hardest. An automated fuzzy match that picks the most common spelling and moves on will process the archive far faster and will produce a registry nobody can defend in a court. The moment a citizen challenges a single record and the department cannot say who decided it or why, every record becomes suspect.

The part that is about people, not data

Reconciliation queues are worked by department staff, not by us, and not by temporary data entry contractors. This is slower and it is the right choice for two reasons.

The first is competence. The person who has worked in that office for twelve years knows that a particular surveyor in the nineties recorded areas in a non-standard way, and that a specific volume was rewritten after water damage. That knowledge exists nowhere else. A contractor at a keyboard will make a defensible-looking decision that is wrong in a way only that person would catch.

The second is ownership. A registry that staff adjudicated is a registry they will defend. A registry a vendor produced is one they will work around, keeping their own spreadsheet on the side as the real record, which is exactly how you end up with a fourth contradictory source and a digital transformation that made things worse.

This has a cost, and it needs to be stated in the plan rather than discovered. Staff working a reconciliation queue are not doing their normal work. If the department is already at capacity, reconciliation either extends the timeline or requires backfill. Pretending otherwise produces a schedule that slips in month three and a project that gets blamed for it.

There is also a political cost. Adjudication means someone decides that a record a colleague maintained for years was wrong. Done carelessly, that reads as an audit of individuals. We frame it, always and explicitly, as reconciling sources rather than correcting people, and we make sure the interface never displays who originally entered a disputed value. The goal is a correct registry, not an inquest.

When to do less of this

Full reconciliation of an entire archive before launch is not always the right plan, and we would push back on it in some cases.

If a department has a large archive but only a small, identifiable subset is active, reconciling everything before anyone can use the system delays the benefit by years for records that may never be touched. The alternative is to reconcile on demand: migrate everything with its sources and confidences intact, launch with the active subset adjudicated, and route any record that is touched for the first time through the reconciliation queue before it can be transacted on.

That approach requires discipline. The system must refuse to complete a transaction on an unreconciled record, visibly and with an explanation, rather than allowing it through with a warning that everyone learns to click past. If you cannot commit to that enforcement, reconcile up front instead, because a registry where unverified records can be transacted is worse than the paper it replaced. Paper at least does not imply a guarantee.

What to write into the tender

If you are procuring an archive migration, these are the requirements we would not negotiate away.

Every migrated record stores its source artefact reference, a confidence level, the identity of any human reviewer, and, where a conflict was adjudicated, the stated reason. All of these are queryable and exportable, not buried in a log.

The original scan of the source document is retrievable from the record in one action. A registry entry that cannot show its own paper is an assertion.

Historical values are retained rather than overwritten. The record shows its current state and its full history, with the source of each change.

No automated merge without a recorded human decision, for any field the department designates as legally significant.

And the acceptance test is not a count of migrated records. Pick fifty records at random, including ten that the system marks as having had conflicts. For each one, ask the department to produce the original document, the decision, the reason and the reviewer within a few minutes, using only the system. If that works for fifty, the migration is real. If the vendor reports a hundred percent completion but cannot do that for a single contested record, what has been delivered is a large, tidy, undefendable database.