Arabic search is not English search with a different alphabet

A clerk types a citizen’s name into a registry. The system returns no results. The citizen is standing at the counter with an identity card in their hand. Everyone in the room now knows the system is wrong, and the clerk starts writing the name six different ways until something appears.

That scene is the failure mode Arabic search produces, and it is common enough that most departments have an informal workaround for it. We have indexed millions of Arabic records across our own products and client systems. This is what the problem actually consists of.

The five spellings of one name

Arabic offers several places where two spellings of the same name are both correct, or where one is a common variant that citizens and clerks use interchangeably.

Hamza on alif. The same name appears with alif carrying a hamza above, below, or with a bare alif, and all three are typed in practice.

Taa marbuta and haa at the end of a word. Frequently interchanged, especially in handwriting transcribed into a system years ago.

Alif maqsura and yaa at the end of a word. Regional habit and keyboard layout both influence which one appears.

Diacritics. A record entered from a printed document may carry them. A citizen typing on a phone will almost never add them.

Tatweel, the elongation character inserted for typographic reasons. It is invisible in rendering and fatal to an exact match.

Then the ones that are not spelling at all. Leading and trailing whitespace. Double spaces between name parts. Zero-width characters pasted in from a document. Arabic and Latin digits mixed inside a national identity number. Names that include a definite article in one record and not in another. A four-part name stored in one field in one system and four fields in another.

None of this is exotic. It is the normal condition of a registry that has been accumulating records for thirty years through several generations of software and at least one paper era.

Normalise twice, display once

The rule we apply is simple to state. Normalise on index and normalise on query using exactly the same function, keep the original untouched, and display the original.

Using the same function in both places sounds obvious and is the most common bug we find in systems we inherit. Two implementations drift, usually because one is in the database and one is in the application, and the result is a search that works for most names and silently fails for a class of them.

Keeping the original matters for reasons that are not technical. The spelling on a citizen’s identity document is a legal fact. It goes on the certificate, the notice and the court filing. A system that normalises for storage rather than for matching has quietly changed people’s names, and correcting that later is a data migration with legal exposure.

In practice that means every searchable name field is stored twice: the original as entered, and a normalised form used only for matching. The normalised form is never shown, never printed and never exported as the name.

Matching is a ranking problem, not a yes or no

Once everything is normalised, the opposite failure appears. Search returns too much. Normalisation deliberately collapses distinctions, so a common name now matches hundreds of people.

The answer is ranking rather than stricter matching. Exact matches on the original spelling rank above matches on the normalised form. Matches on the full name rank above matches on a single name part. A match on the first and last part together ranks above a match on a middle part alone. Where the record has a date of birth, a district or a national number, that context is worth more than any cleverness in the text matching.

We are careful with fuzzy matching, and we would encourage any evaluation committee to be careful too. Edit distance applied to Arabic behaves badly, because a single character difference can be the whole difference between two unrelated names. Phonetic algorithms designed for English are worse than useless here. If approximate matching is used, it should be a separate, clearly labelled result group under the exact results, never blended silently into the same list.

One practical detail that removes a surprising amount of pain: search each name part independently and require all parts to match something, rather than treating the name as one string. Citizens do not type their name in the order the registry stored it.

The rest of the form has the same problem

Search is where it shows first, but the same normalisation questions run through the whole application.

Digits. Accept Arabic-Indic and Latin digits everywhere, convert to one canonical form on entry, and store that. A national identity number typed with Arabic numerals must be the same value as the one typed with Latin numerals, in the field, in the database, and in every report.

Dates. Where the law expects Hijri, show Hijri, store an unambiguous canonical value, and be explicit about which calendar every displayed date belongs to. The bug we see most often is a date entered as Hijri and stored as if Gregorian, which produces a record that is wrong by centuries and looks plausible.

Sorting. Arabic alphabetical order for a list of names is not the byte order of the encoding. If a clerk is scanning a printed list, it has to sort the way a person expects.

Text direction in mixed content. A name with a Latin fragment or a number inside an Arabic sentence will reorder unpredictably unless the directionality is handled. This is most visible in generated documents and exports, which are also the artefacts that end up in a file.

Where normalisation is the wrong answer

There are places we deliberately do not normalise, and getting this wrong is more damaging than having no normalisation at all.

Never normalise for deduplication decisions taken automatically. Two normalised names being equal is a reason to show both records to a person, not a reason to merge them. Automatic merging on normalised names will eventually merge two different citizens, and that error is very hard to undo once downstream systems have copied the merged record.

Never normalise identity fields for authoritative comparison. If a name must match the identity document exactly for a legal purpose, compare the originals.

Never normalise stored legal text. Regulations, decisions and citations keep the characters they were issued with.

And be cautious about normalising across languages. Transliterated Latin spellings of Arabic names are a separate problem with its own rules, and folding the two together produces matches that are hard to explain to the person affected.

What to test before acceptance

Evaluation committees rarely ask about search. Clerks ask about it on the first morning. The gap between those two facts is where a lot of unused government software comes from.

So put an acceptance test in the tender, and make it concrete. Take two hundred real names from your own registry, chosen to include the hard cases: hamza variants, taa marbuta endings, names with the definite article, four-part names, names that differ only by one character from another citizen. For each one, write down the variants a clerk or a citizen would plausibly type. The system passes if the correct record appears in the top few results for every variant.

Ask the vendor to show you the normalisation function itself, as code, and to confirm that the same one runs at index time and at query time. Ask what happens when a name contains a tatweel or a zero-width character. Ask to see the search logs, specifically the queries that returned nothing, because that list is the most useful quality report a search system produces and most systems do not keep it.

Finally, ask what the clerk sees when there are no results. The right answer is not an empty page. It is the closest candidates, clearly marked as approximate, with the fields that would let a person confirm or reject each one. Getting that screen right is the difference between a system that is used and one that is worked around.