Courts in several jurisdictions have begun using risk-assessment software to help decide whether a defendant awaiting trial should be released on bail, offering each defendant a numerical score meant to predict the likelihood of reoffending or failing to appear in court. The appeal is straightforward: an individual judge's intuition can vary considerably from one case to the next, shaped by fatigue, mood, or unconscious bias in ways even the judge cannot fully see, while a data-driven score promises the same consistent calculation applied identically to every defendant who shares the same measurable characteristics. Replacing inconsistent human judgment with a consistent, data-driven score sounds, described this way, like a straightforward improvement, trading variable intuition for measurable, repeatable objectivity.

The complication is that these algorithms learn from historical data generated by the very criminal-justice system whose past patterns they are meant to improve upon. If a particular neighbourhood was historically subject to heavier policing, arrests recorded from that neighbourhood will be more numerous in the historical data purely because more people were stopped and charged there, regardless of whether actual offending was any higher than elsewhere. An algorithm trained on this data can learn that a defendant's address predicts risk, and score defendants from that neighbourhood accordingly, without ever being told to consider race, ethnicity, or any other protected characteristic directly. The address becomes what statisticians call a proxy: a facially neutral factor that, because of how it correlates with the very policing pattern the algorithm was trained on, quietly reproduces the effect a directly named, protected characteristic would have produced if it had been used openly.

Proxy discrimination is not a problem unique to algorithms; a literacy test once used to restrict voting worked on exactly this logic, excluding by race without ever naming race as the criterion. What algorithms change is the scale and opacity of the problem. A historical literacy test used a single, identifiable proxy that a court could examine and strike down directly. A modern risk-assessment score can combine dozens of variables -- address, employment history, prior court appearances, family circumstances -- into a single number through a statistical process complex enough that identifying exactly which combination of factors is doing the discriminatory work becomes genuinely difficult, sometimes even for the engineers who built the model in the first place.

Engineers and statisticians have responded by proposing formal fairness criteria a risk-assessment algorithm could be built to satisfy -- for instance, requiring that the algorithm produce the same false-positive rate across every demographic group, so that no group is disproportionately flagged as high-risk when it in fact was not. The difficulty is that several such criteria, each independently reasonable and appealing on its own terms, have been mathematically proven incompatible with one another except in narrow, special-case circumstances: a model satisfying one fairness criterion will generally, and unavoidably, violate another. Choosing which criterion to prioritise is therefore not a technical question with one neutral, correct answer waiting to be calculated more carefully; it is a value judgment about which kind of unfairness a society is more willing to tolerate, dressed in the language of mathematics.

The honest lesson is that adopting an algorithm does not remove the value judgment a human judge previously made when deciding bail; it simply relocates that judgment to whoever selected the historical training data and chose which fairness criterion the algorithm would be built to satisfy. The promised neutrality -- consistent, data-driven, free of individual bias -- was never actually on offer in the unqualified form it was sold in. What was really on offer was a choice between two different homes for an unavoidable value judgment: one, a visible judge's reasoning, open to appeal and cross-examination in open court; the other, a fairness criterion buried inside code that almost no one outside the design team is equipped to audit, contest, or even fully understand.