Will the AI machinery just drive faster towards wrong conclusions or can Australia avoid replacing human fallibility with digital damnation, asks STEVEN FENNELL.
Today, for now, a wrongful conviction requires humans to make mistakes. That is, admittedly, inefficient. A police officer has to jump to the wrong conclusion. An investigator has to overlook something. A witness has to misidentify someone. A prosecutor has to become convinced that the evidence is stronger than it really is. Perhaps a jury then has to make the final mistake. It is a wonderfully old-fashioned system. Now we have artificial intelligence. And the solution to human fallibility becomes a machine capable of making the same mistakes at considerably greater speed.
That is the uncomfortable lesson emerging from the United States, where law-enforcement agencies increasingly have access to systems capable of combining enormous quantities of information: criminal records, dismissed charges, vehicle registrations, property records, financial information, social-media activity, commercial databases, facial-recognition results and other fragments of a person’s digital existence. Individually, much of that information may appear innocuous. Put it together and something rather different emerges. A profile. And once the machine has constructed the profile, the danger is that investigators begin looking at the person through the profile rather than looking at the evidence about the person. That distinction matters enormously to anyone interested in wrongful convictions. And the interests of justice.
The net is already everywhere
If you think this digital net is a distant, futuristic prospect, look around. In Australia, the infrastructure is already fully operational. It starts at passport control and airport security, but it has quietly bled into everyday civilian life. Today, facial-recognition cameras line the aisles of major retail chains like Bunnings, Coles and Woolworths. In-taxi security cameras, state driver’s licence databases, police automatic number-plate recognition (ANPR) speed cameras, and millions of private smart doorbell cameras are all rapidly becoming interconnected.
On paper, this vast web is designed to catch known, wanted criminals. In practice, it sweeps up everyone—creating an inescapable dragnet where a single algorithmic mismatch can upend an innocent life. We need only look at the massive rollout of automated “Flock” safety cameras across the US to see where this leads: a system plagued by misidentifications, illegal vehicle stops and false arrests. Even when a falsely accused citizen undergoes the nightmare of arrest, charging and a legal battle only to be acquitted,* the digital record does not simply evaporate. The erroneous match, the record of the arrest and the suspicion remain permanently archived in the system, waiting to taint their next interaction with authority.
Anonymity, malice and the percolating profile
This perpetual storage creates an extraordinarily dangerous opportunity for personal malice. Imagine someone looking for payback—a disgruntled ex-partner (life or business), a hostile neighbour or a vindictive former colleague—who decides to make an anomalous, anonymous report. They lodge an unverified claim of sexual harassment, wage theft, workplace vandalism, or something far worse.
Under traditional policing, an uncorroborated, anonymous tip might sit quietly in a physical file until properly investigated. Digitised and fed into an AI system, however, it becomes permanent fuel for the algorithm. The unverified claim sits there, quietly percolating alongside old traffic fines, innocent associations and retail facial-recognition logs. Then, months later, an ordinary citizen is pulled over for a broken taillight. The officer taps the registration into their terminal, and the computer screen erupts with warnings: potential thief, rapist, tax evader, paedophile. The officer isn’t responding to proven facts; they are reacting to a spew of unverified digital noise that transforms a routine traffic stop into a high-risk confrontation.
The computer doesn’t have to say “guilty”
The most interesting danger isn’t necessarily a computer announcing that someone is guilty. It doesn’t have to. It only has to say: “This person is worth looking at.” That sounds harmless enough. Except that once someone is worth looking at, somebody looks. Once they look, they find things. And once enough things have been found, those things can begin to look like confirmation of the original suspicion.
The machine produces a lead. The investigator investigates the lead. The investigation produces information. The information is fed back into the system. The system now has more information suggesting that the investigator’s original suspicion was justified. Congratulations. We have invented a technological version of confirmation bias. Only this one comes with impressive graphics.
This is particularly troubling for individuals who have had previous interactions with the law, as algorithms inherently treat past contact as a predictor of future criminality. Yet databases are filled with information that does not mean what software assumes it means. An arrest is not a conviction. A dismissed charge is not proof of guilt. Living in a particular neighbourhood is not evidence of criminality. Knowing somebody is not necessarily participating in what they do. Appearing in the same photograph is not proof of association in a criminal enterprise. And being identified by a retail facial-recognition system is certainly not the same thing as being identified by a human witness who actually saw the person commit an offence. Yet once these disparate pieces of information are assembled into a coherent-looking intelligence product, the distinction between evidence and information about a person can become dangerously blurred.
America is not Australia — yet
It would be easy to dismiss this as another American problem. Australia is different. Our legal system is different. Our policing arrangements are different. Our rules of evidence are different. All true. But there is another Australian tradition worth remembering: If America develops an exciting new technology for policing, someone in Australia will eventually ask whether we should have one too. Usually the question begins with the words “But think of the efficiency.”
Efficiency is an extraordinarily persuasive word. It has the useful property of making almost anything sound sensible. Nobody proposes an expensive new surveillance capability by saying: “This will allow us to collect enormous quantities of information about innocent people, make probabilistic assessments about them, and occasionally arrest the wrong person.” No. The brochure says: “Enhanced intelligence capability.” Much better. It sounds like something you would buy.
And Australia is already moving towards increasingly sophisticated forms of automated analysis, data sharing, facial recognition and intelligence-led policing. The question is therefore not whether artificial intelligence will eventually find its way into Australian policing. The more important question is whether the legal safeguards will arrive before the technology becomes embedded.
The wrongful conviction problem is already bad enough
Anyone who studies wrongful convictions knows that the problem rarely begins with a single spectacular act of corruption. More often, it is a chain. A mistaken identification. An unreliable witness. An investigator convinced of the wrong theory. Evidence interpreted through that theory. Information that should have weakened the case instead being explained away. A failure to disclose something important. A prosecutor believing the case is stronger than it is (or wanting to). And eventually a courtroom presented with a narrative in which all the pieces appear to fit. AI has the potential to make that chain considerably more efficient. That should worry us.
Because an algorithm does not need to be spectacularly wrong to cause spectacular damage. It can be slightly wrong, repeatedly. A false facial-recognition match here. A misleading association there. A risk score influenced by historical policing patterns. An old arrest appearing alongside genuine convictions. A social-media connection treated as meaningful when it is not. A financial transaction that looks suspicious to software but has a perfectly innocent explanation. Each individual error may seem trivial. The problem comes when the errors are combined. The machine does not necessarily create a lie. It can create something more dangerous: a convincing story assembled from facts that do not actually prove the conclusion.*
And then there is the courtroom
Here we reach the question that should concern organisations devoted to wrongful convictions. Suppose the police use an algorithm to identify a suspect. The algorithm is not presented to the jury as proof. Fine. But what happened before the case reached the jury? Who was investigated? Who wasn’t? Which witnesses were interviewed? Which lines of inquiry were pursued? Which evidence was considered important? Which evidence was dismissed? Which suspect became the focus?
A courtroom can only examine the case that arrives at its door. It cannot easily examine all the cases that never arrived because an algorithm pointed investigators elsewhere. That is the hidden problem. AI can influence a prosecution without ever appearing in the evidence box.
And that makes conventional ideas about disclosure increasingly important. If an algorithm contributes materially to identifying a suspect, shouldn’t the defence know that? If data from multiple databases helped generate an investigative lead, shouldn’t the defence know what those databases contained? If a risk score influenced police attention, shouldn’t its methodology be capable of challenge? If the system has known error rates, shouldn’t those error rates be disclosed? And if the system cannot explain why it produced the result it did, why should a defendant be expected to accept the result simply because the computer produced it? “Computer says so” was never intended to become a principle of criminal law.
The particularly dangerous word is “probability”
There is a fundamental difference between intelligence analysis and proof. Intelligence can tell police where to look. Evidence must establish what happened. Those are not the same thing. A probability may be enormously useful to an investigator. It is not automatically proof beyond reasonable doubt. That distinction sounds obvious. But humans have a remarkable tendency to treat complicated calculations as more authoritative than ordinary human judgment. Give something a percentage, a graph and a colour-coded dashboard and suddenly everybody feels much more scientific. The defendant may still be innocent. But the spreadsheet looks very confident.
Australia should learn the lesson before learning it the hard way
The United States provides Australia with something valuable: an opportunity to watch what happens when these technologies are introduced into a criminal justice environment. We don’t have to repeat every mistake simply because somebody else has already demonstrated them.
The answer isn’t to ban artificial intelligence from policing. That would be simplistic. AI can undoubtedly assist investigators. It can identify patterns humans might miss, process enormous datasets and potentially make some forms of investigation more efficient. But efficiency is not the same as justice.
The safeguards need to be designed around the possibility that the machine is wrong. Not as an afterthought. Before deployment. There should be independent testing. Known error rates. Clear rules about what information may be used. Strict separation between convictions and mere allegations. Protection against associative guilt. Disclosure obligations when algorithmic systems materially influence an investigation. A genuine ability for the defence to challenge the technology. And, perhaps most importantly, a rule that should have been obvious long before anyone invented AI: An algorithmic lead is a lead. It is not a verdict.
The irony of all this
For decades, wrongful-conviction reform has been trying to persuade the justice system to recognise its own fallibility. We have learned that witnesses can be wrong. Police can be wrong. Experts can be wrong. Prosecutors can be wrong. Judges can be wrong. Juries can be wrong. And now we are developing machines trained on enormous quantities of information generated by all those wonderfully fallible humans. Then we ask the machine to tell us who deserves closer scrutiny.
Sherlock AI
Perhaps the machine will be right. Perhaps it will even be right most of the time. But wrongful convictions are not measured by how often the system gets it right. They are measured by what happens to an innocent person when the machine gets it wrong.
The great temptation of AI is that it promises to remove human error from decision-making. The greater danger is that it may simply industrialise it. And when that happens, the old wrongful conviction problem doesn’t disappear. It scales.
Australia therefore has a choice. We can wait until somebody is wrongly arrested, wrongly prosecuted or wrongly convicted because an algorithmic chain of assumptions turned an innocent person into a suspect. Then we can appoint a commission, hold hearings, express concern and solemnly announce that safeguards should have been introduced earlier. Or we can do something considerably less dramatic. We can ask the uncomfortable questions now.
Because there is one thing an AI system cannot tell us: whether the person it has identified is actually guilty. That, inconveniently, remains the job of evidence. And evidence, unlike artificial intelligence, still has to be proved.
*The author, Steven Fennell, draws on personal experience with similar characteristics that led to him being charged and convicted of murder, spending almost 7 years in prison before his rapid exoneration by the High Court.
EDITOR’S COMMENT: Ironically enough, just yesterday we published an article assisted by AI: “Can reasonable doubt overcome public abhorrence at child killings? Exploring the Clancy case.” An upside of AI?