THE BLACK BOX MUTINY

Why Less Intelligent AI May Be the Smarter Legal Choice

Corporate optimism tends to peak just before litigation. There’s usually a polished deck, a consultant repeating “at scale”, and an AI system nobody in the room can explain without calling the vendor. The board approves it. Procurement applauds. Legal stays quiet, which isn’t necessarily reassurance.

Welcome to the friction economy – the expensive collision between hyper-automated systems and stubbornly un-automated humans. Machines can score, rank, flag and reject in milliseconds. Courts, regulators, customers, and employees still ask why. That’s the point at which speed stops looking like an advantage and starts looking like a liability, particularly when “proprietary” is offered as the complete explanation.

The commercial issue isn’t whether artificial intelligence works. It often does. The issue is whether an organisation can prove that a consequential output was lawful, rational, properly governed, and open to meaningful challenge. A model may be statistically impressive and still be unusable when the organisation has to defend what happened to a particular person. Average performance won’t answer that person’s case.

AI Explainability Is Becoming a Legal Right, Not a Product Feature

The law is moving from general unease to specific obligations. Article 86 of the EU Artificial Intelligence Act gives affected people – in defined high-risk contexts – a right to clear and meaningful explanations of the AI system’s role and the main elements of the decision. It doesn’t require an organisation to hand over source code on demand. But it does require an explanation that means something to the person whose rights or interests have been affected.

The European Data Protection Board’s guidance on automated decision-making and profilingsimilarly addresses safeguards around significant automated decisions. The legal direction is plain: if a system materially affects a person, meaningful information, human intervention and contestability are not decorative governance accessories. They are part of the architecture, whether the implementation team finds that convenient or not.

South African organisations are already part of this debate. Section 71 of the Protection of Personal Information Act 4 of 2013 restricts solely automated decisions that have legal consequences or substantially affect a data subject. Where an exception applies, the responsible party must provide an opportunity to make representations and sufficient information about the underlying logic. That final phrase is the awkward bit: many fashionable AI procurement strategies promise innovation long before anyone asks whether the logic can actually be explained.

Credit scoring is the obvious example, but the risk travels well: automated fraud flags, insurance decisions, employee profiling, contract termination triggers and supplier-risk rankings. If the output changes rights, money, access or reputation, a court will want more than the system’s confidence score. That number may help a technical team measure performance; it will not, by itself, explain what happened to the person challenging the decision.

The Litigation Risk Hidden Inside the AI Black Box

The evidentiary problem is brutal. Corporations must show not merely that the model performs well on average, but why this output occurred in this case, using this data, under this governance regime. Post-hoc explanations may describe patterns without faithfully reproducing the model’s actual reasoning. In her influential paper on high-stakes black-box models, Professor Cynthia Rudin argues that organisations should use AI systems whose decisions people can understand, rather than trying to explain unclear decisions afterwards.

This matters when lawyers question the evidence in court. A tool that offers a likely explanation for an AI decision can be challenged because it is making an educated guess about another system’s decision. A lawyer may show that certain information appearsconnected to the result, but that does not prove the original AI system actually relied on that information in the individual case. The explanation may be useful, but its reliability remains open to challenge.

Trade-secret claims won’t make the problem disappear. A vendor may resist disclosure of proprietary logic – while a litigant may insist that without it – there can’t be a fair challenge. Courts will have to balance confidentiality against procedural fairness, relevance, and the ability to evaluate causation. Protective orders may limit access, but they won’t remove the underlying tension – a party can’t fairly challenge reasoning it isn’t permitted to examine.

Should Businesses Choose Simpler, More Explainable AI?

The AI industry often claims that easier-to-understand systems are less accurate. That can be true, but not always. Sometimes businesses choose complicated systems simply because they appear more advanced, even when the job doesn’t require that level of complexity. Professor Cynthia Rudin has shown that, in many high-risk situations, simpler AI can perform just as well. The real question is whether a small improvement in accuracy is worth the extra cost and risk of using a system that is difficult to question, check, correct or defend.

The answer depends on how the AI is being used. A difficult-to-understand system that merely suggests labels for documents creates less risk than one that refuses someone finance or triggers the end of a contract. Finale Doshi-Velez and Been Kim’s work on understandable AI systems explains that a system must be judged in its real setting. Calling AI “explainable” is not enough. The explanation must make sense to the people affected and suit the purpose and seriousness of the decision.

Businesses should therefore look beyond which AI system gets the highest test score. They should choose the system that gives the best result after legal, financial, and reputational risks have been considered. For an important decision, a system that is slightly less accurate but much easier to check, record, challenge and explain may be the better business choice. This doesn’t weaken AI. It recognises that technical accuracy is only one measure of success.

A business may also use different systems for various levels of risk – simpler AI for decisions that seriously affect people, more complex AI for lower-risk tasks, and clear rules requiring human review when a result is uncertain, unusual, or likely to cause harm. Human judgement then becomes a planned safeguard, not a last-minute excuse when automation fails.

Human Oversight Must Be Real, Recorded and Able to Say No

“Human in the loop” is now used so casually that it can conceal more than it reveals. In practice, the human reviewer may receive a recommendation, 11 seconds, and no real authority to override it. Anyone who has worked inside a target-driven process knows what happens next – the recommendation becomes the decision. Calling that oversight doesn’t make it so.

Meaningful review requires access to relevant inputs, understandable reasons, sufficient time, trained reviewers, and actual discretion. Overrides must be recorded, monitored, and protected from retaliation by performance metrics that reward blind throughput. Otherwise, the reviewer becomes a rubber stamp who’ll later be introduced in affidavits as “the decision-maker”. Courts tend to notice when the decision-maker can’t describe the decision. Or they should in any case.

The NIST AI Risk Management Framework treats accountability, transparency, explainability and interpretability as characteristics of trustworthy AI. The ICO and Alan Turing Institute guidance likewise offers practical advice on selecting explanation types, choosing appropriately explainable models, and communicating decisions to affected people. Together, these frameworks give legal, risk and technical teams a common starting point – decide what must be explained, to whom, at what stage and with what evidence before the system goes live.

What General Counsel Should Demand Before AI Deployment

General counsel should not wait for a pilot to become infrastructure. Before deployment, the legal team needs a clear decision inventory: what the model decides or influences; who is affected; the legal basis for its use; the data it relies on; and the consequences if it gets the answer wrong. “It supports the process” is too vague. Counsel must establish whether the system merely informs a human decision or effectively makes it.

Contracts must allocate explainability obligations, audit access, record retention, change notification, incident support, and litigation co-operation. The vendor should disclose enough about model limitations, training and testing to let the customer govern the system properly. If the sales team says that’s impossible because the product is proprietary, ask who’ll fund the defence when proprietary becomes discoverable.

Discovery readiness should be designed upfront. Preserve model versions, logs, prompts, input sources, outputs, thresholds, approvals, overrides, validation results, and material changes. Identify which witnesses can explain the system from the moment it receives the information to the final decision or action it takes. While staff turnover is foreseeable, an organisation shouldn’t lose the ability to explain a consequential system because the only person who understood it has moved on.

Most importantly, define prohibited use cases and escalation points. Sure, a model can suggest, but it doesn’t mean it should always decide. High-impact outcomes need independent validation, bias testing, appeal routes, and periodic review. IBM’s Global AI Adoption Index 2023 found widespread enterprise deployment and continuing barriers including skills, data complexity, and ethical concerns. The pace of adoption is already outstripping many organisations’ ability to govern what they’ve bought.

Explainable AI Is a Board-Level Commercial Strategy

Boards are often told that opacity is the unavoidable tax on advanced performance. They should ask for the invoice. Calculate expected gains against regulatory exposure, dispute costs, customer attrition, operational reversals, and the cost of replacing a system that can’t survive scrutiny. Then apply the same scepticism used when somebody proposes “synergies”.

Human judgement becomes valuable precisely because automation is abundant. The scarce capability isn’t producing another score. It’s knowing when the score is wrong, when the rule is unlawful, when the context has shifted and when a technically permissible result is commercially idiotic. That judgement must be trained, empowered, and paid for. A business can automate the decision, but it can’t outsource accountability to a probability.

Legal tech providers have an opportunity here. The winners won’t merely offer smarter outputs. They’ll offer traceable workflows, versioned reasoning, configurable controls, defensible audit trails, and evidence packages counsel can use when optimism meets a summons. Explainability won’t be a compliance footnote. It’ll be a sales feature for buyers who’ve finally met their own risk register.

The Verdict: An Unexplainable AI Tool Is an Evidentiary Time Bomb

An unexplainable model can be a corporate asset – until it affects someone who’s willing to litigate. Then its celebrated complexity becomes the reason nobody can establish causation, justify a threshold, or identify responsibility. The system doesn’t need to be malicious. It merely needs to be important, opaque, and wrong at an inconvenient time.

The sensible response isn’t to abandon AI or worship simplicity. It’s to price explainability into every consequential deployment. Choose models according to legal impact, not technical vanity. Build discovery readiness before the dispute. Make human review genuine and givepeople a route to challenge outcomes. Above all, name the person who owns the automated decision and make sure that person can explain how it was reached.

Before approving another black-box deployment, bring legal, risk, data and operational leaders into the same room and test one question: could we defend this exact decision, for this exact person, under oath? If the answer depends on the vendor explaining it later, the organisation hasn’t yet done the work required to deploy it responsibly.

---

If your organisation is considering an AI pilot, AJS can help you put the controls, audit trails,and accountable workflows in place before difficult questions arise. Speak to the AJS team about legal technology that works in practice and can still be explained when a decision is challenged.

– Written by Alicia Koch on behalf of AJS

(Sources used and to whom we owe thanks: European Parliament and Council, Regulation (EU) 2024/1689 (Artificial Intelligence Act), including Article 86; European Data Protection Board, Guidelines on Automated Individual Decision-Making and Profiling; South African Legal Information Institute, Protection of Personal Information Act 4 of 2013, section 71; Cynthia Rudin, “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead”; Finale Doshi-Velez and Been Kim, “Towards a Rigorous Science of Interpretable Machine Learning”; National Institute of Standards and Technology, AI Risk Management Framework 1.0; Information Commissioner’s Office and The Alan Turing Institute, Explaining Decisions Made with AI; and IBM, Global AI Adoption Index 2023).