What mistakes does AI make?

AI still makes things up, gets simple counting and sums wrong, and misses details buried in long documents, and the hard part is that its wrong answers look exactly like its right ones. That doesn't make it useless. It means the right jobs to hand it are the ones where a mistake gets caught before it matters.

We set up AI for a living, so we're keen on it, and that's exactly why we'd rather you heard about the weak spots from us.

A wrong answer looks just like a right one

Witness statements and letters put before the High Court in one claim cited forty-five legal authorities. Eighteen of them didn't exist. Of the ones that were real, many didn't contain the quotations attributed to them or support the points they were cited for. One of the invented cases was attributed to the judge hearing the application.

That was in Al-Haroun v Qatar National Bank, heard alongside R (Ayinde) v London Borough of Haringey by a Divisional Court (a panel of senior judges) and reported at [2025] EWHC 1383 (Admin). We've covered what the case means for solicitors using AI separately; here it's simply the clearest example of what a confident wrong answer looks like.

In the Ayinde case, a barrister put five non-existent cases into grounds for judicial review, one of them presented as a Court of Appeal decision. Ritchie J ordered her and the law centre instructing her to pay £2,000 each in wasted costs, which are costs a court orders a lawyer to cover personally because of how they handled a case, and referred the matter to the barristers' and solicitors' regulators. The Divisional Court held that the threshold for contempt proceedings had been met, and decided, while saying expressly that it wasn't setting a precedent, not to bring them.

It's worth being precise about one thing. In the Ayinde case the court couldn't establish whether AI had been used at all, only that the citations were false and hadn't been checked. In Al-Haroun it wasn't in dispute: the client said the citations were generated using publicly available AI tools, legal search engines and online sources, and his solicitor accepted he'd relied on that research without checking it.

The court described the tools like this: generative AI tools "can produce apparently coherent and plausible responses to prompts, but those coherent and plausible responses may turn out to be entirely incorrect. The responses may make confident assertions that are simply untrue. They may cite sources that do not exist. They may purport to quote passages from a genuine source that do not appear in that source."

The material read correctly. It looked exactly like properly done work, and that's the whole problem: a wrong answer and a right answer arrive at the same speed, in the same confident tone. The Solicitors Regulation Authority made the same point in a warning notice published on 17 August 2026, noting that AI tools generate "fictitious cases, references or seemingly factual assertions that may appear convincing despite having no basis in fact."

Why AI sounds sure even when it's guessing

People call this "hallucination": the AI filling a gap with something that sounds right instead of saying it doesn't know. Think of a pupil in an exam who writes something plausible rather than leaving a blank, because a blank definitely scores nothing.

That's close to the explanation in Why Language Models Hallucinate, published in September 2025 by three OpenAI researchers and a Georgia Tech professor. They argue that models guess because guessing scores better in testing: "language models are optimized to be good test-takers, and guessing when uncertain improves test performance." Saying "I don't know" earns nothing. A confident guess sometimes earns a point. So it isn't a simple fault that one update will remove.

The paper gives a small, telling example. Asked the title of one author's PhD dissertation, three widely used models each answered confidently, naming three different universities and three different years. None of them gave the correct title or the correct year.

The judiciary's guidance for judicial office holders, updated on 31 October 2025, puts it plainly: the output "is what the model predicts to be the most likely combination of words... It is not necessarily the most accurate answer", and "'wrong' answers are not infrequent."

It doesn't really count, and sums can trip it up

AI writes by predicting what words usually come next, a bit like a very well-read version of your phone's predictive text. That's brilliant for writing and poor for counting.

Ask a model how many times the letter D appears in the word DEEPSEEK. The same paper records ten attempts at that question against one model, which answered "2" or "3". Two other well-known models answered as high as "6" and "7". It isn't counting the letters. It's predicting what an answer to that sort of question usually looks like.

Word problems go better, but not reliably. GSM-Symbolic, a study by Apple researchers first published in October 2024, took a standard set of school-level maths questions and re-asked them with different names and numbers. Every model tested scored worse on the re-asked questions, and scores fell even when only the numbers were changed. Adding one extra sentence that looked relevant but didn't affect the answer caused drops of up to 65 per cent.

That study was about school maths, not your quotes, so it's worth not over-reading it. What it does tell you is useful: small, surface-level changes can change the answer. Something that works perfectly on the tidy example you tried may behave differently on the real email a customer sends, with no warning that it has.

Long documents are harder than short ones

The natural thing with a big document is to paste the whole thing in and ask a question. That's actually close to the weakest way to use it.

AI tools measure how much text they can take in using "tokens", which are small chunks of words. Roughly speaking, 128,000 tokens is a long book's worth.

NoLiMa, presented at the ICML conference in 2025, tested thirteen models that all claim to handle at least 128,000 tokens. It hid one relevant fact in a long body of text, worded so that the question and the fact shared almost no words, so it couldn't be found just by matching words. All the models did well on short inputs. At 32,000 tokens, eleven of the thirteen fell below half their short-input score. GPT-4o, one of the better performers, went from 99.3 per cent to 69.7 per cent.

An earlier study, Lost in the Middle, found accuracy is highest when the important material sits near the start or the end, and drops when it's in the middle. It's rather like a reader who remembers the opening and the last page of a report better than page 40.

Both of these are benchmarks, which are standard tests researchers use to compare models, rather than your own filing cabinet. They tested particular models at particular dates, so treat the exact figures as a snapshot rather than a measure of whatever tool you use today.

How to decide which jobs to hand it

Every one of those failures has the same character: quiet, believable, and hard to spot at the moment it happens. So the useful question isn't "is AI reliable?" It isn't, in the way a calculator is. The useful question is "which of my jobs can cope with a wrong answer that looks right?"

Three tests, in order of importance.

Does someone catch the mistake before it counts? A draft you read before sending is safe. An automatic reply to an enquiry is riskier, because any error reaches a customer with your name on it.

Is checking the result quicker than doing the job? Summarising a recorded call passes: you can skim the notes in a minute. Reconciling figures doesn't, because checking the reconciliation properly means doing the reconciliation.

Does an error stay put, or does it spread? A clumsy sentence in a draft stops there. A wrong figure copied into a spreadsheet that feeds a quote that feeds an invoice doesn't.

If working through those tests for your own business sounds like a chore, we're happy to go through them with you and set AI up on the jobs that pass.

The judiciary drew its own version of this line, which is useful because it comes from an institution that has every reason to be careful. It lists as useful: summarising text, drafting presentations, and admin work such as composing and prioritising emails, and transcribing and summarising meetings. It doesn't recommend legal research or legal analysis, on the basis that "the current public AI chatbots do not produce convincing analysis or reasoning."

What we'd suggest

Have AI draft and a person approve, on everything, until each task has earned your trust. Anthropic's own documentation says the same about its own model: the techniques for reducing hallucination "don't eliminate them entirely. Always validate critical information, especially for high-stakes decisions."

Then, for the first month, keep a note of what it gets wrong. You're looking for a pattern. Mistakes that look the same every time usually mean the setup needs adjusting, for example the AI is working from the wrong documents or the task has been described too loosely, and that can be fixed. Mistakes scattered all over the place usually mean the task itself isn't a good fit, and fiddling with the instructions won't change that.

If you'd like the draft-and-approve step built into the tools you already use, so nothing goes out without a person seeing it first, we're happy to set that up for you. You can see what we set up for small businesses on our main page.

The jobs that pass these tests are less exciting than the ones in the adverts: drafting, summarising, sorting, and pulling together a first version for a person to correct. They're also where most of the real time savings are. None of the failures above are a reason not to use AI. They're simply the things to design around.

Sources

• Ayinde v London Borough of Haringey and Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin) - Courts and Tribunals Judiciary
• Misuse of AI - warning notice, 17 August 2026 - Solicitors Regulation Authority
• Artificial Intelligence (AI) Guidance for Judicial Office Holders, 31 October 2025 - Courts and Tribunals Judiciary
• Why Language Models Hallucinate - Kalai, Nachum, Vempala and Zhang, September 2025
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models - Mirzadeh and others, October 2024
• NoLiMa: Long-Context Evaluation Beyond Literal Matching - Modarressi and others, ICML 2025
• Lost in the Middle: How Language Models Use Long Contexts - Liu and others, TACL
• Reduce hallucinations - Anthropic

Checked on 13 October 2026. The benchmark figures are snapshots of particular models at particular dates.