A miscategorised transaction is not a cosmetic bug. On a tax return it is a misfiled number that flows into a calculation a government agency will act on. So when I built the receipt and transaction pipeline for TapTax, a Making Tax Digital app for UK sole traders, the hard part was never getting a model to read a receipt or guess a category. Models do that well on a good day. The hard part was building a system that is honest about the bad days: one that can tell the difference between "I am sure this is travel" and "I have no idea, a human should look." This post is about that difference, and the small set of design decisions that make an OCR-plus-LLM pipeline safe enough to point at someone's tax.

The shape of the problem is specific. A sole trader uploads a photo of a receipt or a CSV export from their bank. We have to turn that into structured line items, assign each one an HMRC category (the exact field names HMRC expects on a quarterly submission, like travelCosts, premisesRunningCosts, or costOfGoods), and do it at a volume where asking the user to check every single row defeats the point of the product. The temptation is to let the model decide everything and ship it. For a Your-Money-or-Your-Life domain like tax, that is the one thing you must not do.

Split reading from deciding: two stages, two models

The first decision is structural. OCR and categorisation are different jobs with different failure modes, so they are different stages with different models.

Stage one is extraction. A receipt image goes to a vision model that returns structured fields: vendor name, total amount, currency, date, and line items. In the repo this is a Claude vision call that asks for JSON only:

// receiptService.ts - extract structured fields from a receipt image
const response = await anthropic.messages.create({
  model: 'claude-haiku-4-5-20251001',
  max_tokens: 1024,
  messages: [{ role: 'user', content: [
    { type: 'image', source: { type: 'base64', media_type: mimeType, data: imageBuffer.toString('base64') } },
    { type: 'text', text: 'Extract the following from this receipt image as JSON: vendor_name, total_amount, currency, date, line_items (array). Return only valid JSON, no other text.' },
  ]}],
});

Stage two is categorisation. The extracted (or bank-fed) transaction goes to a separate text model whose only job is to map a description and amount onto the HMRC vocabulary. Keeping them apart matters because a misread total and a misjudged category need different recovery. A blurry photo is a data-capture problem; an ambiguous coffee-shop charge is a judgement problem. Conflating them hides which one actually went wrong.

Make failure cheap, not loud

The most important line in the OCR stage is not the prompt. It is what happens when the model returns something useless. The extractor never throws. If the vision call fails or returns non-JSON, it logs and hands back an empty result, and the receipt row is still saved so the user keeps their upload and can fill in the gaps:

try {
  ocrData = JSON.parse(stripJsonFence(text));
} catch {
  logger.warn({ text }, 'non-JSON OCR response - using empty ocrData');
}

That stripJsonFence call is a small, real lesson: even when you say "return only valid JSON," these models routinely wrap the object in a markdown code fence. Parse the raw string and you fall into your error branch on perfectly good output. We hit this in both the OCR and the categorisation paths, and stripping the fence before parsing was the fix. The broader principle: degrade to a known-empty state, never to a confident-looking wrong one.

Constrain the output, then score it

The categoriser is where confidence enters. The prompt is deliberately closed-world: it is handed the exact list of valid HMRC categories for the transaction type and asked to return a category plus a confidence between 0.0 and 1.0.

// categorisationService.ts
const prompt = `You are a UK tax categorisation assistant for HMRC Making Tax Digital.
Given a transaction, return the best HMRC MTD category as JSON.
Response format: {"category": "<category_name>", "confidence": <0.0-1.0>}
Valid categories for ${type}: ${categories.join(', ')}
Transaction: type=${type}, amount=£${amount}, description="${description}"`;

Two guards sit behind that. First, the returned category is validated against the allowlist; a category the model invented is rejected. Second, the confidence is clamped to [0, 1] so a stray 1.7 or negative cannot poison downstream logic. And critically, every failure mode collapses to the same safe value:

if (!parsed.category || !validCategories.includes(parsed.category)) {
  return { category: 'other', confidence: 0 };   // invalid category
}
// ...and on any API error or JSON parse failure: { category: 'other', confidence: 0 }

Confidence 0 is not a magic number here. It is the system saying "I do not know," and as you will see, zero is exactly the value that guarantees a human gets involved. A self-reported confidence from a language model is not a calibrated probability, and you should not treat it as one. What it is good for is ranking: it lets you separate the rows the model is comfortable with from the rows it is hedging on, and route them differently. That ranking is enough to build the safety mechanism that matters.

The auto-apply threshold and the review queue

Here is the core pattern. Every categorised transaction is written with two distinct pieces of state: the category field that will actually be filed, and an ai_category plus ai_confidence record of what the model suggested and how sure it was. The filing field and the audit trail are kept separate on purpose, so a later human correction can overwrite what gets filed without erasing what the model originally thought.

Confidence then decides routing against a single threshold. Rows at or above it are treated as auto-applied; rows below it surface in a human review queue:

// summary path - low-confidence rows are counted for the review queue
if (typeof r.ai_confidence === 'number' && r.ai_confidence < 0.6) {
  lowConfidenceCount += 1;
}

In TapTax the review threshold is 0.6. Below it, the row is flagged for the user to confirm or correct; at or above it, the suggestion stands unless the user touches it. Every row keeps a review_status of unreviewed until a person acts, so "the model auto-applied it" and "a human signed it off" are never confused. The number itself is a product decision, not a law of nature: tune it against your own labelled corpus and your tolerance for false confidence. The architecture is the point. There is a line, below which the machine is required to ask.

A cheap heuristic, and why it gets exactly 0.7

Not every row deserves an LLM call. A keyword pre-pass catches the obvious cases (an Uber or a train fare is travelCosts, a hosting bill is adminCosts) before any model runs, which keeps cost down on a free tier. But the interesting detail is the confidence we assign those heuristic hits: 0.7. That is a deliberate placement. It sits above the 0.6 review threshold, so a confident keyword match is auto-applied, but it stays well clear of 1.0 because a regex is a guess, not a guarantee. Confidence here is a budget you spend carefully, and the gap between 0.7 and the threshold is the margin of safety.

The ambiguous long tail is the whole game

Clear cases are easy and rare in aggregate; the long tail is where a categoriser earns its keep. A payment to a marketplace could be stock or equipment. A "consulting" line could be professional fees or staff costs. The correct behaviour for these is not a cleverer prompt, it is humility: a middling confidence, a trip to the review queue, and a human making the call. When the user corrects a row, that correction becomes the filed value and a labelled example you can learn from later. The review queue is not an admission of failure. It is the mechanism that lets you ship an imperfect model into a domain that does not tolerate silent errors, because the errors are caught at a checkpoint instead of inside a submission to HMRC.

This is what "knowing what it does not know" buys you in practice. The model is allowed to be wrong; it is not allowed to be wrong quietly. Every uncertain decision is visible, attributable, and reversible before it reaches the return. For anyone building AI into a regulated or high-stakes workflow, that property matters more than raw accuracy. A categoriser that is right 95% of the time and silent about the other 5% is dangerous. One that is right 90% of the time and flags the rest is something you can actually trust a tax filing to. We took the same approach across the OCR, bank-feed, and CSV paths in TapTax's Making Tax Digital pipeline: one threshold, one queue, one rule that uncertainty is never hidden.

Takeaways

• Split reading (OCR) from deciding (categorisation) so their failure modes recover separately.

• Constrain the model's output to a known vocabulary and validate it; reject anything off-list.

• Collapse every failure (bad JSON, invalid category, API error) to a single low-confidence value so the unknown always routes to a human.

• Pick an auto-apply threshold, send everything below it to a review queue, and keep the filed field separate from the model's suggestion.

• Treat self-reported confidence as a ranking signal, not a calibrated probability, and tune the threshold on your own data.

Solomon Amos is the founder of TapTax, a Making Tax Digital app for UK sole traders. He built TapTax's HMRC integration (open banking, AI transaction categorisation, OCR), spent two years embedded at HMRC's digital programmes, and holds a PhD in machine learning. https://www.linkedin.com/in/solomonudoh/