Over the last few months we've been building a system that reads the paperwork behind bursary applications for independent schools. Payslips, bank statements, insurance schedules, benefit letters, that sort of thing. It goes through everything a family has submitted and gives the bursar a recommendation with its reasoning, so a person still makes the decision but doesn't spend hours on each application getting to the point where they can make a decision.
It works now. But, it didn't for quite a while! The things we learned along the way from the problems we encountered are more useful to write about than the bit where it did work.
The first thing it did with a payslip, was take the year-to-date figure, multiply it by twelve, and report a household income of about a quarter of a million pounds. It did this with complete confidence. No flag, no "this looks high", just one number, in a box, that a busy person could easily have taken at face value. When we gave the same payslip to the same model with a plain question, it got the answer right. So the model was fine. We'd wired it up badly; we were feeding it documents one at a time, with no sense of the application as a whole, and that's where this systemic nonsense crept in.
This was nto the only instance of this behaviour. An insurance schedule where it took the total contents figure and treated it as one very valuable item, rather than the individually listed items it was meant to identify. A Universal Credit letter where it read the headline amount and ignored the deductions below it. Several payslips with different figures on them, and instead of averaging or asking, it picked the biggest. Every one of those would have made a family look better off than they were. But, none of these were hard problems to resolve, once we knew about them. And there's the important bit ... knowing about it.
Which brings me to the mistake I'd most like other people to avoid. We didn't have someone who had actually assessed bursary applications sitting with us from the start. We had the client, we had the requirements, we had years of previously decided cases to test against, and we assumed that was enough. It wasn't. Once we got a proper domain expert involved the questions changed completely. We'd been asking whether the extraction was accurate. They were saying things like "a bursar would never read it that way" and "the deduction is the whole point of that letter". I reckon the absence of that person cost us about a month.
Some of the time went on things that weren't the AI at all. A bug we'd fixed came back one morning because someone had patched it directly in the AWS console, and a later deployment wiped the patch out. Documents the system had been told to expect turned out never to have been uploaded, so we spent time chasing a fault that didn't exist. A server reboot on the client's side dropped a permission and everything downstream fell over. None of that is unusual on an integration project. It just doesn't get mentioned when people talk about AI.
We also started on one of the cheapest models Amazon offers, deliberately, because the client wanted to understand running costs before anything else. Fair enough. But it meant we spent a while trying to work out whether a wrong answer was the model being cheap or the pipeline being wrong, and it was nearly always the pipeline. We've since turned that round: get it right first, then see what it costs, then decide what you can live with.
By the time it was reliable we had put the same application through the system something like 150 times. Each pass found something smaller than the last. And that right there,is the important bit that nobody budgets for.
If you're thinking about doing something like this with your own paperwork, and most businesses have a pile of it that somebody reads every week, three things I'd do from day one:
1. Get the person who does the job today into the room, and keep them there. Looking at the output, telling you what's wrong with it. A sign-off at the end is too late.
2. Test with real documents, including the scruffy ones. Old scans, odd layouts, half-complete submissions. One of the nastier problems we had came from a PDF saved in 2023 with an encryption format our tools didn't recognise. The nastier the better!
3. Treat it like software, because that's what it is. Version control, proper deployments, a record of what changed and why. The AI is the smallest part of the build.
I'm happy to talk through whether something you're looking at is worth doing. No charge for the conversation, and I'll tell you if I think it isn't.
Roy Long, skyPurple Cloud
