Seamless services? Only on better foundations

Politicians often talk about seamless public services but don’t want to tackle the underlying problem - public sector data is a mess and it makes our services grind against each other where they should click.

Electricity, sewage and the bureaucracy that manages frontline public services all have something important in common – if you can see them, there is something wrong. In Scotland, at times, public sector bureaucracy is more visible than frontline service.

What I want to argue here is that the First Minister, who this week said he wants 'seamless public services fit for the 21stcentury', has little chance of success because he hasn't carried out a serious analysis of why they're not seamless to begin with. I want to show that Scotland's clunky public bureaucracy starts with bad data structures, and that sorting those is how you fix the bigger problem

But first I want to describe what the problem looks like. A friend of mine is currently trying to get a simple, straightforward job in the public sector, has been given a firm letter of offer and a start date in early May. To get the job a series of basic checks are made and references are taken.

Everything apart from the references is public data – police checks, right-to-work status and all the rest of it. My friend contacts me regularly in a state of anxiety, afraid she has lost the job because she never hears anything. She provided all this information in March. Then again in April. Then again in May.

One of the referees has now had to submit the same reference three times. It's an online form so it's not even a 'send again' thing, it's a 'start again' thing. The other referee has sent the references twice. This week the person who has offered the job phoned pulling out their hair because my friend was supposed to start a month ago – but they still don't have the references.

Everyone keeps saying 'so sorry, we're massively understaffed'. And I think 'there is no amount of staffing that will help if you lose the references three times'. So again my friend is convinced they've changed their minds and don't want her, so much so she started applying for other jobs. I tell her again and again 'this is Scotland, this is just what it's like'. She doesn't believe me. This must be intentional. How could it be this bad? It would be massively easier to do it right.

The public realm is a single thing, a single entity. All the data is it's own data (give or take some HMRC or Companies House data). There should be nothing to wrestle. No phone calls should be required. In fact – this is the bit that is the madness of it all – no humans should be involved.

Let's take a step back and think how this job hunt should have gone. The person who actually offered the job should have been able to authorise the full set of checks on the person to whom they've offered it. The candidate should have an identifier number of some sort – at the moment probably a National Insurance number, but we should move to a proper system of digital ID.

The job being applied for is a routine public sector job. Thousands of people have this job. It's specifications and requirements are standard across all of them. When this category of post was created it should have been created with a digital specification of the requirements of applicants. Only the references wouldn't be public data.

Which means the time to carry out the checks should have been measured in milliseconds. I'm serious. Not three months; if this took more than a second it was inefficient. There were I think seven or eight basic checks to be made and they were all public data. A simple piece of computer coding should do it – really, really simple.

A single data server would have all the data inside it, structured not around the organisation which created the data (because why on earth would that make sense?) but around the person whom the data is about. The whole point of the data is that it is telling us about a person, not about some Scottish Government quango.

It means that when data was needed about someone you'd look at their consolidated data. It takes about 30 seconds for everything to click into place when you realise all of this – we each ought to have our own data store with every bit of public data ever generated about us attached. It then takes about ten further seconds to realise that if we care about privacy, it is us who should control that data.

It's all just bits and bytes after all. For example, let me tell you a bit about my data. I have all the core data you have – a date of birth, a home address, a sex, a height and weight, an income and a tax code and so on. I then have data that is a function of my life. There is a big file on a dodgy ankle I have resulting from a car accident from more than 25 years ago. That simply grows.

I have all the mountains of paperwork that it takes to go through the adoption system. I have been the director of a couple of companies so have a Companies House register. I'm a trustee of two charities so Oscr has a record of me. One of them is a children's charity so I have a PVG, as well as a PLG because I have passed my driving test. And so on.

“Scotland doesn’t have a civil service so much as a loose coalition of semi-autonomous bodies that all do whatever they like and over which politicians have only limited control.”

That's just text files and bitmapped images, nothing more. It will all slot easily into the same store. Now think about this logically; how much of my data has a date attached? Almost all of it. So I should have a personal data calendar too, quickly showing when I started and ended employment, when I became a charity trustee – and then you realise it should also have my upcoming doctor's appointment details.

(Last year I lost the letter for an important appointment which I'd put in my diary. I knew what time on what day, but forgot to note which hospital. So I had to phone – which costed you money because you pay for that member of staff too.)

And then you realise how easy it is to link everything up. Small things – like if the system knows my postcode then it knows whether it's blue bin or black bin today. It knows I've got kids so it knows when the school holidays are. It knows I dissolved a company eight years ago (about which I still get letters asking for accounts followed by a letter telling me to ignore the letter about the accounts).

So you realise at this point how slick the interface could be. You could wake to 'blue bin today, and you're remembering the kids are off on Monday? Plus it's the local Gala Day tomorrow so there will be road closures locally between 9 and 11. Plan ahead. Try to avoid travel on Sunday if you can because it's a yellow weather warning.' Oh how easy life would be...

That rather begs the question that since this is so overwhelmingly obvious and technically so easy, why don't we do it? The answers to this are endless. Let's begin with the fact that Scotland doesn't have a civil service so much as a loose coalition of semi-autonomous bodies that all do whatever they like and over which politicians have only limited control. Empires is problem one.

Procurement is problem two. I find it hard to believe the people who procure IT systems in the public sector are really that bad at it, but the alternative conclusion is that it is corruption. Every organisation commissions their own insanely complicated and expensive system from a private supplier which nine times out of ten rips them off.

I have heard this from government IT people, from IT consultants that work with government, endlessly from everyone who has to use these systems and personally from politicians who know it is all true. So why does it continue? Because it is still being driven by the commercial interests of corporations actively ripping off a fragmented public sector.

In the vast, vast majority of cases the only software which is needed is simple, off-the-shelf packages you've probably got at home. I was indulged by a senior medic on one extended hospital visit as he showed me round their software. Really it was just word processing with an image-reading function and a calendar – and the thing that stored the data.

That's the point; they shouldn't be storing the data there, they should be storing the data in a proper national data archive. All we should be doing in hospitals or local authorities or schools or in police stations is reading from and writing to the data. What we use to read and write with doesn't really matter. The data matters.

What the First Minister appears to believe is that we will be able to compensate for our third-world data anarchy with Artificial Intelligence. The idea is that rather than fix the foundations on which public services are built, an AI will run around trying to patch it together each time anything needs to be done.

There is a role for AI – AI could comfortably scrape the existing data stores of every public body and unify that data. Whatever it missed would be picked up over time. And then we'd have a real chance at seriously seamless services fit for our era rather than the 'admit it – this is basically rolodex and filing cabinets' approach we have now.

We would release frankly phenomenal amounts of public resource if we could do this. My guess is that it would very easily indeed represent a 20 per cent increase in the funding available for frontline public services, and it could be twice that in some instances.

But only if someone who understands this and is willing to face down the IT corporations and reengineer our systems from the bottom up. Because if my friend messages me again and it says anything other than 'phew, finally through it', I think I'll be too embarrassed to type the words 'sorry sweetie – this is just what Scotland is like' again.