From 2016 to 2020 I worked as a business intelligence solutions architect with NSW Health. It was serious work with serious people. I also want to be honest about what the work actually was, because I think most people outside the field imagine something quite different.

They imagine analysis. Clever people studying charts and finding the insight that changes a decision. That happened. It was perhaps the last tenth of the effort.

The other nine tenths was plumbing.

What BI really was

If you have never worked in it, here is the shape of an enterprise BI project as I lived it.

Someone senior asks a reasonable question. To answer it you need data from several systems that were bought in different decades, by different teams, for different purposes, and were never designed to meet. So the project becomes:

  • Finding the data. Which system holds it, which table, which of the four columns with similar names is the one people actually use.
  • Getting access to it. Often the longest step, and rightly so when the data is about people’s health.
  • Wrangling it. Dates in three formats. Codes that changed meaning in a given year. Free text where a category should be. Duplicates, gaps, and records that are wrong in ways only a veteran of that system can explain.
  • Merging it. Matching the same person, place or thing across systems that identify them differently. This is the hard one, and it is where most of the quiet errors live.
  • Building the pipelines. Extract, transform, load, on a schedule, with logging and failure handling, so that a load which breaks is noticed and fixed before anyone relies on the numbers.
  • Modelling it into a warehouse so that the question, and the next forty questions, can be answered consistently.

Then, finally, a report. By which time months had passed, the person who asked had sometimes moved on, and the question had often changed.

None of this was waste. Every step was necessary with the tools we had. But it meant that a discipline called business intelligence spent the great majority of its budget and its best people on moving and cleaning data, and a small remainder on thinking about what the data said.

What changed

I now do the same category of work with AI, for much smaller organisations, and the difference is hard to convey to anyone who did it the old way.

Reading an unfamiliar schema and working out what the columns mean: minutes. Writing the extraction code against an API or a database: minutes. The transformation logic for those three date formats and that code that changed meaning: minutes, including the tests. A first pass at matching records across two systems, with a list of the uncertain matches for a human to review: an afternoon rather than a quarter. Documentation of the whole pipeline, which we always meant to write and rarely finished: produced as a by-product.

The pipework that took a team months is now a few days of one person’s attention. I do not say that lightly, and I do not say it with any pleasure on behalf of the many good ETL developers in the field. It is simply true.

So is BI dead?

The plumbing trade inside BI is, I think, close to it. If your value was writing transformation code by hand, that value has collapsed, in the same way it has for a lot of hand-written code.

But BI was never supposed to be plumbing. The plumbing was the toll you paid to reach the part that mattered. What AI has done is remove most of the toll. And that exposes the two things we never had enough time for.

Data governance. Who owns this number? What is the agreed definition of an admission, an active customer, a completed job? Who is allowed to see what? How do we know the data is right, and who is told when it is not? In my years in health these questions were always acknowledged as important and always squeezed by delivery deadlines, because the pipeline had to be delivered. They are now the main job. They are also more urgent than before, because when anyone can generate an analysis in ten minutes, the organisation needs to be very clear about which numbers are official and which are someone’s exploration.

Data analysis. Actual thinking. Why did this move? What would we expect to see if our theory is right? What is the data unable to tell us? What should we do differently on Monday? This is the work that people joined the field to do, and most of them got to do it on Friday afternoons. An analyst who is freed from assembling the dataset can spend the week on the question.

So my answer is: AI killed the part of BI that was never the point, and handed the time back to the parts that were.

Three cautions from someone who did it the slow way

Three things I would say to anyone now building data pipelines with AI.

Fast to write is not the same as correct. The dangerous failures in data work are silent: a join that quietly drops two percent of records, a filter that matches nothing, a code mapping that is right for every year except one. AI-written pipelines need the same reconciliation we always did. Count the rows in, count the rows out, tie the totals back to the source system, and keep doing it.

The knowledge was never in the code. The hardest part of the old job was knowing that a particular field was unreliable before a particular date, because of how one system was configured. That knowledge lives in people. AI can ask better questions of them, faster, but it cannot replace the conversation.

Sensitive data stays sensitive. Health data taught me to be careful for life. Work on company plans that do not train on your inputs, keep the data in your own environment, give the tooling the least access it needs, and keep a record of what ran and what it touched.

What this means if you run a smaller organisation

The most interesting consequence is not for big enterprises. It is that the kind of joined-up reporting that used to need a warehouse team is now within reach of a business with thirty people and a not-for-profit with twelve. The economics that kept BI an enterprise luxury were the economics of pipework, and those are gone.

I have written about what an AI analyst looks like in a small business if you want the practical version.

And if you are sitting on several systems that have never been joined, with a question you have stopped asking because the answer was always too expensive, ask me. It is probably not too expensive any more.