AI training gets judged on attendance and satisfaction because they're easy to count. Here's how to measure what matters: whether anyone works differently afterwards.
TL;DR
Ask a leadership team how its AI training went and you'll often hear a number. Ninety people trained, 4.6 out of 5 satisfaction, 82% course completion. Those numbers sound like evidence, and they're the easiest ones to collect.
They don't answer the question the business cares about, which is whether the work changed. This piece is about measuring that, using a framework that's been around for decades and suits AI training surprisingly well.
The problem predates AI. Harvard Business Review called it the great training robbery in 2016: US companies spent around $160 billion on employee training and education in 2015, close to $356 billion globally, and much of that learning didn't improve organisational performance because people soon went back to their old ways of working.
AI training falls into the same trap, with an extra twist. McKinsey notes that companies spend disproportionately on AI literacy because it's visible and easy to measure, while adoption, which is harder to measure, gets less attention. What gets measured gets managed, and the easy measures point in the wrong direction here.
I've written about why training alone rarely shifts behaviour in our piece on AI training for employees. This piece picks up from there: if you're going to train people, here's how to find out whether it worked.
Attendance tells you people turned up. Satisfaction tells you they enjoyed it, which often reflects the facilitator's charm more than the session's usefulness. Completion rates tell you people clicked through the modules. Licences issued tell you people can log in.
None of these are useless, but none of them tell you whether a finance analyst now drafts month-end commentary differently, or whether a recruiter checks AI-generated shortlists for bias. If your report to the exec team only contains these numbers, it describes activity rather than impact.
The Kirkpatrick model is the most widely used framework for evaluating training. It has four levels: reaction, learning, behaviour, and results. Its current guidance makes two points I find especially useful for AI: start with the result you want and work backwards, and measure behaviour early rather than waiting months.
Start here, before you design the training. What outcome should change if the training works? For AI, this should be tied to the specific use cases you chose. Shorter turnaround on client proposals. Less time spent on first drafts of policy documents. Fewer hours on monthly reporting.
Kirkpatrick recommends identifying both leading and lagging indicators, and points out that most of them are already tracked somewhere in the organisation. You rarely need new systems, only access to data you already have.
This is the level that matters most for AI. Behaviour means people performing the critical behaviours on the job, supported and held accountable for them. For an AI programme, define two or three critical behaviours per team, for example: "uses the approved tool to produce first drafts of client summaries" and "checks every figure in AI output against the source before sending".
Kirkpatrick warns against waiting 90 days to measure behaviour, because by then the chance to correct course has passed. I'd check at two weeks and again at four, through short manager check-ins and a look at the work itself.
Did people gain the knowledge and skills? Kirkpatrick now includes confidence and commitment at this level, describing them as early predictors of whether people will apply what they learned. For AI training, a before-and-after confidence question and a named commitment from each person are simple to capture and more useful than a quiz. Atlassian's AI training workshop play builds exactly this kind of opening and closing survey into its format.
Satisfaction belongs here, but Kirkpatrick puts the emphasis on relevance: if the training doesn't feel relevant to someone's work, they won't apply it. "How relevant was this to your actual job?" is a better question than "Did you enjoy the session?"
Here's how this might look for a finance team trained to use AI on month-end reporting.
| Level | What to measure | How | When |
|---|---|---|---|
| Results | Hours spent on month-end commentary, number of review rounds | Existing time tracking and the reviewer's notes | Baseline, then the next two month-ends |
| Behaviour | Drafts produced with the approved tool, figures checked before sending | Manager check-in and a sample of drafts | Weeks two and four |
| Learning | Confidence, and a named commitment per person | Short survey before and after | Start and end of the session |
| Reaction | Relevance to their own work | One question at the end | End of the session |
None of this needs a learning analytics platform. It needs a decision, made before the training, about what success looks like and who will check.
The most recent version of the Kirkpatrick model adds the performance environment: the conditions that support or block change regardless of how good the training was. For AI, I'd check five things before redesigning the training.
Access. Do people have the approved tool, logged in and working, on the device they use? Rules. Do they know what they're allowed to put into it? Uncertainty stops people faster than any technical problem. Managers. Is their manager asking about it, using it, and making space for it? Time. Was the new way of working allowed to be slower for the first few weeks? Incentives. Is performance still judged in a way that rewards the old method?
I'd expect the answer to sit in one of those five, and fixing it tends to cost less than a second round of training. Our guide to AI adoption goes deeper into each.
Kirkpatrick suggests agreeing "return on expectations" with stakeholders before the programme starts: what they'll consider success, defined up front. That's the most useful habit you can adopt here. If the exec team agreed beforehand that success means faster proposals and fewer review rounds, the report writes itself, and nobody can redefine success afterwards.
I'd report all four levels on one page, with behaviour and results at the top. Activity numbers can sit at the bottom as context.
If you're in the EU, note that the European Commission's guidance on AI literacy says there's no obligation to measure employees' AI knowledge. So measuring behaviour is for your own benefit: it's how you find out whether the money you spent did anything. For the wider programme design, our guide to AI upskilling covers how measurement fits in.
Define the results you want before the training, then measure behaviour on the job at two and four weeks, learning and confidence at the session, and relevance at the end. The Kirkpatrick model's four levels give a sound structure.
Usually because the environment hasn't changed: people lack access, clear rules, manager support, time to adjust, or incentives that reward the new way of working. Training changes what people know, and the environment decides what they do.
A framework for evaluating training across four levels: reaction, learning, behaviour, and results. Its current guidance recommends starting with the results you want and working backwards.
Capture reaction and learning at the session, check behaviour at around two and four weeks, and look at results over the following one to three months, depending on how often the task happens.
The share of people using AI on the specific tasks they were trained on, combined with evidence that they check the output properly. Licence counts and log-ins tell you far less.
If you want a baseline before you train anyone, our free AI Readiness Diagnostic scores your organisation across five dimensions, including people and process, and gives you a starting point to measure against.
Ready to move from reading about AI to getting your organisation ready for it? Start with the diagnostic.
Take the AI Readiness Diagnostic