AI centre of excellence: what one looks like in a mid-sized company
A small group with real decision rights does more for AI adoption than a large team with none.
BY STEFAN MARITZ··7 MIN READ
TL;DR
- An AI centre of excellence (CoE) is a small, cross-functional team with the authority to set AI standards, approve tools, prioritise use cases, and measure results across the organisation. In a company of 50 to 500 people, we usually recommend a part-time group of four to six people with one accountable lead.
- Start centralised, then hand delivery back to the teams. The recommended path is to move from a centralised CoE to an advisory one once central control starts slowing teams down.
- Build it on the teams you already have, and only create a standalone AI team if they cannot carry the work.
- Adoption is where a CoE earns its keep: 83% of CEOs say AI success depends more on people's adoption than on technology, and they estimate only 25% of their workforce uses AI regularly.
- The first jobs are an approved tools list, a simple intake for ideas, reusable templates, and a measure for each workflow that changes.
Most mid-sized organisations we work with already have an AI centre of excellence. They just do not call it that. It is the two or three people everyone asks which tool to use, which data is safe to paste in, and whether that vendor demo was any good. The question is whether that group has the authority and the time to do the job properly.
This guide covers what an AI CoE does, which structure suits a company your size, who should be in it, and how to set one up in a quarter.
What is an AI centre of excellence?
An AI centre of excellence is a small team that sets the standards for how an organisation uses AI and helps everyone else meet them. It works as a central hub for AI expertise, governance, and use, so that individual experiments add up to something the business can scale.
The name comes from large enterprises, where a CoE may hold data scientists, engineers, and dedicated governance staff. In a mid-sized business it is usually far smaller. What matters is that someone owns the decisions and has the time to make them.
Without one, the same pattern appears in almost every organisation we meet. Someone in marketing pays for an assistant on a personal card, finance uses a different one, and nobody in leadership can say which company information has gone where. That is how shadow AI grows, and banning the tools rarely stops it.
What does an AI CoE do?
A working CoE makes a short list of decisions once, for everyone, and keeps them current.
- Approves tools and models. The question we are asked most, in workshops and boardrooms alike, is which AI tool to use. A CoE answers it for the whole organisation and revisits the answer as models change, which is why staying model-agnostic matters.
- Runs intake and prioritisation. One simple way to submit ideas, judged on the same criteria every time: business value, data availability, risk, and a named owner.
- Sets governance people can follow. Data rules, human review, and approval steps, written so a team lead can apply them without a meeting. A practical AI governance framework is the starting point.
- Builds reusable assets. Prompt libraries, templates, and evaluation checklists. When one team builds something that works, the CoE turns it into a pattern others can copy.
- Builds skills. Short, hands-on sessions using the team's own work as the material.
- Measures results. Per workflow, against a baseline, so leadership can see which AI investments return something and which to stop.
Governance deserves early attention as agents gain autonomy. Only one in five companies has a mature governance model for autonomous AI agents, and a CoE is the natural place to build one before it is needed.
Which AI CoE model fits a mid-sized company?
There are three common structures, plus a lighter version we recommend for most companies of 50 to 500 people.
| Model | How it works | Best for | Watch out for |
|---|---|---|---|
| Centralised | One core team owns standards, platforms, and most delivery | Early adoption where strong control matters | Becoming the bottleneck for every request |
| Federated | A central team sets standards; AI leads in each function run local use cases | Larger organisations with several business units | Uneven quality between functions |
| Hybrid | Central control for high-risk systems, local experimentation for low-risk use | A mix of regulated and everyday work | Unclear lines on what counts as high risk |
| AI council | Four to six people from the business, part-time, with one accountable lead | Companies of 50 to 500 people | Meetings that discuss but never decide |
The first three follow the centralised, federated, and hybrid models used in larger enterprises. The council is our own recommendation for mid-sized organisations, which rarely have the headcount for a dedicated team and rarely need one. Our current view is that a council works until several teams are building their own workflows, at which point a federated model makes more sense.
A CoE works like the quality function in a factory. Quality sets the standard, checks the output, and teaches the line what good looks like. If quality starts building the product itself, everything queues at its door. A CoE that insists on delivering every AI project ends up in the same place, which is why the advice from Microsoft's Cloud Adoption Framework is to shift towards an advisory role once approval delays and knowledge bottlenecks appear.
Who should be in an AI CoE?
- An accountable lead with the authority to decide. The role has spread fast: 76% of surveyed organisations had a Chief AI Officer in 2026, against 26% a year earlier. In a mid-sized business this can be a Head of AI, an existing executive with protected time, or an outsourced Chief AI Officer.
- An executive sponsor, usually the managing director or COO, who backs the decisions and owns the budget. Without that backing, a CoE cannot enforce anything.
- One person from each function that uses AI most, typically operations, marketing, and HR, chosen because they understand the work.
- The person who owns data and systems, often the IT lead.
- Someone covering privacy and compliance, part-time.
Each person giving a few hours a week is a reasonable starting shape. If the lead has no protected time, the council becomes a monthly meeting with no follow-through.
How to set up an AI CoE in 90 days
- Name the lead and the sponsor, and write down what they can decide without asking anyone else.
- Find out what is already in use, including personal accounts. You cannot govern tools you have not found, so run it as an amnesty rather than an audit and people will tell you the truth.
- Publish an approved tools list and a short policy in the first month. A short policy that is live and followed does more than a long one still in review.
- Open an intake form and score the first ideas on value, data, risk, and owner.
- Pick two workflows to redesign properly, with baselines and named owners, using the steps in our guide to the AI operating model.
- Show what works every month. A 30-minute session where a team demonstrates a workflow it changed spreads adoption faster than another document.
- Review at day 90. Count what the CoE decided, what it unblocked, and what changed in the work. If the answer is mostly meetings, change the membership or the mandate.
Frequently asked questions
What is an AI centre of excellence?
A small, cross-functional team that sets AI standards, approves tools, prioritises use cases, and measures results for the whole organisation. In a mid-sized company it is often a part-time council with one accountable lead.
Does a mid-sized company need an AI CoE?
It needs the functions of one: someone deciding which tools are approved, which use cases go first, and how results are measured. It rarely needs a large dedicated team to do that.
What is the difference between an AI CoE and a Chief AI Officer?
A Chief AI Officer is the person accountable for AI across the organisation. The CoE is the group that helps that person set standards and spread what works, and it is often led by a CAIO, a Head of AI, or an outsourced equivalent.
How big should an AI centre of excellence be?
For companies of 50 to 500 people, four to six people giving part of their week is usually enough to start. Larger organisations tend to move towards a federated model with AI leads in each function.
How do you measure whether an AI CoE is working?
Look at the decisions it made, the use cases that moved into daily work, and the measured change in those workflows against a baseline. Policy documents published and meetings held tell you very little.
If you would like to see which of the five readiness dimensions your CoE should tackle first, the AI Readiness Diagnostic gives you a scored picture in about five minutes.
WORK WITH PRAXES
Ready to move from reading about AI to getting your organisation ready for it? Start with the diagnostic.
Take the AI Readiness Diagnostic