This is the complement to the piece on the unobvious signs that an AI implementation is working: a collection of the signs that one is going wrong, observable from the CEO’s or CTO’s chair and easy to misread as “early days”. I hope it helps you notice the drift before a metric does.
We have watched these arrive in the organisations we work with — usually several at a time, because they share a root, which the last section names. One simplification: I assume you pay for tokens rather than seats, as larger organisations with real data-handling constraints tend to. The arithmetic differs on a subscription; the signs do not.
1. Nobody Is Using It
You bought the tool, cleared it through security and procurement, announced it — and the usage line stays flat.
The instinct is to treat this as a training problem. Sometimes it is. Just as often it is a human one: people come to work to do the job they agreed to do, and for some the prospect of a machine writing code — and, God forbid, writing it better than they did — is genuinely frightening. That is a reasonable thing to feel, and nobody has addressed it.
The question to ask first is not why aren’t they using it but who wanted this rolled out, and what was the goal? Bottom-up and top-down both work; what fails is one side assuming the other will simply follow. Sometimes the answer really is training. Sometimes it is something deeper — not being able, or not wanting to — and worth finding out. The simplest way to tell them apart is to start with that question.
2. Costs Climb, and Nothing Changes
The opposite sign, and the more common one. People are using it; the bill says so, and the bill grows fast. Yet nothing at the level of the organisation has moved: the product changes at the pace it always did, the reviews look the way they always looked, and if you asked what the tokens bought, nobody could point at it.
That is an evident sign something is wrong — and the one most likely to be read as success, because a rising usage line is the one thing a rollout produces on schedule. I wrote earlier that a usage dashboard measures attendance, not proof; here the same line has gone one step further, from proof of adoption to proof of cost, without ever passing through proof of effect.
3. A Proof of Concept for Everything
Demos multiply. Every idea gets a prototype, every prototype gets a presentation, and the organisation is visibly busy with AI. Then a month passes, then two, and nothing beyond the prototypes has appeared.
Making something that works on the happy path is trivial with today’s tools. The work is the part after: making the thing that demos well coherent with the architecture you actually run and the way the business actually makes money. That is where the difficulty is, and where prototypes go to stay prototypes.
Some proofs of concept belong at the start. If after a couple of months they are still all you can see, the organisation is playing with the tools rather than solving problems with them.
4. Pull Requests Nobody Can Explain
Code volume goes up and understanding does not go with it. We have been on the receiving end: several pull requests dropped on a team with the instruction “integrate this somehow”, and nobody on that team able to see how they connected to the code it was running. Merge, or not? There was no good answer, because the work had been done without understanding the system it was meant to join.
Generating code is cheap now. Understanding the code you generated is still work, and if it is not done, what you are accumulating is code that cannot be maintained. Signs 2, 3 and 4 share one mechanism: the tool was handed out, and the how was not.
5. The Conversation Is About Cost, Never About Effect
This one arrives later. Cost is on the agenda of every leadership meeting; effect is on none. The question being asked is how much has it grown, when the question that matters is does it pay back — and the first question always has an alarming answer, because burning tokens is what usage looks like.
There are three ways to be here: the effects exist and nobody counts them; the effects do not exist; or someone is reading the whole thing purely financially rather than as a balance of gains and costs. From the chair that sets budgets, all three look identical.
The simplest effect needs no instrument. Is there a meaningful increment every week or two? Do the teams say at the daily what they finished? Do completed, genuinely useful pieces of the product show up at the review? If that improved, the cost line reads differently. Either way the question about delivery comes before the question about spend — and in a failing rollout it is never asked.
6. Everyone Gets the Same Cut
The move that follows from sign 5, and it is an averaging. The organisation looks at how much it is able to spend, divides it equally, and does not look deeper or try to understand where the spend came from. Fair, defensible upward, and the one control that does not look like picking favourites — I have held versions of the position myself. The trap is on the other side of the average: the teams that may have been using the tools effectively now have a budget that lasts a week instead of a month.
The trouble is what the flat rule does not look at. Spend is not spread evenly, and it is uneven for reasons: some teams never picked the tool up, some use it without judgement, and some have built a working workflow in which increments arrive quickly and the tools are used heavily because they are used well. A cap that treats those three the same barely touches the first, trims the second, and sends the third back to writing code by hand — or to rationing tokens across a feature to make sure it can be finished.
That is the organisation shooting itself in the foot. The teams that had made the investment work are exactly where its potential was, and they are the ones a flat cut hits hardest — precisely because they were trying. A team on the receiving end will ask the question the cut skipped: why did nobody ask who delivers how much? Before any cut, find the teams using it well, look at what they do, and optimise with them. Cut per delivery, not per head.
7. Token Usage Becomes Something to Brag About
The trap at the end of this road. An organisation with no way of seeing the work has one number it can see, and the temptation is to show it: usage, per team, per month. Once that is accepted as the way work is presented, the final form of the problem arrives — people start taking pride in how many tokens they used rather than in what they did, and a good team comes to mean a heavy user.
A team showing its usage next to what it delivered is fine. In the first two weeks, when the only question is whether anyone uses the tool at all, even usage alone is fine. Past that, a beauty contest on consumption is a very red light: high token usage says nothing about whether it was worth it, and an organisation that rewards it has stopped asking.
They Share a Root
Every sign here is the same fault from a different angle: the organisation is watching the input — tokens, licences, prototypes, pull requests — because it never wrote down what the output would be. Each is measurable the day the contract is signed; none is an effect.
I have watched this habit for years: organisations using story points as a proxy for something they simply could not see in the data. This is the same situation with a new number on it. Every metric you watch should come with one more question attached — what decisions will we make on the basis of it? If you have an answer, the metric is probably a good one. But for a metric to work, you usually have to look at the thing from several perspectives, and an organisation that wants to simplify a complex problem down to a single number that is easy to track on a slide is giving up visibility it ought to have.
An organisation that can count its tokens and cannot name its effect is not measuring the rollout. It is measuring the bill.
The room the sibling piece describes is the effect. When the room has nothing to show, the numbers start talking instead, and the seven signs above are what that conversation sounds like.
What This Asks of You
Before the next cost conversation, write down what the effect was supposed to be, and put it next to the spend, team by team, in whatever form you can already see: increments shown, not tokens burned. Then decide the cut, if there is one, per delivery rather than per head.
That costs two real things. The uniform rule, which looked fair and was easy to defend. And the admission that the rollout was bought without anyone saying what it would have to change.
Which of these could you see from your chair this month — and which number were you looking at instead?