Lately, I see more and more companies measuring their success or even engagement with AI based on the number of tokens that are spent with the large language model. I think that using this metric is just fundamentally wrong. And here is why.
Remember the quote from Bill Gates who said “Measuring programming progress by lines of code is like measuring aircraft building progress by weight“. It’s a metric that’s easy to get by, but it’s a complete red herring. Lines of code don’t mean anything with regard to how sophisticated a program is or not. A developer can use very many lines of code to write very bad programs and developers can use very few lines of code to write very sophisticated programs, perhaps even code so opaque that other developers cannot read it anymore. And that discussion has been going on for decades, but this is not the point of this topic.
Now, why do you bring it up in regard to tokens? Let’s take a step back. What are tokens? Tokens are the unit that the large language model operates on. Whatever you provide as input and whatever the LLM provides as output, are essentially tokens. So, you have an input, such as characters, text, images, those get tokenized and then these tokens get processed by the large language model, and in return, the large language model produces tokens, and those could then, again, be characters or images, and so forth. There’s a fundamental difference as to input and output, and that’s what they call input tokens and output tokens. Very simple so far.
Now, why is measuring your AI engagement on the number of tokens wrong? Well, just as with lines of code, use of an LLM can either use very many input tokens or very few input tokens. And the LLM can produce very many output tokens or very few, depending on the task. For example, if you were to measure input tokens, just a simple question, for example, “what languages is this program written in that you are investigating?“, that question itself produces very few input tokens. Likewise, the answer, probably being a popular programming language like Python or JavaScript or something, will also produce very few output tokens. That doesn’t mean, however, that the use of the LLM by the user was meaningless. It may have been very meaningful, maybe the LLM scanned a program that’s millions of lines of code long. And likewise, a developer could, in the same vein, use a lot of input tokens by saying, “here’s a program that’s millions of lines long, ingest all of those million lines of code, and then tell me in what programming language this program is written in“. Now, all the code becomes input tokens, and it will be significantly more than what we had before. Or perhaps, if you ask an LLM to create a one-minute video about how to be active on social media. That natural language question will have very few input tokens, but the video itself is output tokens that then become a video. And because it’s a minute long and a video, there’re many, many output tokens. And the question becomes again whether measuring tokens is useful or not.
So where am I going with this? If you start measuring your AI engagement with tokens, you really don’t have any grasp in how or whether the LLM has been useful to your actual KPIs or business case that you have. If you just measure engagement with tokens, you actually run the danger that people will try to spend as many input on output tokens as possible, and because the measurement is just tokens and very opaque in its nature, you will not know whether the user spent the day creating cat videos or actually doing something useful. You may see that the user has produced a lot of output tokens or created a lot of input tokens, but productivity hasn’t gone up. And this is the important aspect, productivity. Why do we use AI? Well, to get more done with less. Less time, less effort, less errors, etc. So that should be your unit of measurement, productivity. And that highly depends on what you’re doing, of course. What does it mean to be productive? Well, if we stay in the software engineering world, and we say, “Look, we want to use AI to help us triage and fix bugs”, then your unit of measurement shouldn’t be how many tokens you generated in a week’s time but the unit of measurement should be how many bugs you closed this week now that you are using AI. And is that number significantly larger than the number of bugs you closed in previous weeks, going back to two, three years? If the answer is yes, then you probably have a good productivity gain thanks to AI. If the answer is no that doesn’t immediately mean that AI is not really useful for the job. It means that there is room for further exploration as to why. It could be that the user doesn’t know how to prompt the AI correctly or just gives it very mundane tasks or doesn’t want to really engage with the AI. There are many reasons as to why. But the important thing is your unit of measurement for AI engagement should be what are you actually trying to accomplish: the productivity output.
I understand that it is a lot easier for companies when they start out with AI to start measuring somewhere, and tokens is a fundamental metric. It’s very, very easy to get by because, again, this is the unit of measurement that is used for LLMs. Tokens go in, tokens go out, and every LLM provider will charge you based on the number of tokens that you have used. So, it’s very easy metric to get by. And I understand that if you want to gauge productivity, whether your employees are using AI or not, that would be the very first metric to use, fair enough. And it’s okay to use the tokens, but only in the beginning. You should not fall into the trap that you start measuring tokens and because of that measure nothing else and go like, “No, we got tokens, so we see whether the number of tokens increase every week”. Because, again, your employees may generate many, many tokens, but are not really more productive, because they generate random cat videos, or they just make it a game and go like who can generate the most tokens and show up at a leaderboard that is perhaps shown to management once a week, and so forth. These are all real case scenarios that I’ve seen happen.
Start out with token by all means, but very quickly move on from that and say, “Okay, look, we use tokens only until we have figured out our real units of measurement that really matter to us”. And that, again, comes back to what are you trying to achieve. You are trying to achieve fixing more bugs? Well, then see whether you actually fixed more bugs. And this goes twofold. Did the number of bugs closed and fixed go up, but also, did the number of bugs reopened remain low because you don’t want AI either to say, “Yeah, we claim to have fixed the bug”, only then to find out that the bug was actually never fixed. So, it goes hand in hand, the number of bugs fixed must go up, but also, the number of valid bug fixes must go up as well. If all these bugs come back as reopened, because the bug fix wasn’t actually a bug fix, well, then something went wrong in the process.
I understand that all of this it’s very hard to get by also because these metrics are just so domain specific, right? If you try to measure the productivity of a product manager using the LLM, well, it’s a whole different ballpark, right? You have to first define what you expect the product manager to do. Is it market and competitive analysis, and do you get insights into markets quicker than ever before? And to some extent, even that is probably subjective based on the PM and how good the person was beforehand or not. Or is it documentation and writing good and correct documentation. For each of these cases, maybe not executive management, but middle management has to sit down and determine what are our units of measurement that are meaningful to the business that we are trying to use AI for, and how do we get to these units of measurement and implement them? And that’s really where you have to start. Just telling everybody, “Hey, here is AI, now go figure it out”, will get you a multitude of possible outcomes. You will, of course, have the people that know exactly what they are doing that are super enthusiastic and use AI productively. But you will also have the people that are probably very averse to it, maybe even feel threatened by this thinking “Will this take my job away?” and actually try to actively either boycott it altogether – which would be a whole different issue altogether, it would be a cultural issue to fix in the company. Nobody should feel threatened by it, they should all embrace it – or just use up a lot of tokens without actually getting anything meaningful done. That third category, the people that use a lot of tokens without getting anything meaningful done, I think, are people that deserve to be listened to and find out what they’re doing and how they’re doing it and see what’s going wrong. It could be that the user doesn’t quite understand how to best use the LLM. Or it could also be that perhaps the wrong model was used to begin with. You were hoping to have bug fixes done by a model that was never really trained for these kinds of tasks, or you hope to get market analysis written up by a model that was never trained for that either. So there’s a large spread on that, but the one thing remains, tokens are the wrong unit of measurement. If your company measures AI engagement by number of tokens for anything longer than a couple of weeks, that’s a sign that management itself has not yet fully embraced leveraging AI and is not yet fostering a culture that embraces employees to use AI for the everyday tasks in the most effective way. So, don’t measure by tokens, measure by productivity.