AI’s new cost equation: Why token economics matters
At DevSparks Hyderabad, NVIDIA’s Jigar Halani explained why as AI moves toward agents and more complex workloads, managing the utility, demand, supply, and monetization of tokens is becoming increasingly important.
As AI applications move beyond answering prompts to running agents, writing code and powering physical systems, the economics of tokens is becoming a growing concern for developers and businesses.
Speaking at YourStory's DevSparks Hyderabad 2026, Jigar Halani, Senior Director of Enterprise Solutions Architecture & Engineering, NVIDIA South Asia, broke down the emerging ‘token economy’ into four parts: utility, demand, supply, and monetization.
His argument was simple: "When you waste too much, it is when the concept of saving comes into the picture."
Token consumption is already rising rapidly. Halani said NVIDIA had initially budgeted to consume 16 trillion tokens across the company in 2026. By August, it had already crossed 18 trillion, he said, excluding token consumption through tools such as Microsoft Copilot.
For developers, he said, the first question is therefore not simply which model to use, but which model is appropriate for a particular task.
A simple query does not necessarily require a large model, Halani pointed out.
"Moving to more complex tasks, you run into a lot more latency issues, and the throughput you're looking for isn't as fast. The model type you need also becomes far more complex."
Choosing models according to the workload, rather than defaulting to one model for everything, could therefore become an important part of managing AI costs.
The demand side gets more complicated as organizations move toward agentic AI. An AI application may generate tokens not just from a user's initial request, but through reasoning, feedback, tool calls, and repeated agentic loops.
For CIOs and CTOs, Halani suggested that token demand can be understood through a relatively simple equation: the number of concurrent users, the number of requests per user and the average number of tokens consumed per request. From there, organizations then need to account for workload types, usage patterns, and whether requests can be served through a cache rather than generating new tokens.
“The more you can hit the cache instead of generating a new token, the more you can reduce cost, by at least one-fifth or one-eighth. That's the standard metric in the market right now for how much cost saving you can do in your token economics,” he said.
AI infrastructure as a full-stack optimization problem
Halani said managing supply requires optimization across three layers: model efficiency, system efficiency, and software efficiency.
"Not having knowledge of systems and hardware used to be acceptable on your interview profile. Those days are gone," said Halani, arguing that developers will increasingly need to understand the systems underneath their applications. ”The new era has come.”
That means thinking about memory, networking, storage, and how workloads are distributed across GPUs. For large models, for instance, the question is not only how many GPUs are being used, but how those GPUs communicate with each other and whether they should sit within the same rack or across racks, he explained.
The shift could eventually reach the laptop itself. Halani suggested that organizations may begin evaluating whether employee devices can run smaller AI models locally, reducing the need to send every request to an API, cloud, or internal AI factory.
The value of a token
For Halani, however, the bigger opportunity lies beyond simply reducing token costs.
"The future is not about how much you use tokens and make yourself more efficient," he said. "The future is how much you are feeding the right data, training the right model, or giving a post-training to that model, and thereby making it more and more intelligent to give you a more intelligent token back."
This could create what Halani described as customer 'stickiness'. The more an AI system understands a returning customer, the more valuable it can become to that customer, and potentially the more they may be willing to pay for it.
"That's where the token monetization piece will come into the picture and that will be the most profitable business that you will do," he said.
As agentic and physical AI expand, the infrastructure behind them will become increasingly complex. For developers, the boundaries between software, models, hardware, and economics are beginning to blur.
Edited by Teja Lele