The Real Cost of Deploying AI Goes Beyond Tokens

At the Open Data Science Conference, I spoke about a challenge many teams are beginning to confront: the cost of deploying AI is growing alongside the ways people use it. More token-intensive work, a rapid shift toward agentic tasks, and wider adoption are putting pressure on budgets. Yet the cost of a model call is only one part of the picture.

The conversations at the conference pointed to a broader set of questions. How should organizations choose models for different tasks? How can they tell whether a model is performing well for their needs? What context should be supplied to an LLM? And what operational capabilities are required to put AI into production responsibly? These questions are connected, and answering them calls for more than simply choosing a model.

Token Demand Is Changing the Cost Conversation

As organizations adopt more agentic workflows, the amount of work delegated to AI can grow. These tasks can be much more token-hungry than a simple, isolated prompt. When that pattern combines with the pace of adoption and the current focus on using more tokens, costs can rise quickly.

It is tempting to focus only on the price of an individual model call. But a useful cost conversation starts by looking at the work being done and the choices made along the way. Teams need to consider whether a model is well matched to a particular task and whether every request needs the same kind of model. The answer may vary by task, and a single default choice may not be the right fit for all of them.

This is why cost management is becoming a practical part of AI deployment, not a concern reserved for later. It is a design question that sits alongside the decision to adopt agentic systems in the first place.

Model Selection Is a Portfolio Decision

At the conference, I discussed how leading AI practitioners, including teams at AT&T, are responding to cost pressure. Their approaches include model routing and deploying a suite of open models internally. These examples reflect a move away from treating model selection as a one-time decision. Instead, it can be understood as a portfolio of choices made in light of the task at hand.

At Datasaur, we have a playbook for model selection and use a number of tools to bring significantly smaller models to frontier-level intelligence. The underlying aim is to make model choice more deliberate: smaller models can be part of the solution when the right approach helps them meet the required level of performance. The specifics depend on the work being done, which is why thoughtful selection matters.

This does not mean that one model or one routing strategy is right for every organization. It does mean that teams can look beyond a simple largest-model-by-default approach. A considered mix of models, paired with the tools needed to use them effectively, can help organizations address both capability and cost.

Evals and Knowledge Graphs Help Ground AI

Another clear theme at ODSC was automated evaluation. Public benchmarks can be useful reference points, but they do not necessarily answer the questions an organization has about its own work. Teams need ways to evaluate models against the tasks and requirements that matter to them. Building those evaluations is becoming an essential part of understanding whether an AI system is useful in a specific setting.

Knowledge graphs were also back in the conversation. The interest makes sense: knowledge graphs developed through traditional machine learning can help provide context to modern LLMs. That context is important because a model's answer is shaped not just by the model itself, but also by the information available to it.

Evals and knowledge graphs address different needs, but both connect AI capability to organizational reality. One helps teams assess performance against their own requirements. The other can help supply useful context. Neither should be treated as a passing trend when organizations are working to make AI more relevant to their operations.

The Full Cost Includes the Work Around the Model

Token costs are visible, but they are only part of the total effort involved in deploying AI. Governance and observability matter too, as do evaluation and the work of providing useful context. As these requirements become part of the deployment picture, organizations may find that operationalizing AI takes a broader, coordinated effort.

That is why deploying AI can involve building a full-fledged department, rather than simply purchasing access to a model. The responsibility spans model selection, evaluation, governance, and observability. These are not unrelated extras. Together, they shape how AI is introduced and supported within an organization.

A more complete cost conversation therefore asks not only, “What does this model call cost?” It also asks what needs to be in place for an AI system to be effective and manageable. Those questions help bring the work around the model into view.

A Broader View of AI Deployment

The discussions at ODSC reinforced that the economics of AI are about more than tokens. Rising demand makes model selection important. Organization-specific evals help teams assess what works for their needs. Knowledge graphs can contribute context, while governance and observability are part of the operational foundation.

These themes are closely connected. When teams consider them together, they can make more informed decisions about deploying AI and the work required to support it. That is the larger conversation I wanted to bring to ODSC, and one that will continue as agentic adoption grows.