Aiven builds LLM gateway after cutting token costs with open weight models

European open source database service provider Aiven says it has saved 80 percent of its AI token costs internally by moving to open weight models. The Finland-based company built its business on hosting and providing open source databases as a service, and in the last couple of years has moved into machine learning. Speaking to The Register, Ville Lehto, Aiven’s VP of strategy, said that when dealing with open source AI infrastructure, it is not trivial to make something that can run across the different clouds. However, in giving its developers access to LLMs, the company wanted to manage its own token costs and control its own destiny when it comes to AI usage.

How the gateway works

After a wave of debate around token-maxxing, developers have generally become more interested in reducing their costs by using a mix of models, including cheaper Chinese alternatives to Western frontier models, and open weight models which come at a lower cost. Open weight models have publicly released learned parameters, principally their weights and biases, and can be run locally or hosted at lower cost than frontier models from OpenAI and Anthropic. Examples include the Llama family from Meta, the Qwen family from Alibaba Cloud, China’s DeepSeek, Mistral AI’s Medium 3.5 and Devstral, and Google’s Gemma. Lehto said that when it comes to token use, a fraction of users are heavy users, and a lot of their spend could be saved by using open weight models. He said Aiven faced a choice between strict token budgets or a more efficient way of using tokens, and opted for the latter by implementing an internal gateway.

Aiven built its gateway around the open source framework called LiteLLM, which it hosted. It added features such as budgeting and access management, and added other open source components. It runs the models on rented GPUs and self-hosts the models as well. Lehto said Aiven continued to use frontier models but only to help plan work. When it comes to actually doing things, it uses a lot of open weight models. The majority of the work is done with open weight models, but the most critical bits are planned with frontier models. However, since the performance of open weight and frontier models varies, it was also important to use other metrics, such as the cost per task, he said. The company says it has seen roughly an 80 percent decline in spend among heavy users.

Customers and plans

Aiven plans to offer the gateway as a service to customers to help them manage token costs by making it easier to swap open weight models in and out of their workflows. Its customers include German ecommerce company Conrad, UK payment processing company Dojo, cyber security vendor Sophos, and communications firm Avaya. Lehto said that since Aiven’s heritage is in open source, offering the gateway as a service fits enterprise needs. The company is set to be just one of the vendors positioning themselves to try to help organizations reduce costs when using AI.

Sources: The Register