Fireworks AI VP says on-prem AI can waste GPU spend as managed inference grows

Fireworks AI supports on-premises AI deployments, but VP Rob Ferguson argues that many companies should avoid running models in their own data centers because GPU hardware is expensive and costly if underused. The managed inference provider also says routing tasks across a portfolio of models, caching, data handling and speed can change the real cost of AI work.

On-premises AI and GPU economics

Ferguson told The Register that Fireworks AI is trying to push customers away from on-prem AI because it is just the economics. He said 10,000 GPUs represent a billion dollars of capital and that companies do not want those systems sitting idle. For sensitive workloads, Fireworks offers a service called Sovereign and has partners such as Microsoft, though Ferguson said most inference work probably does not need that level of protection.

Model routing instead of one model

Ferguson said the best model is a portfolio, not a single model such as Astra or Fable, and that routing between a few models is better than using one model alone. Fireworks does this through Nexus and its API, alongside competitors Together AI, Baseten and DeepInfra. The company says it is not doing naive routing like an AI gateway; it evaluates tasks as they arrive and directs them to the optimal model.

Fireworks builds custom routers, and its auto model continuously updates. It grades task difficulty and notes model affinities, such as Kimi being slightly better at JavaScript than Python. Ferguson attributed the company's edge to its engineering team and founders' work on PyTorch, saying the team understands numerics, mixture of experts topologies, floating point precision and routing through complex fabrics that may involve more than one provider.

Cost, data and speed

Ferguson said AI keeps coming back to economics, including how much work can go to cached or open models and when specialization is needed. He said frontier model companies are not transparent about caching rates, citing a data science company where Fireworks cached about 90 percent of tasks while Anthropic's Opus cached 30 percent. He said AI rework is the biggest cost, token cost is not the real cost, and customers should track cost per task.

Ferguson also said companies underestimate the value of their data. Fireworks has a zero data-retention policy, but he said other companies often do not, and customers should get value if their data is used. He said models are memorizers and there is evidence for that. Speed, he added, is underappreciated and changes how he works with AI. Ferguson claimed model ensembling can make a bill at least 50 percent smaller. He said Fireworks says six to eight times cheaper, while guaranteeing a bill drops by half when customers still use a mix of closed and open models.

Sources: The Register