The Real Price of Building on Foundation Models

Photo by Gustavo Fring via Pexels

The average AI startup burning $50,000 a month on inference in 2024 is spending closer to $180,000 today, and that number is not falling fast enough to matter. The promise sold to founders was simple: rent the intelligence, ship the product, capture the margin. The reality is a cost structure that scales with usage in ways that make traditional SaaS unit economics look quaint. When OpenAI, Anthropic and Google control the underlying model, they also control the single largest variable in your cost of goods sold. That is not a technical problem. It is a structural one, and most cap tables have not priced it in.

The shift from experimentation to production is where the trap closes. During the prototype phase, token costs are invisible because volume is low and investor capital absorbs the friction. The moment a product hits scale, inference costs surface as the dominant line item, often exceeding hosting, headcount and tooling combined. OpenAI's pricing on GPT-4o has come down meaningfully since launch, but the cadence of those reductions has slowed, and the models founders actually need for production-grade reasoning remain expensive. Meanwhile, Anthropic's Claude 3.5 Sonnet and Google's Gemini 1.5 Pro have introduced genuine capability competition, which sounds like good news for buyers until you realise that switching costs between foundation models are far higher than the API pricing tables suggest.

The switching cost problem is underappreciated and underreported. Prompts engineered for GPT-4o do not port cleanly to Claude or Gemini. Evaluation frameworks, fine-tuning workflows and retrieval-augmented generation pipelines are all model-specific in ways that take weeks of engineering time to rebuild. A well-funded startup like Harvey, which has raised over $300 million and built deep integrations on top of OpenAI's stack, faces a meaningful re-platforming cost if it ever needs to move. Smaller companies with less runway face the same problem with none of the buffer. The venture community funded dozens of companies in 2023 and 2024 on the assumption that model commoditisation would compress input costs toward zero. That assumption is increasingly hard to defend in 2026.

Model dependency is not a technical risk. It is a business model risk, and most AI startups have not priced it into their cap tables.

There is a second hidden cost that almost nobody talks about in fundraising decks: model deprecation risk. OpenAI has already sunset GPT-3.5 Turbo in its original form, forcing dependent applications to migrate on a timeline set by the provider, not the customer. Google has deprecated multiple Gemini model versions inside twelve months. Every deprecation event triggers an unplanned engineering sprint, regression testing across the full product surface and, in many cases, a prompt re-engineering cycle that can consume two to four weeks of senior engineering time. For a twenty-person startup, that is not a maintenance cost. It is a strategic disruption. Founders who baked zero deprecation overhead into their roadmaps are now quietly repricing it as a recurring line item.

The investors who spotted this earliest are now pushing portfolio companies toward one of three defensive strategies. The first is model abstraction, building an internal routing layer that can shift workloads between providers based on cost, latency and capability, a pattern that companies like LangChain and LlamaIndex have tried to productise at the infrastructure level. The second is fine-tuning on smaller open-source models, using Mistral, Meta's Llama 3 family or Cohere's Command R series to handle high-volume, low-complexity tasks at a fraction of the per-token cost, reserving frontier models only for tasks that genuinely require them. The third, and most capital-intensive, is building proprietary model capability in-house, a path that makes sense for companies with large, unique datasets and the engineering depth to exploit them, but is a distraction for everyone else. None of these strategies is free, and all of them require a deliberate architectural decision that most early-stage teams are not making early enough.

The next twelve months will force a reckoning that the bull market in AI applications has so far allowed founders to defer. As interest rates on venture debt remain elevated and pressure mounts on path-to-profitability, investors will start demanding gross margin visibility that most foundation-model-dependent businesses cannot yet provide. The companies that survive and scale will be the ones that treated model dependency as a first-order architecture decision from day one, not an optimisation problem to be solved at Series B. The hidden costs were never really hidden. They were just easy to ignore when capital was cheap and conviction was high.