US companies are quickly bringing in generative AI systems and greatly raising their spending. But the biggest problem is still figuring out how much money they will get back on their investment.
Generative AI has caused a big spending surge among top US companies. A survey by Andreessen Horowitz of 70 leaders from Fortune 500 companies shows that almost all the companies asked plan to spend 2-5 times more on large language models (LLMs) and other AI systems by 2024.
In 2023, companies spent about $7 million each on base model APIs, running models themselves, and improving models. But now, large, ongoing software budgets are being used for this, not short-term innovation funds.
Measuring ROI remains a major challenge
Some companies are already paying for AI using money saved on customer service staff. One company said it cut costs for calls by 90% ($6 per call) with LLM help, so it made its AI budget eight times bigger.
Even with large investments, companies aren't sure how to tell if AI spending is worth it. "While leaders are figuring out the answer to this question, many are taking it on faith when their employees say they’re making better use of their time," said the analysts at Andreessen Horowitz.
Even if employees can work more efficiently and with higher quality, as a study by Ethan Mollick and others suggests, the question remains whether they are actually passing on these gains in their day-to-day work. Companies often lack meaningful metrics to reliably measure success.
Productivity increases are seen as a positive indicator, reports a16z, but leaders want clearer signs like more sales, lower costs, better efficiency and accuracy - based on how AI is used.
Clear ways to track AI gains will likely stay hard for 2-3 more years. Until then, AI spending will grow even if business benefits aren't always clear.
Multimodel AI growing
Cloud AI models are still widely used. Over half the surveyed companies use models hosted by their cloud provider. And companies are also starting to use several AI models to improve their apps and rely less on single providers. Open-source models are becoming more popular since companies feel they offer more control and customization.
Still, models from big providers like OpenAI remain key. OpenAI is far ahead (~40%) of Google, Llama, Anthropic, Mistral and Cohere. For customizing AI models, fine-tuning (72%) beats database chats (RAG, 22%). Only 6% of companies make models from scratch.
According to the report, companies are focusing on building popular AI applications, such as chatbots, in-house rather than buying market solutions. The most common applications are still aimed at making workers more productive.
The availability of foundational AI models would reduce the amount of work needed to create useful applications in-house. Organizations should also experiment with more specific interfaces tailored to their needs.