summary Summary

US companies are quickly bringing in generative AI systems and greatly raising their spending. But the biggest problem is still figuring out how much money they will get back on their investment.

Generative AI has caused a big spending surge among top US companies. A survey by Andreessen Horowitz of 70 leaders from Fortune 500 companies shows that almost all the companies asked plan to spend 2-5 times more on large language models (LLMs) and other AI systems by 2024.

In 2023, companies spent about $7 million each on base model APIs, running models themselves, and improving models. But now, large, ongoing software budgets are being used for this, not short-term innovation funds.

Measuring ROI remains a major challenge

Some companies are already paying for AI using money saved on customer service staff. One company said it cut costs for calls by 90% ($6 per call) with LLM help, so it made its AI budget eight times bigger.


Even with large investments, companies aren't sure how to tell if AI spending is worth it. "While leaders are figuring out the answer to this question, many are taking it on faith when their employees say they’re making better use of their time," said the analysts at Andreessen Horowitz.

Measuring the success of generative AI remains a challenge. | Picture: a16z

Even if employees can work more efficiently and with higher quality, as a study by Ethan Mollick and others suggests, the question remains whether they are actually passing on these gains in their day-to-day work. Companies often lack meaningful metrics to reliably measure success.

Productivity increases are seen as a positive indicator, reports a16z, but leaders want clearer signs like more sales, lower costs, better efficiency and accuracy - based on how AI is used.

Clear ways to track AI gains will likely stay hard for 2-3 more years. Until then, AI spending will grow even if business benefits aren't always clear.

Multimodel AI growing

Cloud AI models are still widely used. Over half the surveyed companies use models hosted by their cloud provider.  And companies are also starting to use several AI models to improve their apps and rely less on single providers. Open-source models are becoming more popular since companies feel they offer more control and customization.

The number of models used in the companies surveyed. | Picture: a16z

Still, models from big providers like OpenAI remain key. OpenAI is far ahead (~40%) of Google, Llama, Anthropic, Mistral and Cohere. For customizing AI models, fine-tuning (72%) beats database chats (RAG, 22%). Only 6% of companies make models from scratch.

According to the report, companies are focusing on building popular AI applications, such as chatbots, in-house rather than buying market solutions. The most common applications are still aimed at making workers more productive.

Internal applications dominate. |Image: a16z

The availability of foundational AI models would reduce the amount of work needed to create useful applications in-house. Organizations should also experiment with more specific interfaces tailored to their needs.

Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
  • According to an Andreessen Horowitz survey of 70 Fortune 500 executives, U.S. companies plan to increase their spending on artificial intelligence (AI), such as large language models (LLMs), by two to five times by 2024.
  • Despite the high level of investment, measuring the success and profitability of AI applications remains a challenge, as companies often lack meaningful metrics.
  • To optimize their applications and reduce dependency on single providers, companies are increasingly turning to multiple AI models, including open-source models.
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.