Spotify says its TensorRT-LLM work cut podcast recommendation beam search costs up to 54%

Spotify says the beam search step of the language model behind its podcast recommendations can cost up to 54% less to run after the company and NVIDIA added missing features to NVIDIA's open source TensorRT-LLM inference library. The other half of the job, scoring candidate podcasts, got a much smaller saving of up to 7%, according to an October 16 post written jointly by Spotify and NVIDIA engineers.

The gap says a lot about where the money goes when a recommender is built from a language model rather than a classic ranking system.

Why Spotify ran the model on cheaper chips

Spotify started serving LLM-generated podcast recommendations in January 2026. The model generates candidates as semantic IDs, short token codes that stand for podcasts, using beam search, which keeps several possible answers alive at each step instead of just the most likely one.

On vLLM, beam search ran outside the core GPU engine. That caused bottlenecks on the CPU side and extra GPU work to recompute partly filled blocks of the attention cache. Because the expensive GPUs were not being kept busy, Spotify served the launch on lower-tier accelerators to hold down costs. The post does not name the chips.

Teaching the model who you are without a wall of text

Personalisation was the other problem. Feeding a listener's history to the model as plain text was the obvious approach, but the authors write that "the major issue with fully text based user history is the amount of text required to accurately represent the user." Long prompts pushed latency past the request budget and raised GPU use.

The method that worked best was soft tokens: Spotify takes user embeddings it already has and projects them into the language model's embedding space, so a person is represented by a few vectors rather than paragraphs. Giving each user their own semantic IDs looked promising but is not ready for production, the post says.

What had to change in TensorRT-LLM

TensorRT-LLM handled beam search well but lacked a scoring API, a way to inject soft tokens, and the ability to return several beams at once. The two companies added those features to the open source project across releases 1.2 and 1.3, including multi-item scoring, which avoids sending one request per candidate podcast.

The cost figures quoted are for a 1.7 billion parameter variant of Alibaba's Qwen3 model. The post does not name the baseline they are measured against, and it does not say outright that production traffic now runs on TensorRT-LLM. It also gives no absolute costs, latency numbers or GPU fleet size, and its benchmark charts were published only as images.