How Spotify runs agents across 20M+ lines of code, with Niklas Gustavsson

Study Guide

Overview

In this Anthropic interview, Niklas Gustavsson (a senior engineering leader at Spotify, trained originally as a molecular biologist with nearly 30 years in the industry) describes how Spotify deploys coding agents across a codebase exceeding 20 million lines of code. He walks through the evolution from deterministic migration scripts to Spotify's agent platform HONK, and argues that the durable foundations of good engineering, such as verification, test automation, and standardization, are exactly what make AI-driven development work at scale.

Key Takeaways

  • The shift was faster and deeper than expected. Niklas predicted no one would be using a traditional IDE by year-end; two months later he had stopped using one himself, calling it a change he had not seen in 30 years of this work.
  • Opus 4.5 was the personal breakthrough. Models went from "smart autocomplete" to something he could throw real problems at, eliminating the "last mile" IDE edits he previously always had to make.
  • Monorepos work great with agents. Despite early worries about indexing issues, Claude works "amazingly well" in repos over 20M lines, in part because it draws inspiration from existing code in the repository.
  • Verification is the single most important investment. In closed-loop, human-out-of-the-loop development, under-investing in verification is the common mistake Niklas sees companies make.
  • Standardization helps agents the same way it helps humans. Consistent code, tools, and frameworks reduce agent confusion; inconsistent code (the same thing done 10 ways) confuses Claude.
  • Measured impact is large and direct. A 75%+ improvement in PR frequency attributable to AI tooling, and ~73% of PRs now AI-authored.

From Migration Scripts to HONK

Five to six years ago, Spotify realized its codebase was growing roughly seven times faster than its engineering headcount, leaving the company bogged down in maintenance. They built an infrastructure called Fleet Management to apply mutations across the entire codebase (thousands of repos) instead of asking hundreds of teams to perform the same migration manually over thousands of components.

The early approach relied on deterministic scripts and static analysis (AST transformation). This hit a ceiling fast: code has an enormous API surface, and even swapping one method could require variable and state tracking, turning each script into thousands of lines handling edge cases. Migrations took "months and months," with maybe 10 completed per year.

LLMs were applied early but naively (just putting code in front of the model and asking for a one-shot change), and it "didn't work at all that well." Over time, with better models plus LLM-as-judge, problem decomposition, and many internal iterations, the work consolidated into HONK. The product released as "V2" is realistically more like V8, but the iterations weren't all tracked.

HONK Architecture

  • Built on the Claude Agent SDK, running in a Kubernetes pod.
  • The agent has access to a set of tools. Pre-V2 these were a predefined, allow-listed, trusted set; now users can add their own internal tools.
  • A critical tool runs verification via CI builds, on both Linux and macOS (macOS matters for iOS development).
  • The agent can go as deep as integrating the iOS simulator with Claude, for example going from designs in Figma to UI implementations, and is being used for porting (such as TV apps from iOS apps).
  • The original LLM judge was removed. It once lifted PR success rates from roughly 20-30% to 80%, but models (again, 4.5) and the agent harness caught up, so it was no longer needed.

Verification, Test Automation, and Quality vs. Speed

Because Spotify now auto-merges most automated PRs without the owning team ever seeing them, it had to change team expectations and dramatically strengthen test automation. The codebase is split into thousands of components, each with well-defined ownership; previously teams could be "a bit sloppy" on testing because they reviewed every PR. That no longer holds.

Niklas frames the reliability-vs-speed trade-off as a false dichotomy: to go faster, you encode quality practices into infrastructure (skills, CLAUDE.md, MCPs) rather than leaving them in someone's head. Productivity gains come from investing in infrastructure, not working more hours. Spotify keeps quality metrics neutral while significantly improving speed, but this has not come for free. With ~4,500 production deployments per day, continued investment in reliability is required, and the faster they go, the more they must invest.

Measuring ROI

Early ROI was easy to see given large, obvious improvements. Across roughly 2,900 engineers, Spotify reports a 75%+ improvement in PR frequency attributable to AI tooling and ~73% of PRs AI-authored. The harder, ongoing work is connecting engineering deliverables (PRs, deployments) to "work items", then to A/B tests and rollouts, and ultimately back to user value and revenue. As maturity and cost precision expectations rise, attribution down to tokens and hours per unit of productive output matters more.

Prototyping as a Major Unlock

Spotify built infrastructure so anyone (engineers and non-engineers) can express an idea in natural language and have Claude implement an end-to-end prototype in the real mobile apps and backend within an hour or two. There is an internal app store for sharing and trying out each other's prototypes with real data. Adoption spans the org: even one of the co-CEOs and senior leaders have prototypes in the store. Ideas that once required motivating an engineering team can now be tested in a day instead of weeks or months.

Personal and Career Perspective

Niklas, who does competitive programming for fun, worried he would miss the hard mental challenge of solving problems by hand. He found he was wrong: what he loves is solving problems, and the specific method of solving them turned out not to be the critical piece. He now runs about five agents in the background, is more productive, and can jump into unfamiliar codebases (that once took days or weeks) and contribute things he simply could not before. His time has shifted from implementation toward thinking about what's next, talking to customers, and far more prototyping.

Advice to Leaders and Engineers

  • To CTOs / VPs of Engineering: Do not ignore foundational investments. Test automation, verification, and standardization (consistent codebases, aligned tools and frameworks) all transfer extremely well from helping humans to helping agents. The same engineering practices still apply, even with "a new actor in your code base."
  • To engineers: Give it a real shot and find the way these tools fit how you like to work. Focus on the problems you can solve rather than the method. The transition will look different for different people.

Notable Quotes & Data Points

  • "Two months later, I found myself not using an IDE anymore... a change I had not seen in the 30 years I've been doing this type of work."
  • Opus 4.5 "went from being this like smart autocomplete to something that I could actually throw real problems at."
  • Codebase grew ~7x faster than headcount; the monorepo is 20M+ lines of code.
  • Judge lifted PR success from ~20-30% to ~80% in early HONK, later removed.
  • ~4,500 production deployments/day; ~2,900 engineers.
  • "75% plus improvement in PR frequency... 73% of PRs are directly attributed to being AI authored."
  • "The more consistency we have, the better our agents work."
YouTube