LLM Inference Is a Memory Problem, Not a Compute Problem

Frequently Asked

Why does the KV cache become a bottleneck during LLM inference specifically?

During inference, each new token must attend to every prior token in the context window, which requires loading the full KV cache from memory on every generation step. The arithmetic involved is modest relative to that data movement — so the GPU sits idle waiting for memory transfers, not waiting for compute. Longer contexts and larger batch sizes make this worse, not better.

What should infrastructure teams ask NVIDIA about Vera before committing to a deployment?

The key unanswered question is latency under mixed agentic workloads — Vera's 1.2 TB/s bandwidth spec addresses throughput, but production inference involves irregular batch sizes and context lengths that rarely match benchmark conditions. Teams should demand real-workload benchmarks on their specific model families before treating Oracle's hyperscale commitment as validation for their own use case.

What is the strongest argument that FLOPs still matter more than memory bandwidth for AI hardware decisions?

Training runs — which still determine frontier model capabilities — remain compute-bound, and organizations that expect to train as well as serve at scale cannot optimize purely for inference bandwidth. Vera is an orchestration and inference chip; it does not displace H100/H200 class GPUs for the training workloads that define the research roadmap. The inference-bandwidth argument is correct for serving, not for the full hardware stack.

Continue reading