Patel explains o1's 10,000 token thinking phase requires constant memory-intensive KV cache generation and reading.
“When I do this with o one, right, because it's doing that thinking phase of 10,000, right, this is this is basically the whole context length thing.”
Patel says avoiding repeated pre-fill on 30,000-token inputs can reduce inference costs by 75% through KV cache.
“So then that 30,000 tokens, that pre fill, that 30,000 tokens still cost a lot more than that output.”
Patel states avoiding redundant prefill through KV cache can cut inference costs to one-fourth of previous levels.
“So you can cut your cost to one fourth of what it was previously if you just don't do the pre fill. Right?”
Patel observes Asian researchers consistently reference KV cache reduction papers unknown to American researchers, citing DeepSeek and TurboQuant.
“It's the fucking best thing ever. DeepSeek and TurboQuant were the, like, most precipitous ones that popped up the most,”
Patel observes Asian researchers consistently produce KV cache reduction papers unknown to American researchers.
“Some paper that reduces KV cash, and no researcher in America has even heard of this paper.”
Patel explains KV cache storage and reuse enables massive cost decreases in inference.
“you calculate that once, you store it off in memory, whether it be system memory or storage, and then you pull it back in when you run the turn.”
Patel argues layering KV cache offload and multi-token prediction on open-source engines drastically cuts inference costs.
“So if you take the, you know, just open source inference engines off the shelf, that gets you a certain level of cost.”