Patel says anyone can profitably run inference by renting GB300 racks and deploying open models.
“Go download the Kimi weights. Go download VLM or SGLANG. Set it up. You know, Codecs and Fable can actually help you do this.”
Patel argues labs will allocate less compute to inference over time, contrary to consensus belief.
“So, this is sort of what I personally believe that the labs are going to allocate less and less compute to inference over time, I think is very non consensus, right?”
Patel argues that if model progress pauses while compute supply grows, demand growth will slow and prices will collapse.
“If model progress at the labs pause, then more compute comes online. It has to slow down. Right? Sort of right now we have supply demand, right?”
Patel describes agentic workflows using 30,000-100,000 input tokens but generating only 1,000 output tokens.
“And and that makes, you know, initially, that'd be like, okay, well now I need a ton ton of compute to calculate all the prefilled tokens.”
Patel explains KV cache storage and reuse enables massive cost decreases in inference.
“you calculate that once, you store it off in memory, whether it be system memory or storage, and then you pull it back in when you run the turn.”
Patel reports cache hit rates above 95% for many agentic workflows in production.
“we're seeing cache hit rates above 95% for many AgenTeq workflows, which which means the cost for a cache hit is it's not free,”
Patel predicts OpenAI and Anthropic will have over 100 gigawatts combined by 2030, terawatts by 2040.
“I think by 2030, just OpenAI and Anthropic will have over 100 gigawatts combined, and then you'll add Meta and Google and so on and so forth.”
Patel distinguishes OpenAI hiding reasoning chains from Anthropic showing them in code generation models.
“In the case of OpenAI, they don't show you the whole reasoning chain. In the case of Anthropic, they do.”
Patel predicts reasoning chains will close as AI response times extend from immediate to hours or days.
“And then as the horizon of AI models gets longer, right, rather than a question answer immediately, the question answer becomes ten minutes, hours, days, the value of all the reasoning is gone.”
Patel notes software dependencies change daily or multiple times weekly across the entire stack.
“And furthermore, with software changing literally multiple times a week, right, PyTorch has nightlies, VLM has nightlies, Asteeling has nightlies, CUDA drivers update constantly. You just go You go through the whole list.”
Patel explains AI software stacks update multiple times per week making performance measurement a moving target.
“with software changing literally multiple times a week, right, PyTorch has nightlies, VLM has nightlies, Asteeling has nightlies, CUDA drivers update constantly. You just go You go through the whole list.”
Patel identifies disaggregated PD and wide EP as networking techniques where performance matters significantly.
“When someone has really good networking with these leading edge techniques to run a model such as KIMI or DeepSeq called disaggregated PD or wide EP, these are two different techniques, network performance is a humongous factor.”
Patel claims code agent revenue grew from a couple billion to over $10 billion in a very short time.
“code agent revenue has gone from a couple billion to north of 10,000,000,000 in like a very short amount of time. And these, the horizon of these has also increased dramatically,”
Patel cites DeepSeek's open-sourced inference system requiring 140 GPUs communicating over RDMA networks for single replica.
“One example is one that DeepSeek open sourced over December of last yearJanuary, February of this year, where a single replica of inference for a single model is going to be like 140 GPUs.”
Patel confirms OpenAI runs production inference on GB200 despite reliability requiring workloads handle 64 of 72 GPUs.
“OpenAI has said they're running production inference on GV200 a couple of months ago, in fact. Right?”
Patel says B200 is better for training while GB200 is better for inference, reversing expected use.
“And so you've sort of Which is the exact opposite of what you would have expected. Oh, use the big thing for training and use the small thing for inference.”
Patel reveals NVIDIA's next generation will split inference into separate context processing and decode workloads, not training versus inference.
“NVIDIA's next generation actually has something very different. They're not saying, Hey, there's a training GPU and an inference GPU, right? Because either is fine.”
Patel explains NVIDIA is splitting inference into decode and prefill workloads, but provisioning for unknown future ratios is challenging.
“what sort of NVIDIA's pitching is like a split of inference into two workloads, and we'll see if they're successful. There's a lot of challenges on the infrastructure side.”
Patel reveals o1's reasoning process sometimes switches between Chinese and English during hidden thinking phase.
“It generates tons of things. It's like it it sometimes switches between Chinese and English. Right? Like, whatever it is. It's thinking. Right?”
Patel says o1 generates 40k sequence lengths versus 4k for standard models, requiring lower batching and higher prices.
“If you ask it to, like, generate a web scraper, in the standard one it'll just start outputting code, and it'll be maybe like a four k sequence length.”
Patel explains o1's thinking time creates memory bandwidth issues that prevent batching users at high levels.
“But if you batch higher, k b cache is not just a memory capacity issue, it's also a memory bandwidth issue.”