Patel says o1 generates 40k sequence lengths versus 4k for standard models, requiring lower batching and higher prices.
“If you ask it to, like, generate a web scraper, in the standard one it'll just start outputting code, and it'll be maybe like a four k sequence length.”
Patel explains o1's thinking time creates memory bandwidth issues that prevent batching users at high levels.
“But if you batch higher, k b cache is not just a memory capacity issue, it's also a memory bandwidth issue.”
Patel reveals o1's reasoning process sometimes switches between Chinese and English during hidden thinking phase.
“It generates tons of things. It's like it it sometimes switches between Chinese and English. Right? Like, whatever it is. It's thinking. Right?”
Patel calculates o1 reduces concurrent users per server to one-fourth or one-fifth versus standard models.
“E. Concurrent users I can have is a fraction of that. One fourth to one fifth the number of users can currently use the server.”
Patel argues export controls primarily limit AI inference deployment in China, not frontier model training capabilities.
“A large part of export controls, if they work, is just that the amount of AI that can be run-in China is going to be much lower.”
Patel notes OpenAI is releasing custom kernels with model weights so everyone has optimized inference day one.
“Make your own inference implementation, But OpenAI is, like, actually, like, dropping the model weights and, like, all these custom kernels for people to implement in inference.”
Patel predicts AWS revenue growth will reaccelerate after consistent deceleration due to Anthropic and new data centers.
“AWS has been decelerating revenue. Year on year revenue has been falling consistently. And and our big call is that it's actually going to start reaccelerating.”
Patel confirms OpenAI runs production inference on GB200 despite reliability requiring workloads handle 64 of 72 GPUs.
“OpenAI has said they're running production inference on GV200 a couple of months ago, in fact. Right?”
Patel reveals NVIDIA's next generation will split inference into separate context processing and decode workloads, not training versus inference.
“NVIDIA's next generation actually has something very different. They're not saying, Hey, there's a training GPU and an inference GPU, right? Because either is fine.”
Patel says B200 is better for training while GB200 is better for inference, reversing expected use.
“And so you've sort of Which is the exact opposite of what you would have expected. Oh, use the big thing for training and use the small thing for inference.”
Patel explains NVIDIA is splitting inference into decode and prefill workloads, but provisioning for unknown future ratios is challenging.
“what sort of NVIDIA's pitching is like a split of inference into two workloads, and we'll see if they're successful. There's a lot of challenges on the infrastructure side.”
Patel announces InferenceMax is completely open source under Apache 2.0 license for full transparency.
“InferenceMax is aiming to change all of that in the benchmarking space. So it's completely transparent in that it's completely open source, Apache 2.0 license.”
Patel says avoiding repeated pre-fill on 30,000-token inputs can reduce inference costs by 75% through KV cache.
“So then that 30,000 tokens, that pre fill, that 30,000 tokens still cost a lot more than that output.”
Patel cites DeepSeek's open-sourced inference system requiring 140 GPUs communicating over RDMA networks for single replica.
“One example is one that DeepSeek open sourced over December of last yearJanuary, February of this year, where a single replica of inference for a single model is going to be like 140 GPUs.”
Patel explains users will pay 10x more for 10x faster inference completion, justifying Cerberus economics.
“for a lot of people, I'm fine to spend 10x the price on something that completes 10x faster. So Cerberus sort of just makes a ton of sense there.”
Patel claims code agent revenue grew from a couple billion to over $10 billion in a very short time.
“code agent revenue has gone from a couple billion to north of 10,000,000,000 in like a very short amount of time. And these, the horizon of these has also increased dramatically,”
Patel claims network performance drives 5x to 10x difference in inference for models using disaggregated PD or wide EP techniques.
“And so if you look into inference x, it turns out network performance drives a not just a, hey, 30%. It's actually multiple, like, x's.”
Patel identifies disaggregated PD and wide EP as networking techniques where performance matters significantly.
“When someone has really good networking with these leading edge techniques to run a model such as KIMI or DeepSeq called disaggregated PD or wide EP, these are two different techniques, network performance is a humongous factor.”
Patel notes software dependencies change daily or multiple times weekly across the entire stack.
“And furthermore, with software changing literally multiple times a week, right, PyTorch has nightlies, VLM has nightlies, Asteeling has nightlies, CUDA drivers update constantly. You just go You go through the whole list.”
Patel explains AI software stacks update multiple times per week making performance measurement a moving target.
“with software changing literally multiple times a week, right, PyTorch has nightlies, VLM has nightlies, Asteeling has nightlies, CUDA drivers update constantly. You just go You go through the whole list.”
Patel predicts reasoning chains will close as AI response times extend from immediate to hours or days.
“And then as the horizon of AI models gets longer, right, rather than a question answer immediately, the question answer becomes ten minutes, hours, days, the value of all the reasoning is gone.”
Patel distinguishes OpenAI hiding reasoning chains from Anthropic showing them in code generation models.
“In the case of OpenAI, they don't show you the whole reasoning chain. In the case of Anthropic, they do.”
Patel notes AI inference libraries update twice weekly, causing constant performance changes.
“In fact, the update cycle for most of these libraries is twice a week, so you basically have the software updating all the time and therefore performance changing.”
Patel explains SemiAnalysis publishes public research on GPU benchmarking to build scale beyond institutional client work.
“Let's figure out how to figure it out and just post it publicly, and that begets more and more scale. So we've done this with a lot of GPU benchmarking and testing”
Patel predicts OpenAI and Anthropic will have over 100 gigawatts combined by 2030, terawatts by 2040.
“I think by 2030, just OpenAI and Anthropic will have over 100 gigawatts combined, and then you'll add Meta and Google and so on and so forth.”
Patel predicts OpenAI and Anthropic alone will deploy over 100 gigawatts of compute by 2030.
“by 2030, just OpenAI and Anthropic will have over 100 gigawatts combined, and then you'll add Meta and Google and so on and so forth.”
Patel reports cache hit rates above 95% for many agentic workflows in production.
“we're seeing cache hit rates above 95% for many AgenTeq workflows, which which means the cost for a cache hit is it's not free,”
Patel explains KV cache storage and reuse enables massive cost decreases in inference.
“you calculate that once, you store it off in memory, whether it be system memory or storage, and then you pull it back in when you run the turn.”
Patel describes agentic workflows using 30,000-100,000 input tokens but generating only 1,000 output tokens.
“And and that makes, you know, initially, that'd be like, okay, well now I need a ton ton of compute to calculate all the prefilled tokens.”
Patel argues that if model progress pauses while compute supply grows, demand growth will slow and prices will collapse.
“If model progress at the labs pause, then more compute comes online. It has to slow down. Right? Sort of right now we have supply demand, right?”
Patel argues labs will allocate less compute to inference over time, contrary to consensus belief.
“So, this is sort of what I personally believe that the labs are going to allocate less and less compute to inference over time, I think is very non consensus, right?”
Patel says anyone can profitably run inference by renting GB300 racks and deploying open models.
“Go download the Kimi weights. Go download VLM or SGLANG. Set it up. You know, Codecs and Fable can actually help you do this.”