Patel explains OpenAI's o1 model thinks for 5-20 seconds before outputting, creating inference throughput challenges.
“When you use OpenAI's o one, it thinks for ten seconds, twenty seconds, five seconds. It varies a lot, but thinks for a while and then it sends you tokens.”
Patel says o1 generates 40k sequence lengths versus 4k for standard models, requiring lower batching and higher prices.
“If you ask it to, like, generate a web scraper, in the standard one it'll just start outputting code, and it'll be maybe like a four k sequence length.”
Patel states OpenAI's o1 reasoning model increases average sequence length because it reasons before outputting.
“This is really important because with reasoning models like OpenAI's o one, the average sequence length grows a lot.”
Patel explains o1's thinking time creates memory bandwidth issues that prevent batching users at high levels.
“But if you batch higher, k b cache is not just a memory capacity issue, it's also a memory bandwidth issue.”
Patel explains o1's 10,000 token thinking phase requires constant memory-intensive KV cache generation and reading.
“When I do this with o one, right, because it's doing that thinking phase of 10,000, right, this is this is basically the whole context length thing.”
Patel reveals o1's reasoning process sometimes switches between Chinese and English during hidden thinking phase.
“It generates tons of things. It's like it it sometimes switches between Chinese and English. Right? Like, whatever it is. It's thinking. Right?”
Patel calculates o1 reduces concurrent users per server to one-fourth or one-fifth versus standard models.
“E. Concurrent users I can have is a fraction of that. One fourth to one fifth the number of users can currently use the server.”
Patel describes o1's reasoning process as generating extensive internal thought including language switching before outputting.
“It generates tons of things. It's like it it sometimes switches between Chinese and English. Right? Like, whatever it is. It's thinking. Right? It's churning.”
Patel calculates o1 models increase inference costs by roughly 50x compared to standard models per query.
“Cost increase for a single token to be generated is four to five x, but then I'm generating 10 x as many tokens.”
Patel states Microsoft's inference costs are a few billion for ten billion in revenue but o1 multiplies costs 10x.
“Their cost for that is a few billion dollars for $10,000,000,000 of revenue. Right. If now, obviously, the better model gets to charge more. Right?”