Patel explains DeepSeek v3 base is trained once, then post-trained differently to create chat versus reasoning models.
“This reasoning model has a lot of overlapping training steps to DeepSeek v three, and it's confusing that you have a base model called v three that you do something to to get a chat model, and then you do some different things to get a reasoning model.”
Patel explains training a 70 billion parameter model requires passing four times that data every training step without overlapping communication.
“So if you're training like a 70,000,000,000 parameter model, you need to pass four x that number every single training step. And when you're passing that, you can't really overlap communications and compute.”
Patel says GPT-5 will be first model to massively scale both pre-training and post-training simultaneously.
“Right? And this would be the first time we see a model that was was a step up in both at the same time.”
Patel explains that Apple's rushed model has empty experts not receiving tokens due to flawed sparse MOE routing.
“Basically in between every layer, the router can route to whatever expert it wants to, And it learns which expert to route to. And and each expert learns its own independent things”
Patel describes Apple's model having empty experts that never get routed to, indicating wasted training capacity.
“if you actually look at the model, oftentimes won't even route tokens to certain experts. So it was like a waste of training.”
Patel explains how MOE routing works and how some of Meta's experts weren't being used at all.
“And and each expert learns its own independent things and it's like really not something observable by people. But what you can see is tokens when which experts do they route to?”
Patel argues three years of improvement came from model layer advances plus hardware-software co-design, citing smaller Qwen models outperforming GPT-4.
“If you look back three years, it's GPT-four. Now it's maybe like quen, one of the smaller quen models that's like 27B parameters total and 2,000,000,000 active is way better.”
Patel differentiates OpenAI's sparse model architectures from Anthropic's denser approach with distinct trade-offs.
“I think they're quite different model architectures, in fact. Interesting. Open eyes are much more sparse and that has benefits.”