Patel notes DeepSeek's routing innovation removing auxiliary loss represents compounding small improvements over time.
“this type of change can be big, it can be small, but they add up over time.”
Patel explains AI hardware companies made architectural bets that failed when model architectures evolved in unexpected directions.
“But still the model's way too big to fit on it. This is, like, very simple. Right? You know, the same thing's happening in the other direction.”
Patel reveals at least two AI hardware companies made wrong bets as models shifted to sparse MOEs.
“This is actually the case with at least two of the AI hardware companies today. Don't shittock them just because, you know, it's a let's be friendly.”
Patel reveals NVIDIA's next generation will split inference into separate context processing and decode workloads, not training versus inference.
“NVIDIA's next generation actually has something very different. They're not saying, Hey, there's a training GPU and an inference GPU, right? Because either is fine.”
Patel says OpenAI models are much more sparse while Anthropic models are denser, with different architectural benefits.
“Open eyes are much more sparse and that has benefits. Then anthropics are, they're still sparse but more dense in general and that has different benefits.”