Patel explains that Apple's rushed model has empty experts not receiving tokens due to flawed sparse MOE routing.
“Basically in between every layer, the router can route to whatever expert it wants to, And it learns which expert to route to. And and each expert learns its own independent things”
Patel describes Apple's model having empty experts that never get routed to, indicating wasted training capacity.
“if you actually look at the model, oftentimes won't even route tokens to certain experts. So it was like a waste of training.”
Patel explains how MOE routing works and how some of Meta's experts weren't being used at all.
“And and each expert learns its own independent things and it's like really not something observable by people. But what you can see is tokens when which experts do they route to?”