Patel explains that Apple's rushed model has empty experts not receiving tokens due to flawed sparse MOE routing.
“Basically in between every layer, the router can route to whatever expert it wants to, And it learns which expert to route to. And and each expert learns its own independent things”
Patel describes the Bumpgate incident where NVIDIA laptop GPUs had defective solder balls.
“There's a generation of NVIDIA GPUs for laptops. Right? And and chips have solder balls on the bottom. Right?”
Patel explains Bumpgate issue where thermal expansion differences caused GPU solder ball failures in Apple laptops.
“And what ended up happening is because of that different rate of expansion, the solder balls connecting the chip and the board would crack.”