Patel states mixture of experts architecture can achieve the same model performance with 30% less compute.
“So you can get effectively the same performance model and evaluation scores with numbers like 30% less compute.”
Patel notes DeepSeek's routing innovation removing auxiliary loss represents compounding small improvements over time.
“this type of change can be big, it can be small, but they add up over time.”