Patel identifies distributed training across data centers with lower bandwidth as a key unsolved problem that would unlock massive scaling.
“Everything that we've seen so far is that large scale training has to happen in an individual data center with very high speed networking.”
Patel advises becoming a foremost expert in what you enjoy and reaching out to anyone online.
“Or try and talk to those people because one of the things I learned early on is you can talk to anyone on the Internet,”
Patel says you can reach out to anyone on the Internet and most say yes, but few people try.
“you can talk to anyone on the Internet, and you can just reach out to anyone. Yeah. And most people just say yes, but no one does it.”
Patel says most people will respond to outreach but almost nobody tries.
“One of things I learned early on is you can talk to anyone on the Internet. You can just reach out to anyone, and most people just say yes, but no one does it.”
Nadella describes connecting multiple Fairwater facilities on a petabit network spanning to Milwaukee for distributed AI training.
“Fairwater 4, which you're going to see under construction nearby, will also be on that one petabits network so that we can actually link the two at a very high rate.”
Patel says OpenAI shifted from InfiniBand to Ethernet for Stargate projects, following Meta's similar transition.
“All of OpenAI's cluster deployments in '23 and '24 were pretty much all InfiniBand. Now with their Stargate projects, they're all ethernet. And the same applies to other folks.”
Patel claims network performance drives 5x to 10x difference in inference for models using disaggregated PD or wide EP techniques.
“And so if you look into inference x, it turns out network performance drives a not just a, hey, 30%. It's actually multiple, like, x's.”
Patel identifies disaggregated PD and wide EP as networking techniques where performance matters significantly.
“When someone has really good networking with these leading edge techniques to run a model such as KIMI or DeepSeq called disaggregated PD or wide EP, these are two different techniques, network performance is a humongous factor.”
Patel says networking represents 15-20% of cluster cost but has outsized impact on performance.
“with GPU rental pricing soaring, networking is 1520% of the cost of the cluster, but it can have a much more outsized impact on performance.”
Patel contrasts Nvidia's NVLink connecting 72 GPUs via switches versus Google's ICI connecting 8,000 chips without switches.
“For Google, their ICI can connect 8,000 chips at super high bandwidth, but you have to pass through other chips to get there because there's no switch.”