Patel identifies distributed training across data centers with lower bandwidth as a key unsolved problem that would unlock massive scaling.
“Everything that we've seen so far is that large scale training has to happen in an individual data center with very high speed networking.”
Nadella describes connecting multiple Fairwater facilities on a petabit network spanning to Milwaukee for distributed AI training.
“Fairwater 4, which you're going to see under construction nearby, will also be on that one petabits network so that we can actually link the two at a very high rate.”