Patel explains o1's thinking time creates memory bandwidth issues that prevent batching users at high levels.
“But if you batch higher, k b cache is not just a memory capacity issue, it's also a memory bandwidth issue.”
Patel explains GB200's 72-GPU configuration creates reliability challenges compared to 8-GPU systems due to higher failure probability.
“If the reliability of each GPU is the same, then when a single GPU fails in 72 GPUs, you have a much higher chance of something failing, right?”