Patel says OpenAI has a lot less total FLOPS than Google despite having more GPUs than most perceive.
“whole point was that Google you know, OpenAI, who everyone would be like, oh, yeah. They have more GPUs than anyone else. Right? But they have a lot less flops than Google.”
Patel says Google's TPU production numbers are closely guarded secrets that even most Google DeepMind employees don't know.
“Right? That's like a very closely guarded secret, most people that work at Google DeepMind don't even know number.”
Patel claims Google is massively ramping TPU production, a closely guarded secret most DeepMind employees don't know.
“Google's actually ramping up TPU production massively. And I think people in AI would be like, Well, duh. But like, okay, who has the capability of figuring out the number?”
Patel claims Google will have more compute than any other company by a large factor.
“Google is going to have more compute than any other company in the world period by like a large, large factor.”
Patel reports Google has publicly confirmed training across multiple data centers in one region.
“Google has actually publicly stated that they've done this. They've trained across multiple data centers in a region.”
Patel says Google built rack-scale AI systems with Broadcom in 2018 before NVIDIA deployed similar architecture.
“Google did something very similar in 2018, right, with the TPU. Now they couldn't do it alone. Right? They know the software.”
Patel says Google deployed water cooling years before NVIDIA and achieves higher reliability than NVIDIA GPUs.
“Google's brought in water cooling for years. Right? NVIDIA only just realized they needed water cooling on this generation. And Google's brought in a level of reliability that NVIDIA GPUs don't have.”
Patel reports Google is spending $250 million with Scale AI this year but backing out of the partnership.
“And they're backing out. Obviously, they've spent a lot of money and there's stuff they can't back out of, but it's like, that's gonna go down a lot.”
Patel reports Google is backing out of Scale AI, spending around $250 million this year but reducing future commitment.
“Google's backing out, I think they're gonna spend on the order I've heard like $250,000,000 this year with them. And they're backing out.”
Patel notes YouTube holds unique video data that gives Google training advantage despite scraping restrictions.
“I don't know what ungodly percentage of video is only on available on YouTube. And sure, you can scrape YouTube, blah blah blah.”
Patel says Google will make five to seven million TPU chips while Amazon targets three to five million.
“They're going to make something like five to 7,000,000 chips, right? Of their own TPUs. You look at Amazon, they're trying to make three to 5,000,000.”
Patel predicts Google will eliminate its $100 billion annual cash flow next year by spending everything on AI infrastructure.
“Google will have no cash flow next year because they're they see AI so clearly, and they know that they need to spend every dollar they make on compute,”
Patel explains Google woke up to AI compute needs after Gemini models drove user growth.
“Google had Nano Banana and Gemini three, which caused their user metrics to skyrocket, and leadership at Google was like, oh.”
Patel predicts Google and OpenAI will both release new models in two weeks.
“Release New release in two weeks. Everyone's releasing in two weeks. Google, OpenAI, maybe Anthropic. I don't know about Anthropic, but Google and open AI definitely releasing in two weeks.”
Patel reveals Google runs three separate TPU design programs with Broadcom, MediaTek, and undisclosed partner.
“They're making a TPU with Broadcom, that's a different architecture than a TPU with MediaTek, that's a different TPU than the architecture that is, I won't disclose, by research.”
Patel explains NVLink connects only 72 GPUs while Google ICI connects 8,000 chips without switches, creating architectural trade-offs.
“NVIDIA, the NVLink can only connect 72 GPUs. For Google, their ICI can connect 8,000 chips at super high bandwidth, but you have to pass through other chips to get there because there's no switch.”
Patel contrasts Nvidia's NVLink connecting 72 GPUs via switches versus Google's ICI connecting 8,000 chips without switches.
“For Google, their ICI can connect 8,000 chips at super high bandwidth, but you have to pass through other chips to get there because there's no switch.”
Patel asks if Jeff Dean leaving is positive or negative for Broadcom
“IS JEFF DEAN LEAVING POSITIVE OR NEGATIVE FOR BMC”
Patel quotes Deepmind friend saying TPU ecosystem grows stronger every time Googlers leave
“"Everytime Googlers leave, the TPU ecosystem grows stronger" Deepmind friend coping so hard in my DMs rn”
Sanjay Ghemawat and Quoc Le, both Google Fellows, are joining Jeff Dean's new venture along with Oriol Vinyals.
“Joining Jeff are Sanjay Ghemawat, Quoc Le, and Oriol Vinyals. Sanjay and Quoc were both Google Fellows, which is a title reserved for the company’s top dozen or so technical contributors.”
Jeff Dean is leaving Google to start a new lab called Discovery Loop.
“Jeff Dean, former Google Chief Scientist and Gemini co-lead is leaving to start a neolab called Discovery Loop.”
Koray Kavukcuoglu is replacing Demis Hassabis as leader of DeepMind and Gemini.
“Koray Kavukcuoglu, former DeepMind CTO and the one remaining Gemini co-lead, is replacing Demis as the leader of DeepMind/Gemini.”
DeepMind is no longer a frontier lab.
“For all intents and purposes, we believe DeepMind is no longer a frontier lab .”
Google announced a complete overhaul of DeepMind leadership on Wednesday, August 5th.
“On Wednesday, August 5th, Google announced a complete overhaul of DeepMind leadership. A quick recap:”