Baker reveals high bandwidth memory is a bigger cost component for NVIDIA than Taiwan Semi, with only two capable manufacturers.
“High bandwidth memory is a bigger part of NVIDIA's cogs on GPUs than Taiwan Semi is. And today, there's two companies that can make it, Hynix and Micron.”
Baker warns a true DRAM capacity cycle could see prices increase 10X, not the typical 30-50%.
“If the price of DRAM, if like a DRAM wafer is valued at like a five carat diamond.”
Baker believes NVIDIA could sell $11.52 trillion in chips next year if TSMC tripled capacity but bubble risks prevent it.
“And if they doubled or tripled capacity, like, NVIDIA could probably sell $11,520,000,000,000 worth of chips next year. I really believe that. But the other side of that might be very painful for everyone.”
Baker admits he has been hanging on to memory stocks for dear life despite contrarian instincts.
“I've been hanging on to the memory stocks for dear life. And yeah, but that's that's a that that's been a lifelong journey”
Baker takes the over on every AI capex and memory forecast number given by other speakers at the conference.
“And I would just say, I take the over on every number that they gave. Every single number.”
Baker argues this may be the first true capacity cycle and that fundamental shortages help avoid a destructive bubble.
“This may be the first true capacity cycle. And I and and I and I do think that these fundamental shortages are good for us as investors.”
“I would say based on every memory cycle we have had for the last twenty five years, this is the time to be selling memory 100%.”
Baker predicts GPUs will have 10-15 year useful lives due to prefill-inference disaggregation, extending older chips' value.
“The disaggregation of inference means that I think these GPUs are going to have ten or fifteen year lives.”
Memory bottlenecks create severe performance penalties, with a thirty-eight times slowdown when data operations exceed available DRAM capacity.
““38x slowdown once shuffle data exceeds available DRAM and spills to SSD.” From a joint Micron/Meta white paper. The DRAM bottleneck almost certainly lasts the longest. OP from @jukan05”
Baker argues game theory suggests memory LTAs are durable and Nvidia may have significant cloud revenue shares
“Game theory suggests memory LTAs are likely to be durable. And yes, Nvidia may have a significant cloud business via revenue shares. https://t.co/GFN1kDQmSk”
Baker argues breaking memory supply agreements could destroy franchises due to allocation loss risk.
“I just think given that this is, the axis around which kind of everything is revolving, man, like, might blow up your entire business and your franchise by breaking an LTA.”
Baker suggests memory companies should copy NVIDIA's credit wrapper model to participate in GPU revenue streams as credit markets revolt.
“Which is? I would be going to the buyers of GPUs, tradiums, and whoever and say, I'll participate in the NVIDIA credit wrapper.”
Baker explains breaking memory LTAs could destroy companies by losing allocations when market tightens again.
“Well, if they're breaking their LTAs, it probably means, you know, your oversupply, prices are coming down, and then, you know, capacity naturally contracts. Well, what do you think is gonna happen to Google's allocations?”
Baker identifies memory as the single most important factor for increasing token output per compute unit.
“It's the single most important thing you could do to increase token output per unit of compute, and then that obviously, definitionally, actually lowers costs,”
Baker suggests memory companies should copy NVIDIA's credit wrapper model despite business instability risks.
“I would be going to the buyers of GPUs, Traniums, and whoever and say, I'll participate in the NVIDIA credit wrapper. Now their business is just inherently less stable and predictable,”
Baker suggests memory companies should copy NVIDIA's credit wrapper model while they have cash and credit markets are tight.
“I'm just making this up. But, like, do something. Like, you can because you have money now and credit markets are revolting.”
“And CXL!”
“Jitendra Mohan, CEO and co-founder of Astera Labs, is our third confirmed speaker for the @TomorrowXSummit. Really looking forward to discussing the role of scale-up networking in a memory constrained world and the intersection of their switch franchise with optics. https://t.co/stwGpJ8zox”
“Looking forward to interviewing @andrewdfeldman CEO and founder of @cerebras at the @TomorrowXSummit in Austin November 17-18. Scale-up networking and SRAM acceleration are the best solutions to the DRAM shortage from my perspective. Given that Nebius is monetizing at $20 to $25 https://t.co/ZUAzwy8A5d”
Baker announces Micron executive Sumit Sadana as speaker and says memory per FLOP drives tokens per watt
“Happy to announce Sumit Sadana EVP and Chief Business Officer of @MicronTech as a confirmed speaker at the @TomorrowXSummit in November. Memory per FLOP drives tokens per watt. Looking forward to hearing Sumit’s thoughts on how AI is changing the business model of memory https://t.co/k1Pdk94PDu”
Baker says no one thought someone was crazy and everyone will likely do 3D DRAM or zHBM
“@bubbleboi No one thought you were crazy. Everyone likely going to do this or zHBM. *Everyone*”
Baker predicts every accelerator company will do some variant of 3D DRAM or zHBM discussed at HotChips
“Every accelerator company, and I mean every accelerator company, is likely going to do some variant of 3D DRAM or the “zHBM” which were both discussed at HotChips today. The larger companies will probably run (are running) parallel programs, but these two memory variants”
Baker dismisses SRAM scaling concerns as settled for anyone close to logic, memory and packaging roadmaps at wafer scale
“@IanCutress Luckily for me, I have never been near a cliff or a bridge when I have read a post about how “SRAM cannot scale.” This topic has largely been put to bed for anyone remotely close to logic/memory/packaging roadmaps especially at wafer scale.”
Baker credits Jalapeño as first good ASIC outside TPU and Trainium but says disaggregated GPU plus SRAM will outperform
“Impressive that Jalapeño outperforms the comparable TPU and is in the mix with Rubin. Credit where credit is due - first good ASIC outside of TPU/Trainium. However, will likely significantly underperform a disaggregated GPU/Trainum plus SRAM accelerator setup. Especially with”