Patel reports memory prices have quadrupled over the past year, with SSDs up three to four times.
“Memory has like four x in price over the last like year, and it's gonna continue to go up. And then now storage SSD, right?”
Patel explains reasoning models dramatically increase serving costs due to long output context and memory constraints.
“your memory usage is going way up with these reasoning models, and you still have a lot of users. So effectively, the cost to serve multiplies by a ton.”
Patel states memory cost has quadrupled in NVIDIA GPUs while per-gigabyte pricing remained flat since 2016.
“And now with the h 100, they have 96 or 80 gigabytes of memory, but the cost per gigabyte is the same.”
post Patel attributes governance issues to Hynix's recent stock underperformance versus Micron
https://x.com/dylan522p/status/2085417667071213619