Patel states memory cost has quadrupled in NVIDIA GPUs while per-gigabyte pricing remained flat since 2016.
“And now with the h 100, they have 96 or 80 gigabytes of memory, but the cost per gigabyte is the same.”
Patel explains o1's 10,000 token thinking phase requires constant memory-intensive KV cache generation and reading.
“When I do this with o one, right, because it's doing that thinking phase of 10,000, right, this is this is basically the whole context length thing.”
Patel explains reasoning models dramatically increase serving costs due to long output context and memory constraints.
“your memory usage is going way up with these reasoning models, and you still have a lot of users. So effectively, the cost to serve multiplies by a ton.”
Patel reports memory prices tripled from $3-4 per gigabyte to $12, adding $100 iPhone cost.
“Each gig cost used to cost roughly 3 or $4, so it's $50. But now the price of memory is like tripled. Let's call it if it's now it's $12 per gig for DDR.”
Patel reports memory prices have quadrupled over the past year, with SSDs up three to four times.
“Memory has like four x in price over the last like year, and it's gonna continue to go up. And then now storage SSD, right?”
Patel attributes governance issues to Hynix's recent stock underperformance versus Micron
“@nodiligence nah governance issues have been cropping up lately.”