Patel says NVIDIA GPU bandwidth increased less than 10x while FLOPS increased 100x from 2016 to 2023.
“The bandwidth has not even gone up one order of magnitude. Right? Whereas flops have gone up two orders of magnitude. So so less than one order of magnitude versus two orders of magnitude increase.”
Patel reports PyTorch 2.0 delivered 30% performance improvement on 7,000 popular GitHub models.
“So I think I think I think, the PyTorch Foundation tested the 7,000 most popular most starred models on GitHub, and they got, like, a 30% increase.”
Patel reports PyTorch 2.0 delivered 30 percent performance increase on 7,000 popular models, driving adoption.
“PyTorch Foundation tested the 7,000 most popular most starred models on GitHub, and they got, like, a 30% increase.”
Patel reports reasoning models improved college math test scores from 4% to 80% in six months.
“Basically, our college level math tests were initially 4% for before this innovation was happened to now they're scoring 80%, the best leading edge models. Right? And this has just happened in six months.”
Patel predicts another 50-100% Blackwell performance improvement from software optimization of standard libraries.
“I see another 50% to doubling in the cards, just from the fact that there is so much low hanging fruit or performance left on the table if you are using the general open libraries, right? Whether it's cuDNN, VLM, SGLANG, all these other things,”
Patel says Blackwell cost is up 50-70% over Hopper but performance exceeds 2x, making it an obvious choice.
“while cost is only up, know, call it 50% or 60% or 70% versus Hopper, the performance is well north of 2x, right?”
Patel predicts general-purpose Blackwell users could see another performance doubling over the next year from software improvements.
“for the general purpose user, yeah, you're going to have another doubling of performance potentially on the tables over the next year.”
Patel claims network performance drives 5x to 10x difference in inference for models using disaggregated PD or wide EP techniques.
“And so if you look into inference x, it turns out network performance drives a not just a, hey, 30%. It's actually multiple, like, x's.”
Patel says networking represents 15-20% of cluster cost but has outsized impact on performance.
“with GPU rental pricing soaring, networking is 1520% of the cost of the cluster, but it can have a much more outsized impact on performance.”
Patel claims hardware improved 30x from Hopper to Blackwell for DeepSeek on optimized deployments.
“from Hopper to Blackwell, which is all we've had over the last three years, roughly 30x improvement on DeepSeek, on the most optimized deployment, which you can see on InferenceX there's about a 30x improvement.”
Patel says SemiAnalysis initially doubted Jensen's 25x Blackwell claim, predicting only 15-20x improvement.
“Jensen, when he originally launched Blackwell, had claimed it would be a 25x improvement. And at the time, no one believed him. Right? It's Jensen, right?”
Patel reports cache hit rates above 95% for many agentic workflows in production.
“we're seeing cache hit rates above 95% for many AgenTeq workflows, which which means the cost for a cache hit is it's not free,”