Patel criticizes Hugging Face libraries for achieving only 15% memory bandwidth utilization on inference, calling it horrendous.
“Hugging Face's libraries are actually very inefficient, like incredibly inefficient for inference. You get like 15% MBU on on on on some configurations like eight a one hundreds and Lama 70 b.”
Patel states avoiding redundant prefill through KV cache can cut inference costs to one-fourth of previous levels.
“So you can cut your cost to one fourth of what it was previously if you just don't do the pre fill. Right?”
Patel describes simultaneous gains across pre-training, scaling, quantization, systems, and RL infrastructure and methods.
“There's tons of gains in quantization and systems, but there's also tons of gains in RL stuff and both the, like, you know, infra side of things and non infra side of things.”
Patel notes AI inference libraries update twice weekly, causing constant performance changes.
“In fact, the update cycle for most of these libraries is twice a week, so you basically have the software updating all the time and therefore performance changing.”
Patel argues co-optimizing hardware, software, and models produces 100x gains instead of 8x from multiplicative 2x improvements per layer.
“The real breakthrough innovation is when you leapfrog a few layers, you co optimize and co design them, and now all of a sudden you've taken what could have been a 2x here, 2x here, 2x here, and instead of being multiplicative to 8x, it's actually 100x, because you've optimized it across all three layers.”