Patel explains model bandwidth utilization is the critical metric for inference, unlike training where MFU matters more.
“But on inference, it's not being talked about much, but model MBU, model bandwidth utilization is the important factor.”
Patel argues prefill caching drastically raises GPU utilization by eliminating repeated context recalculation that provides no value.
“Now that means your GPU utilization rises drastically, and the amount of time that the GPUs are generating new tokens is far, far higher than the amount of times that GPUs are generating or recalculating the context, which is not necessarily providing any value to your operations.”