Patel explains prefill costs one-fourth of decode per token, making 30,000 token inputs costlier than 2-4,000 token outputs.
“And that's a very common ratio. Right? And then when you think about, okay, the cost of running pre fill is roughly one fourth of running decode.”
Patel reports that tokens now represent 30% of his 90-person company's costs versus 70% for employees.
“My my own company of 90 people, 30% of my cost now is tokens versus 70% employee costs.”
Patel says token costs now represent 30% of total costs versus 70% employee costs at his 90-person firm.
“My my own company of 90 people, 30% of my cost now is tokens versus 70% employee costs. It's it's it's there's token maxing is happening.”