Patel explains model bandwidth utilization is the critical metric for inference, unlike training where MFU matters more.
“But on inference, it's not being talked about much, but model MBU, model bandwidth utilization is the important factor.”
Patel argues AI software will have lower R&D costs but much higher cost of goods sold from operating services.
“Like the R and D cost is much lower in terms of people, but the cost of goods sold in terms of actually operating the service, I think will be much higher.”
Patel criticizes Hugging Face libraries for achieving only 15% memory bandwidth utilization on inference, calling it horrendous.
“Hugging Face's libraries are actually very inefficient, like incredibly inefficient for inference. You get like 15% MBU on on on on some configurations like eight a one hundreds and Lama 70 b.”
Patel reports H100 rental prices have dropped from $3-4 per hour to $2.15 or less, approaching natural cost of $1.40.
“An hour. Right? For shorter term or midterm deals. Right now, it's like, if you want a six month deal, you could get, like, $2.15 or less.”
Patel explains OpenAI's o1 model thinks for 5-20 seconds before outputting, creating inference throughput challenges.
“When you use OpenAI's o one, it thinks for ten seconds, twenty seconds, five seconds. It varies a lot, but thinks for a while and then it sends you tokens.”
Patel calculates o1 models increase inference costs by roughly 50x compared to standard models per query.
“Cost increase for a single token to be generated is four to five x, but then I'm generating 10 x as many tokens.”
Patel states Microsoft's inference costs are a few billion for ten billion in revenue but o1 multiplies costs 10x.
“Their cost for that is a few billion dollars for $10,000,000,000 of revenue. Right. If now, obviously, the better model gets to charge more. Right?”
Patel estimates Blackwell delivers 10-15x cost improvement for inference despite NVIDIA claiming 30x at GTC.
“But now, like, Blackwell, NVIDIA's pitching 10 to 15 x improvement in cost. It's like, well, you know, they're massaging the numbers marketing.”
Patel explains reasoning models dramatically increase serving costs due to long output context and memory constraints.
“your memory usage is going way up with these reasoning models, and you still have a lot of users. So effectively, the cost to serve multiplies by a ton.”
Patel explains reasoning models dramatically increase memory usage and reduce batch size, multiplying serving costs.
“So your your memory usage is going way up with these reasoning models, and you still have a lot of users.”
Patel says AI model costs dropped 1200x from GPT-3's $60 per million outputs to current pricing.
“GPT three was it cost, you know, it cost $60 for the million output. And over time, that kept reducing. Right? OpenAI released new models.”
Patel says DeepSeek's model is 400 to 600x cheaper than original GPT-4.
“And so now DeepSeek's model is something like 400 to 600x cheaper than the original GPT-four.”
Patel states same-quality AI model output became 1200x cheaper than GPT-3 through successive model releases.
“Anthropic released new models. Google released new models. Meta released new models. Right? And then now it is 1200x times it's 1200x cheaper for the same output.”
Patel reports GPT-3 to Llama 3.2 costs fell 1,200x while GPT-4 to DeepSeek v3 costs fell 600x.
“when we looked at g p d three, the cost fell 1,200 x from g p d three's initial cost to what you can get Lama 3.23 b today.”
Patel reveals DeepSeek inference implementation requires 160 GPUs worth over $10 million of hardware per replica.
“That's over $10,000,000 of hardware, and then that's just one replica, then you'll have a lot of replicas and you share the caching servers between them.”
Patel explains OpenAI's router will send low-value queries to cheaper models but use expensive compute for high-value monetizable queries.
“if the user asks a low value query like, hey, why is the sky blue? Just route them to mini. The model can answer perfectly fine, and that is a chunk of queries.”
Patel says GPU hardware and networking represent 80% of data center costs, with power and cooling only 20%.
“It's the GPU purchases. It's the networking. It's the it's the physical data center conversion power conversion equipment. All of this stuff is, like, 80% of the cost.”
Patel calculates that without Supermicro's liquid cooling solution, servers would consume 30% more power.
“So that's why Shibbol Micro dedicated deep cooling so aggressively. And if the liquid cooling solution from Super Micro didn't exist, then those servers would be consuming 30% more power.”
Patel states Supermicro's liquid cooling reduces Hopper server power from 10 kilowatts to 7-8 kilowatts versus air-cooled competitors.
“Everyone else's hopper servers h 100 air cooled. And so each server consumes, you know, 10 kilowatts almost. Right?”
Patel describes diagnostic challenges in AI data centers where failures occur across multiple infrastructure layers.
“And people have had problems where their data center wasn't ready and something stopped working. They don't know, is it the chip? No.”
Patel shows B200 has 15x raw performance advantage over H100 but only 10x performance per TCO.
“If we don't divide by TCO, then it looks like the performance of B200 is actually 15x that of H100, versus the performance TCO is only 10x,”
Patel breaks down inference TCO as 20% electricity and data center, 80% hardware on NVIDIA deployments.
“20% of your cost is your electricity, your data center real estate, roughly. And then the rest of the cost is that hardware, at least on a standard NVIDIA deployment your GPUs, your networking, etcetera.”
Patel reports GB200 is 10x more power efficient than H200 at certain interactivity rates.
“But it turns out at certain interactivity rates, I. E. Tokens per second per user, it's 10x more efficient per watt, right, compared to H200.”
Patel notes NVIDIA Blackwell performance improved dramatically from launch to present, as did AMD hardware.
“If you tried to use NVIDIA's Blackwell six months ago, the numbers were not amazing, right? But now they're actually amazing. So how did that progress over time? Same with AMD, right?”
Patel says inference providers sell public endpoints at flat or negative margins, compensating through private deployments.
“Most of the inference providers are selling at flat margins or even negative for their public endpoints. And they then make it up when people do private deployments.”
Patel finds AMD MI355 beats NVIDIA B200 on performance TCO in certain publicly usable configurations.
“So we do different scenarios. We do document processing, which is 8,000 context in, 1,000 out. We do chat, which is 1,000 in, 1,000 out.”
Patel describes public inference as a loss leader to generate private sovereign and enterprise engagements.
“And so I think that's what a lot of this is, is you end up with the public business as like a loss leader just to generate private engagements, whether it be sovereigns or enterprises.”
Patel identifies GB200's power efficiency advantage but notes deployment challenges with backplane and liquid cooling.
“GB200 has a huge power efficiency advantage, right? Everyone here understands the challenges of running and deploying GB200. There's a lot of challenges with the backplane.”
Patel calculates 20% power efficiency difference translates to only 4% TCO difference on NVIDIA deployments.
“Because if you have enough power, a 20% difference in performance per watt only ends up being a 4% difference in TCO.”
Patel says the standard inference deployment unit has shifted from single nodes to hundreds of GPUs.
“the standard unit for an inference deployment being hundreds of GPUs instead of a single node. And then there's all these different things about traffic.”
Patel projects AI will progress from 10-30 minute tasks to days of autonomous work, with model companies charging thousands.
“In the future maybe they're doing days worth of work autonomously. And then the model companies are charging thousands of dollars”
Patel explains prefill costs one-fourth of decode per token, making 30,000 token inputs costlier than 2-4,000 token outputs.
“And that's a very common ratio. Right? And then when you think about, okay, the cost of running pre fill is roughly one fourth of running decode.”
Patel states avoiding redundant prefill through KV cache can cut inference costs to one-fourth of previous levels.
“So you can cut your cost to one fourth of what it was previously if you just don't do the pre fill. Right?”
Patel predicts AI startup revenue will exceed $100 billion by end of year, a claim few believe.
“I think that's an insane statement for a lot of people, but that's what it's gonna be. Yeah. Right? And who believes that number? Right? It's like Yeah. Very few people.”
Patel states Anthropic now adds $23 billion revenue monthly versus few hundred million earlier.
“Anthropics, you know, adding $23,000,000,000 of revenue a month now Mhmm. Versus they were just adding a few 100,000,000 of revenue a month earlier. So clearly, we're in the take off period.”
Patel calculates Anthropic added $1.5 billion in compute infrastructure in one month to serve new revenue growth.
“If Anthropic added 2,500,000,000 of revenue, and their gross margin is 40%, they added like $1,500,000,000 of compute in one month. Mhmm. Right?”
Patel reports Claude Code doubled to 4% of GitHub commits in January alone, with overall AI coding likely at 10%.
“Just in this month just in January, it went from 4% of or 2% of commits on GitHub to 4% of GitHub commits were done by Cloud Code.”
Patel reports seeing AI labs sign GPU rental deals at $2.40 per hour for H100s, well above initial $1.40 cost.
“I've seen deals where certain AI labs I'm be a little bit vague here for a reason, have signed at as high as $2.40 for two to three years.”
Patel cites Anthropic adding $67 billion ARR per month as evidence of expanding AI revenue beyond hyperscalers.
“You're starting to see it with Anthropix revenue adding $67,000,000,000 of ARR a month, but there's so many more firms coming online with revenue streaming in,”
Patel reports H100 rental prices increased from $1.71 per hour to over $2.40 in six months.
“And just in the last six months, it's gone from, you know, deals for one year transacting at $171.60 an hour for h 1 hundreds to now $2.40 plus.”
Patel argues Mythos fast mode is probably cheaper than Claude 4.6 fast mode for most tasks due to token efficiency.
“the flip side is, is, is mythos is more token efficient. Methos fast mode is probably cheaper than like, four, six fast mode for most tasks.”
Patel's firm spent under $100k annually on AI in November before Claude Code adoption.
“What we had was that we had a subscription to every model or we had a subscription to the $200 tier for a chat GPT for every user.”
Patel says their AI spending equals over a third of employee compensation for their 90-person firm.
“We're spending, you know, more like, you know, more than a third of the spend, you know, employee spend, a third of it is also on top of that is AI.”
Patel's firm's AI spending jumped from under $100k to $11 million annually within months.
“by the end of January, our ARS, right, our our our annual recurring spend had hit $4,000,000. And that's because people were using Cloud Code. Now today, it's about $11,000,000.”
Patel argues prefill caching drastically raises GPU utilization by eliminating repeated context recalculation that provides no value.
“Now that means your GPU utilization rises drastically, and the amount of time that the GPUs are generating new tokens is far, far higher than the amount of times that GPUs are generating or recalculating the context, which is not necessarily providing any value to your operations.”
Patel reports OpenAI gross margins rose from 30% to 55% overall, 50% to 65% excluding free users.
“You look at OpenAI late last year, their margins had were roughly 30% gross margin, but if you stripped away the free users, they were at 50%.”
Patel reveals Anthropic turned first profit in Q2 and will hit billion-dollar operating profit in Q3 before IPO.
“So Anthropic turned their first gross profit in Q2 in June. And then in Q3, they will be turning a billion dollars of operating profit, slightly over.”
Patel argues silicon supply can be optimized for either high throughput or high interactivity use cases.
“Supply of silicon can go many ways. You can either leverage it to high throughput things or high interactivity things.”
Inference operations generate five times their cost in revenue which can fund additional training.
“Hey, if I spend 10 bucks on inference capacity, I actually generate 50 bucks of revenue, and then I can turn around and incrementally spend all of that profit on training.”
Patel says Anthropic generates $50 million per megawatt revenue, enabling 5x return on inference spending.
“In the case of Anthropic, the the revenue has gone as high as $50,000,000 per megawatt.”