The Minutes of Dylan Patel on benchmarking https://minutesof.com/dylan-patel/on/benchmarking/ Everything Dylan Patel has said on benchmarking: 19 verbatim quotes between October 2025 and July 2026, each with a timestamp and a link to the recording… en Sun, 30 Aug 2026 16:22:17 +0000 Patel says InferenceX now has over $80 million of compute across multiple vendors. https://minutesof.com/q/eb8b0b35-dc54-4b31-af11-cfea01d719e6/ https://minutesof.com/q/eb8b0b35-dc54-4b31-af11-cfea01d719e6/ “we have over $80,000,000 of compute GPUs from NVIDIA AMD, TPUs from, Google, Tranium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest, PyTorch version,” — RAISE Summit Thu, 16 Jul 2026 16:44:57 +0000 benchmarking Patel reveals SemiAnalysis operates over $80 million in compute for daily automated benchmarking across all ma https://minutesof.com/q/3047ee19-387d-4066-8826-f26d2fc2c5b3/ https://minutesof.com/q/3047ee19-387d-4066-8826-f26d2fc2c5b3/ “Every day it runs on an automated CI, and we run it on all the latest Chinese models from GLM, Zebu, Moonshot, Kimi, Alibaba, all these models we run.” — RAISE Summit Thu, 16 Jul 2026 16:44:57 +0000 benchmarking Patel says SemiAnalysis analyzed over $5 million worth of Claude production traces to benchmark agentic worklo https://minutesof.com/q/38c4abf0-d997-4233-82b1-61bd91d353d3/ https://minutesof.com/q/38c4abf0-d997-4233-82b1-61bd91d353d3/ “Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it, you know, fixed context length.” — RAISE Summit Thu, 16 Jul 2026 16:44:57 +0000 benchmarking Patel says InferenceX analyzed over $5 million worth of Claude Code production traces. https://minutesof.com/q/a7fe9e1b-7a52-4f09-a0b0-b1a41da1dfce/ https://minutesof.com/q/a7fe9e1b-7a52-4f09-a0b0-b1a41da1dfce/ “we've analyzed over $5,000,000 worth of Claude code traces. Right? So this is real production traffic that people have donated to us,” — RAISE Summit Thu, 16 Jul 2026 16:44:57 +0000 benchmarking Patel's benchmarking found Blackwell is 30x faster than Hopper on DeepSeek v3, exceeding Jensen's 25x claim. https://minutesof.com/q/be57f11d-1f2b-4256-80d2-16bbf2bed240/ https://minutesof.com/q/be57f11d-1f2b-4256-80d2-16bbf2bed240/ “In DeepSeek v three, Blackwell is 30 x faster than Hopper on on somewhere on the continuum.” — WisdomTree in Europe Thu, 09 Jul 2026 06:16:39 +0000 benchmarking Patel secured over $50 million in donated hardware for InferenceX, expanding to $100 million with TPUs and tra https://minutesof.com/q/f9ad7311-11bb-4dce-afab-53a809a7e91a/ https://minutesof.com/q/f9ad7311-11bb-4dce-afab-53a809a7e91a/ “We've got over $50,000,000 of hardware donated to us. Once we launch TPUs and training, would actually be over $100,000,000 of hardware.” — Sequoia Capital Tue, 30 Jun 2026 12:00:25 +0000 benchmarking Patel says Nvidia and AMD both lie about peak flops specs which are impossible to achieve. https://minutesof.com/q/2805f8b7-b1ab-4aa6-bfaa-e0ca89f29b45/ https://minutesof.com/q/2805f8b7-b1ab-4aa6-bfaa-e0ca89f29b45/ “All their quoted specs are lies impossible to achieve Whether it's Nvidia or AMD, neither of them you can ever hit their peak flops.” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel claims Nvidia achieves 50% sustained performance on common 8k gemm while AMD gets only 30%. https://minutesof.com/q/f0d9f7af-e5f2-4a3f-9331-63fe03e874a1/ https://minutesof.com/q/f0d9f7af-e5f2-4a3f-9331-63fe03e874a1/ “if you just take a eight k by eight k by eight k gem, very common shape and you run it on the map mode unit of AMD and Nvidia, you get like 50% sustained performance on Nvidia and you get like 30% sustained performance on AMD.” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel asserts vendors cannot achieve their advertised flops specifications in real-world conditions. https://minutesof.com/q/dd7daafb-b209-4da7-868d-23f93933e381/ https://minutesof.com/q/dd7daafb-b209-4da7-868d-23f93933e381/ “There there is no functional way to get anywhere close to their flops that they advertise.” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel explains Inference X was created because vendor-claimed performance is unattainable with any available f https://minutesof.com/q/9f06077d-1d55-479f-aedc-dbafa2704a09/ https://minutesof.com/q/9f06077d-1d55-479f-aedc-dbafa2704a09/ “There's no way to get the performance that vendors like to claim. And so our whole thing there was, well, how do we actually have a benchmark that represents what people actually get on performance?” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel notes software dependencies change daily or multiple times weekly across the entire stack. https://minutesof.com/q/7d67261b-caa3-4beb-8c61-2b0c52ec53da/ https://minutesof.com/q/7d67261b-caa3-4beb-8c61-2b0c52ec53da/ “And furthermore, with software changing literally multiple times a week, right, PyTorch has nightlies, VLM has nightlies, Asteeling has nightlies, CUDA drivers update constantly. You just go You go through the whole list.” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel explains AI software stacks update multiple times per week making performance measurement a moving targe https://minutesof.com/q/bf34d090-4f22-4978-bd08-307b1566b305/ https://minutesof.com/q/bf34d090-4f22-4978-bd08-307b1566b305/ “with software changing literally multiple times a week, right, PyTorch has nightlies, VLM has nightlies, Asteeling has nightlies, CUDA drivers update constantly. You just go You go through the whole list.” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel notes AI software stacks update constantly with nightly builds making performance measurement time-sensi https://minutesof.com/q/3aab3b19-37a2-4c04-a1ec-5e5018b0ee10/ https://minutesof.com/q/3aab3b19-37a2-4c04-a1ec-5e5018b0ee10/ “with software changing literally multiple times a week, right, PyTorch has nightlies, VLM has nightlies, Asteeling has nightlies, CUDA drivers update constantly.” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel observes VLLM and SGLANG framework developers compete publicly on Twitter using InferenceX as a leaderbo https://minutesof.com/q/6447730b-2e25-4183-babf-5c0b556ea5a0/ https://minutesof.com/q/6447730b-2e25-4183-babf-5c0b556ea5a0/ “VLLM and SGLANG love competing with each other. They used to post on Twitter all the time about how they had beat the other one in certain some kind of performance” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel reports AMD and Nvidia engineers treat Inference X as a competitive leaderboard. https://minutesof.com/q/90c32f06-2549-483b-a0ec-37f8fcdadb56/ https://minutesof.com/q/90c32f06-2549-483b-a0ec-37f8fcdadb56/ “AMD and Nvidia. They love to compete with each other. And AMD engineers and Nvidia engineers look at Inference X as a leader board.” — TensorWave Thu, 30 Apr 2026 18:18:14 +0000 benchmarking Patel says InferenceMax runs daily benchmarks to track hardware performance improvements as software optimizes https://minutesof.com/q/c8604b65-b25b-4229-8fea-4a277c80372d/ https://minutesof.com/q/c8604b65-b25b-4229-8fea-4a277c80372d/ “And why this is important is you can see the progress of hardware over time as the software stack gets more optimized.” — Open Compute Project Thu, 23 Oct 2025 06:01:19 +0000 benchmarking Patel quantifies InferenceMax uses tens of millions of dollars in GPU hardware with daily software updates. https://minutesof.com/q/ae8c91b8-452e-4194-a8f3-8286559ee830/ https://minutesof.com/q/ae8c91b8-452e-4194-a8f3-8286559ee830/ “we have tens of millions of dollars of hardware of the current and last generation GPUs. As I mentioned before, the software updates every single day.” — Open Compute Project Thu, 23 Oct 2025 06:01:19 +0000 benchmarking Patel quantifies GB200 as 10x more power efficient than H200 at certain interactivity rates. https://minutesof.com/q/5246be24-b713-4f40-9330-362372209a1f/ https://minutesof.com/q/5246be24-b713-4f40-9330-362372209a1f/ “at certain interactivity rates, I. E. Tokens per second per user, it's 10x more efficient per watt, right, compared to H200.” — Open Compute Project Thu, 23 Oct 2025 06:01:19 +0000 benchmarking Patel reports AMD MI355 beats NVIDIA B200 in some document processing scenarios with open source software. https://minutesof.com/q/0ddc9f32-3943-4514-87fd-c275b179bf83/ https://minutesof.com/q/0ddc9f32-3943-4514-87fd-c275b179bf83/ “in some cases, AMD actually does have a publicly usable open source implementation that beats NVIDIA even, right, with the MI355 versus V200, which is a surprise, right?” — Open Compute Project Thu, 23 Oct 2025 06:01:19 +0000 benchmarking