The Minutes of Dylan Patel on bandwidth https://minutesof.com/dylan-patel/on/bandwidth/ Everything Dylan Patel has said on bandwidth: 3 verbatim quotes between February 2023 and November 2024, each with a timestamp and a link to the recording… en Sun, 30 Aug 2026 16:22:20 +0000 Patel explains that AdamW optimizer requires four bytes per parameter, creating 400 gigabytes of data for 100B https://minutesof.com/q/9b0a4eda-79e5-4213-b0df-76f45b715526/ https://minutesof.com/q/9b0a4eda-79e5-4213-b0df-76f45b715526/ “when you look at LLMs, people use AdamW. And AdamW as an optimizer is like four bytes per parameter roughly, if I recall correctly the optimizer state.” — Scaling Intelligence Tue, 12 Nov 2024 04:17:00 +0000 bandwidth Patel calculates that multi-trillion parameter models require transmitting 40 terabytes of data every two seco https://minutesof.com/q/2b1dd77d-4b29-490c-a1f9-fd32caff427b/ https://minutesof.com/q/2b1dd77d-4b29-490c-a1f9-fd32caff427b/ “They're doing it for like multi trillion. Right? So let's call it 10,000,000,000,000 parameters, four bytes parameter, that's 40 terabytes of data you need to transmit in two seconds.” — Scaling Intelligence Tue, 12 Nov 2024 04:17:00 +0000 bandwidth Patel says NVIDIA GPU bandwidth increased less than 10x while FLOPS increased 100x from 2016 to 2023. https://minutesof.com/q/45f2aadd-215b-4a37-97e7-67d1873d78a9/ https://minutesof.com/q/45f2aadd-215b-4a37-97e7-67d1873d78a9/ “The bandwidth has not even gone up one order of magnitude. Right? Whereas flops have gone up two orders of magnitude. So so less than one order of magnitude versus two orders of magnitude increase.” — Gradient Flow Thu, 02 Feb 2023 12:00:39 +0000 bandwidth