The Minutes of Dylan Patel on model architecture https://minutesof.com/dylan-patel/on/model-architecture/ Everything Dylan Patel has said on model architecture: 4 verbatim quotes between February 2025 and June 2026, each with a timestamp and a link to the… en Sun, 30 Aug 2026 16:22:19 +0000 Patel explains how MOE routing works and how some of Meta's experts weren't being used at all. https://minutesof.com/q/d8c531cb-5cc2-4ab8-b1ae-b23a834b6b94/ https://minutesof.com/q/d8c531cb-5cc2-4ab8-b1ae-b23a834b6b94/ “And and each expert learns its own independent things and it's like really not something observable by people. But what you can see is tokens when which experts do they route to?” — Matthew Berman Mon, 30 Jun 2025 17:27:19 +0000 model architecture Patel says GPT-5 will be first model to massively scale both pre-training and post-training simultaneously. https://minutesof.com/q/00545820-383f-45e6-ac01-d8e99c981b60/ https://minutesof.com/q/00545820-383f-45e6-ac01-d8e99c981b60/ “Right? And this would be the first time we see a model that was was a step up in both at the same time.” — Alex Kantrowitz Wed, 23 Apr 2025 16:30:06 +0000 model architecture Patel explains training a 70 billion parameter model requires passing four times that data every training step https://minutesof.com/q/2bc44e6d-a415-482b-bcb0-f9c59a1b4198/ https://minutesof.com/q/2bc44e6d-a415-482b-bcb0-f9c59a1b4198/ “So if you're training like a 70,000,000,000 parameter model, you need to pass four x that number every single training step. And when you're passing that, you can't really overlap communications and compute.” — Prime Intellect AI Tue, 25 Mar 2025 18:21:45 +0000 model architecture Patel explains DeepSeek v3 base is trained once, then post-trained differently to create chat versus reasoning https://minutesof.com/q/02fd4a40-e301-43f7-be83-33801b8010a9/ https://minutesof.com/q/02fd4a40-e301-43f7-be83-33801b8010a9/ “This reasoning model has a lot of overlapping training steps to DeepSeek v three, and it's confusing that you have a base model called v three that you do something to to get a chat model, and then you do some different things to get a reasoning model.” — Lex Fridman Mon, 03 Feb 2025 00:12:13 +0000 model architecture