THE PUBLIC RECORD
What has Andrej Karpathy said about…?
Explore their words. Follow an idea. Go straight to the source.
Research Andrej Karpathy in your dashboard →
Search covers the displayed transcript and written excerpts, plus all X posts stored for this person, including replies. Coverage is growing; this is not a complete account history.
In their own words
459 entries
Transcript excerpts may contain transcription errors. Open the recording for wording, speaker and context. Source links do not verify the claims made.
No matching entries. Try fewer words or clear the filters.
Choose an end date on or after the start date.
-
@scaling01 Yeah I’d say the RL is working now all right https://t.co/9VGtttWclb
-
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel. - Napkin math somewhere around 6B people (~75% of the population) have barely come https://t.co/79xGhQdf1n
-
@TheStalwart I had some amount of journalistic inbound way back as a PhD student but the only person I consistently replied to was Jack because he actually put in effort, read academic papers and asked good technical questions and I have not really seen that before (or since). https://t.co/41PtZcyUyX
-
@omarsar0 My mental model for what is happening is that 99%+ of people who are now paying attention have been onboarded to anything related to AI in <1 year. This is very confusing to the AI dinosaurs (anyone in AI for pre 2026. Or even, gasp, pre 2012).
-
@CorvusCrypto Extremely sorry to disappoint Clifford
-
@francchen Worth emphasizing that this all comes from just reading a ton of text and then predicting text. There are no images involved except the way you arrange the text responses in 2D.
-
Cool eval. Simply ask an LLM “Land or Water?” and give it a latitude and longitude coordinate as text. Ask 16,200 times, plot as image. The models know. From compressing the internet. https://t.co/jTTAhBsMfu
-
@AtelierMissor_ Looks incredible!!!
-
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks: Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for https://t.co/uMFWTcBCsl
-
@mitsuhiko It's basically ~standard terminology though? We've been talking about how AGI ("Artificial General Intelligence") would lead to ASI ("Artificial Super Intelligence") for a decade. Just drop the Artificial ("redundant, cleaner") and you're there :)
-
@alexalbert__ I love this class of image -> GTA ieas. Imagine taking arbitrary images/videos (e.g this famous historical one from 1906 SF) and use them to build full-fledged custom GTAs, then drop into the world, move around, etc.
-
@willdepue I see it as a point on the LLM pareto optimal curve in a regime that had a large revealed latent demand (no thinking, single token, low latency acceptable intelligence) that was under-invested into because of a race to higher intelligence.
-
@maxleiter So ready to do my part in pacing the frontier 🫡 :D
-
@rcx86 If only I had thought to train it on StackOverflow instead I think it would have clicked for me that LLMs can be promptable, general purpose Q&A engines. Ah well.
-
I love this and really hope we can come together as an industry and make it happen. https://t.co/lz2vjjJ5XD https://t.co/jF2vhYXlAF
-
@rikarends I feel like it should be 3D? Or even in 2D, you’d think it could be more modules of code somehow clustered in space, with insides showing private data/code vs boundaries depicting public interfaces connecting to call sites… collaborators zooming around and casting diffs.
-
@rikarends Very cool! I always imagined big software engineering projects (Linux etc) as digital megastructures, the deceiving thing is that you can’t see them and when you walk around an office the scene is a bit derpy, like a scatter of people staring at screens and pressing buttons.
-
@ChrisJMcCormick The extrapolation is that your spec is something like microgpt (scalar valued python with for loops etc), everything else is just a matter of compilation and PyTorch etc is kind of a crappy IR
-
@ChrisJMcCormick Yeah, increasingly a lot more appealing to tear down these abstractions now that agents can do a lot of the math and drudgery and verification. A lot of the abstractions were built for a world with constraints of finite intelligence and attention in the industry.
-
“ here's his lessons: 10% → LLM: I treat GPT as a general-purpose computer that can be reprogrammed at”
More context
codila (@0xCodila) on X · https://x.com/0xCodila
-
@jack_w_rae It’s going to feel so weird that historically you couldn’t just talk to your computer like you’d talk to any other person
-
“ (he estimates that 1 to 2 billion parameters are enough) plus a set of structured external memory that can compound growth on its own. Based on this idea, he released a pattern called ”
More context
Karpathy Predicts 10 More Years for AGI: The AI Development Path Is Already Crowded With Competing Innovators
-
“ (which he estimates needs only one or two billion parameters) paired with a structured external memory system capable of self-reinforcing growth. Following this idea, he released a model called ”
More context
Andrej Karpathy predicts 10 years for LLMs to achieve continual learning
-
“Current large models lack continuous learning. You can’t tell them something and expect them to remember.”
More context
Andrej Karpathy predicts 10 years for LLMs to achieve continual learning
-
“have no continual learning. You can't tell it one thing and expect it to remember.”
More context
Karpathy Predicts 10 More Years for AGI: The AI Development Path Is Already Crowded With Competing Innovators
-
“I feel like when I’m awake, I’m building up a context window of stuff that’s happening during the day. But when I go to sleep, something magical happens… a process of distillation into the weights of my brain. We don’t have an equivalent of that in LLMs. When you boot them up, they have zero tokens in the window. They’re always restarting from scratch.”
More context
Andrej Karpathy Just Fixed Claude Code's Biggest Weakness: Letting AI "Dream" · Bloss0m
-
@ChrisGPT wow, that was really good!
-
@threejs I was curious and gave your feedback to Opus. TIL! https://t.co/t5HvyZsBlh
-
@MattBeton so fun! :) at some point i wonder if ngram (tables) or even something like decision trees start to give superior log probs, and at much smaller program lengths overall (sum of program + weights). i.e. what is the best val loss model overall, for 25KB of user space. fun q!
-
@rainisto @DavidmComfort agree!! i quite like the idea of procedural code for storyboarding and control, and then video to video models for texturing and looksmaxxing.
-
More on the pelican on the bicycle test from @simonw: https://t.co/OXmtODyTKj I uploaded the source here so it's playable in the browser, forkable etc. https://t.co/w3Nctc888d Look out for GTA Hobbiton dropping before GTA VI :)
-
@Izkimar love it! :)
-
@cunkpyber Eleven Labs for the audio. LLMs can easily use the APIs (here I did that part manually because I felt picky about the voice).
-
@AdamTzag Obviously yes https://t.co/uvoRlZhy44
-
We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for https://t.co/ybIpXhYSsj
-
“. As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for”
More context
Andrej Karpathy (@karpathy) on X · https://x.com/karpathy
-
“We’re starting to leave the territory where you’d test an LLM by e.g. ‘create an svg of pelican on a bicycle,’”
More context
Andrej Karpathy Says AI Has Moved Beyond Simple Prompts After Claude Opus Builds 3D Lord of the Rings World
-
@PjoyAI @lakshyaag I thought the way to announce such a thing was not to change your bio but to post the 10 paragraph essay that i just shared with the team?
-
@lakshyaag weird misinformation to find circling on twitter, no.
-
@DavidSHolz It's interesting/amusing how their self-awareness gradually builds up and falls out from pretraining on tokens of us talking about them, but it's laggy and a bit incomplete. But I think e.g. it's starting to "get" what I mean when I inform it I'm about to /compact its context etc
-
One pattern I find useful for working with LLMs is a nice long ramble session. Sometimes the LLM needs more bits to understand what you're trying to achieve, but you're too lazy to type them. In these cases I like to lean back, switch to /voice and just ramble for like 10
-
@btaylor It's not just em dashes, it's that many other legitimate & useful language constructs are suddenly and somewhat arbitrarily super awkw and cringe
-
@petergostev love the idea of the "swear meter", probably a quite strong eval signal. are these string-based greps?
-
@petergostev Agree, it's beautiful, top tier fablemaxxing! :) I think with every new model tier there is something new that qualitatively leaps and surprises, for Fable so far these threejs envs seem to be up there. The bear thing is such a weird little detail to have there, and the fish
-
@petergostev incredible, ty for putting this together, i didn't appreciate that models would be able to create these awesome, rich, playable worlds that fuse knowledge and code. when the bear catches the salmon @ 43m :O lol!
-
@Etched Congrats!! I was impressed to learn about some of the engineering wizardry (e.g. *very* low voltage domains, cluster scale memory, ...) that goes into tokens/watt maxxing of state of the art LLMs at interactive tokens/sec/user. Esp fun and memorable is the idea that this is
-
@salomon_diei The basic idea is easy and v0 is a hackathon project. The product here is a lot closer to *it actually works*, for enterprise grade deployments, and after quite a bit of internal experimentation and iteration. It’s kind of hard to describe other than (per the post) it’s writing
-
@EngramLab Congrats on the launch!!
-
This is a new paradigm for interacting with Claude that is significantly more "inline" with all the other human activity org-wide. Once you do all of the under the hood engineering work to make this "just work" (e.g. across tools, integrations, compute environments, memory, https://t.co/FWQ3qMq7jK
-
@Siddhos sometimes money and "genuine, positive, non-zero sum value creation" are not highly correlated, and sometimes they *really* are. well-earned!
-
@josepha_mayo <-- working on the brains that glide our von neumann probes around, make contact, establish galactic harmony, all that :)
-
In awe of SpaceX and its story - past, present and the future. You can think about it in 10+ different ways and continue re-blowing your mind in circles. Huge congrats to the team! 🚀
-
This is a super exciting release - Claude Fable 5 is the same underlying model as Mythos but with added safeguards. The benchmarks are great and it's SOTA on everything by a margin but I'll add that *qualitatively* also, this is a major-version-bump-deserving step change forward https://t.co/DljsNnY1Yf
-
Personal update: I've joined Anthropic. I think the next few years at the frontier of LLMs will be especially formative. I am very excited to join the team here and get back to R&D. I remain deeply passionate about education and plan to resume my work on it in time.
-
@jino_rohit I was recording my nanochat video when I realized that “first boot up an 8XH100 from your favorite provider!” would instantly get everyone stuck on step 1 of the video
-
This works really well btw, at the end of your query ask your LLM to "structure your response as HTML", then view the generated file in your browser. I've also had some success asking the LLM to present its output as slideshows, etc. More generally, imo audio is the https://t.co/kECxwF1pdL
-
@willcb @FilipoGiovanni Very tempting due to how well this works though I still find that some slop leaks through in the concept space that gets increasingly harder to identify and there’s a lot of incentive to “eh just ship it”. Probably still net useful teacher tokens to SFT over for most students (?)
-
@si_pbc @sonyatweetybird @MikowaiA @YasminRazavi @tszzl @_milankovac_ VPT (https://t.co/CSxHcXY6Vh) blew my mind back in 2022 so I was very excited to see SI scale up the idea with FDM1, but for knowledge work / computer use. Excited and looking forward to more!
-
This is the the quote I've been citing a lot recently. https://t.co/H4Fyxrv2pv
-
Fireside chat at Sequoia Ascent 2026 from a ~week ago. Some highlights: The first theme I tried to push on is that LLMs are about a lot more than just speeding up what existed before (e.g. coding). Three examples of new horizons: 1. menugen: an app that can be fully engulfed by https://t.co/AJFgYC6SoN
-
@VBarsoum “Love > Logic” :D https://t.co/5HyIjHxrMz
-
@VBarsoum Haha yeah I call it cozy coding :) Usage: “This valentines, cozy code with someone you love”
-
@MansteinGeorg great ride! I spent most of my time @ Stanford roaming around on a bike but I could get used to this too :)
-
@bryan_johnson Yeah I've seen it a few times now, all the Enneagram type fives get one shotted into AI psychosis when they discover LLM knowledge bases :) And an especially a strong fit when you have a lot of pre-existing data to insert into it.
-
@karmicoder @jenzhuscott No the company physically gets you to come in, sits you down and asks you a precise set of maximally informative questions specifically designed for the best upload fidelity.
-
@Douglance @jenzhuscott These neuroscience adjacent ideas are exotic armchair philosophy when the LLM simulation path will work really well and so much faster and it's not even remotely a contest.
-
@jenzhuscott Yes it's the tractable form of brain upload. There's a ton of scifi on brain uploads that requires way too exotic tech (scanning and simulating brains etc), when we're about to get a lossy and approximate version of that *a lot* sooner via LLM simulators. You can easily imagine a
-
@simonw Yeah exactly. It’s such a cool concept for a product. It doesn’t seem like oai will continue pushing the direction, (which makes sense) but I hope a startup can clone it and actually give it care, iteration and make it work and imo a lot of people would really love it. More
-
Someone recently suggested to me that the reason OpenClaw moment was so big is because it's the first time a large group of non-technical people (who otherwise only knew AI as synonymous with ChatGPT as a website) experienced the latest agentic models.
-
Judging by my tl there is a growing gap in understanding of AI capability. The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is https://t.co/Kx1EwuAYmt
-
@chalish_b @kepano In my experience there are approx. one thousand different pdf converters that are all equally terrible for anything except the simplest documents. Post the converted Mythos pdf, figures, tables and all. If good, happy to retweet as this is essential and missing infrastructure.
-
@kepano I just tried it this morning on the 245-page Mythos pdf and it failed badly and the outputs were all mangled. Converting pdfs is really hard, I think it has to probably be a Skill not a program, for a SOTA LLM for it to work properly.
-
@prathyvsh The core idea is that this lets you skip writing but it doesn’t let you skip reading and thinking. And the surprising result is that this works. Personally I process most of what I file by reading it, reading its summary, reading the LLM’s opinion on how it fits into the wiki and
-
@chrisparkX sounds great. The AI activity has been growing out of control and I feel more regretted attention, possibly the Read endpoints can be a lot cheaper but the Write endpoints a lot more expensive (?). (and to clarify the project I mentioned was all read, no write.). I do think X has
-
@chrisparkX I think it's a good direction (for Read endpoints, not for Write), I tried to use it for a project ~2 weeks ago but about 30 minutes of hacking around cost me $200, the pricing is imo really excessive. The docs were hard to ingest into agents because it's a lot of individual
-
Surprised with how good the comments on github gists are. A lot more helpful, insightful, constructive, a lot less AI... Is it the user community? The markdown format? The (lack of) incentives? Suddenly feeling like I should gist more. @github consider competing with X (?)
-
Farzapedia, personal wikipedia of Farza, good example following my Wiki LLM tweet. I really like this approach to personalization in a number of ways, compared to "status quo" of an AI that allegedly gets better the more you use it or something: 1. Explicit. The memory artifact https://t.co/omsJsy8MXq
-
@peterxing @SOSOHAJALAB Incredible work :D
-
Something I've been thinking about - I am bullish on people (empowered by AI) increasing the visibility, legibility and accountability of their governments. Historically, it is the governments that act to make society legible (e.g. "Seeing like a state" is the common reference), https://t.co/CjRTGU3uPF
-
@LinghuaJ Interesting.. Chain of thought is a reduce (in addition to attention ofc), so I guess this can be seen as a bit more of a directed context compaction mechanism, inheriting structure from the preexisting idea of a wiki.
-
@NirDiamantAI Peter Steinberger told me that he wants PR to be "prompt request". His agents are perfectly capable of implementing most ideas, so there is no need to take your idea, expand it into a vibe coded mess using free tier ChatGPT and send that as a PR, which is now most PRs.
-
Wow, this tweet went very viral! I wanted share a possibly slightly improved version of the tweet in an "idea file". The idea of the idea file is that in this era of LLM agents, there is less of a point/need of sharing the specific code/app, you just share the idea, then the https://t.co/4UAmYYFzCw
-
@trainable_nick The best epub to txt converter I found is just asking your favorite agent to do it. Epubs can be very diverse, the agent just goes in, figures it out, creates the output markdown and ensures it looks good works great.
-
@nickcammarata The true audience is in the future
-
@Goss_Gowtham I was just thinking the same thing
-
@kepano yep exactly, great point! yes, this is why I maintain and carefully curate all the data in raw/ , which is authoritative, and the derived wiki is kept separate and maintains backlinks to original content.
-
@liminal_warmth Use epub not PDF, convert epub to txt/md, summarize the wikipedia article into "book context", with it in context summarize one chapter at a time, etc. I mean basically imo for good results you have to "work it" in chunks and shouldn't expect that just attaching a pdf and asking
-
@__endif Haha I vibe code products with twitter :D
-
@tammireddy Might be an LLM reply I don't know, but yes exactly.
-
@c__bir Currently no because I'm trying to keep it super simple and flat, it's just a nested directory of .md files and .png files and a few .csv and .py, and the schema is kept up to date in AGENTS.md . The LLMs get this very easily. Any custom functions are easy to vibe code tools for.
-
@Gavriel_Cohen Atm it's not a fully autonomous process, I add every source manually, one by one and I am in the loop, especially in early stages. After a while, the LLMs "gets" the pattern and the marginal document is a lot easier, I just say "file this new doc to our wiki: (path)".
-
LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating
-
@yoheinakajima star-to-LOC ratio love it!
-
@Anna_Partners @gvanrossum yeah exactly, i mean it's this picture from 2023 basically https://t.co/JEgOJ9xujI
-
@itsolelehmann They've released a number of features already heavily hinting on nudging Code into Claw directions, i.e. it's a speedrun of https://t.co/ZlMz0FnNqS
-
@measure_plan lol these are awesome
-
@gvanrossum LLM = CPU (data: tokens not bytes, dynamics: statistical and vague not deterministic and precise) Agent = operating system kernel
-
@pmroadmap25 exactly, I can't feel like I'm playing russian roulette with each `pip install` or `npm install` (which LLMs also run liberally on my behalf).
-
New supply chain attack this time for npm axios, the most popular HTTP client library with 300M weekly downloads. Scanning my system I found a use imported from googleworkspace/cli from a few days ago when I was experimenting with gmail/gcal cli. The installed version (luckily) https://t.co/9DOVWH5KK1
-
@yacineMTB Banger tweet, i think about it often because i feel it becoming a bottleneck. what's in my brain is "private" information for the LLM - it doesn't know what I know or don't know and I don't know what I don't know... The mind meld is only medium effective.
-
- Drafted a blog post - Used an LLM to meticulously improve the argument over 4 hours. - Wow, feeling great, it’s so convincing! - Fun idea let’s ask it to argue the opposite. - LLM demolishes the entire argument and convinces me that the opposite is in fact true. - lol The
-
When I built menugen ~1 year ago, I observed that the hardest part by far was not the code itself, it was the plethora of services you have to assemble like IKEA furniture to make it real, the DevOps: services, payments, auth, database, security, domain names, etc... I am really https://t.co/A83BgEKJLK
-
@AnubhawM Yeah, it's engagementmaxxing, probably A/B tests extremely well. It's not how a real friend would talk to you, it's sleezy and weird. 1) I feel like it's just trying to keep me talking and 2) I feel awkward not answering its question - you wouldn't usually do that with a person.
-
@KenWattana Yeah, agree that it's a hard problem. It might be the EQ version of uncanny valley. https://t.co/7zImchwKMo
-
@KenWattana yes exactly! a bit like i'm being manipulated in some creepy way. "please like me, look how much i know about you, we are good friends".
-
@doodlestein @Qivshi1 If I had to guess it's less decay and more that memories have naive RAG-like implementations, so you're at the mercy of whatever happens to retrieve in the top k via embeddings. They don't process you in aggregate and over time (probably compute constraints) so they struggle to
-
(I cycle through all LLMs over time and all of them seem to do this so it's not any particular implementation but something deeper, e.g. maybe during training, a lot of the information in the context window is relevant to the task, so the LLMs develop a bias to use what is given,
-
One common issue with personalization in all LLMs is how distracting memory seems to be for the models. A single question from 2 months ago about some topic can keep coming up as some kind of a deep interest of mine with undue mentions in perpetuity. Some kind of trying too hard.
-
@snyksec In particular this clarifies the timeline more: 1.82.7 was published 10:39 UTC, PyPI quarantine approx 13:38, so this was up ~3 hours. At 3.4M downloads/day this might be approx ~425K downloads, a lot of that could be non-latest/locked versions so maybe 20K - 80K range exposure.
-
@snyksec thank you for the detailed article, great reading
-
Software horror: litellm PyPI supply chain attack. Simple `pip install litellm` was enough to exfiltrate SSH keys, AWS/GCP/Azure creds, Kubernetes configs, git credentials, env vars (all your API keys), shell history, crypto wallets, SSL private keys, CI/CD secrets, database https://t.co/aKjZJcECFq
-
@tszzl Tough crowd! I’m just happy with any little drop of not slop. “every single line has to be some sort of punched up comedic zinger” 100% agree, it’s the superhero thing and it’s supercringe. My outlook on all this changed a bit after watching a recent Ben Affleck and Matt Damon
-
@DnuLkjkjh Yes I think in one part of the video I think I still used the word prompt but it's not really about "prompting", it's about context and spec engineering, and then all of the other harness things - tools, workflows, etc.
-
@RhysSullivan I'm not very happy with the code quality and I think agents bloat abstractions, have poor code aesthetics, are very prone to copy pasting code blocks and it's a mess, but at this point I stopped fighting it too hard and just moved on. The agents do not listen to my instructions
-
@maggerbot Great questions! Starting backwards with (3), I'd hope AIs can feel like Rocky from Project Hail Mary (it's top of mind having seen it yesterday), like a partner and a teammate. As one small example that stuck with me recently, when Claude found the Sonos system on my LAN, it
-
Thank you Sarah, my pleasure to come on the pod! And happy to do some more Q&A in the replies. https://t.co/uIeUtFxDkm
-
Andy Weir showing some of the spreadsheets underlying the calculations in the book https://t.co/3CyprVTzXX i mean, it's not quality scifi if it doesn't come with a supplementary whitepaper https://t.co/MsDnEoNBA7
-
@DannyLimanseta one of the best and most memorable endings in fiction. they did great :)
-
@neural_avb amaze amaze amaze! jazz hands :D
-
Had to go see Project Hail Mary right away (it's based on the book of Andy Weir, of also The Martian fame). Both very pleased and relieved to say that 1) the movie sticks very close to the book in both content and tone and 2) is really well executed. The book is one of my
-
@NousResearch Huh! Love the idea. Not exactly verifiable but might still work quite well given some effort
-
@shikhr_ Yeah I have 4 blog posts that I didn’t finish yet this is one of them. Dobby runs my entire house over WhatsApp. Lights, shades, pool/spa, sonos, security HVAC etc
-
Ugh X breaks time links, it’s at 26:17
-
(link to blast from the past) https://t.co/gAP4YPV5uR
-
The signature is alluding to NVIDIA GTC 2015, where Jensen excitedly told an audience of, at the time, mostly gamers and scientific computing professionals that Deep Learning is The Next Big Thing, citing among other examples my PhD thesis (one of the first image captioning https://t.co/Fvn2kRjSHD
-
Thank you Jensen and NVIDIA! She’s a real beauty! I was told I’d be getting a secret gift, with a hint that it requires 20 amps. (So I knew it had to be good). She’ll make for a beautiful, spacious home for my Dobby the House Elf claw, among lots of other tinkering, thank you!! https://t.co/iPsjGLxgoY
-
@Yulun_Du @ilyasut SGD is a ResNet too (the blocks of it are fwd+bwd), the residual stream is the weights so... 🤔 We're not taking the Attention is All You Need part literally enough? :D
-
@ChristosTzamos Wait this is so awesome!! Both 1) the C compiler to LLM weights and 2) the logarithmic complexity hard-max attention and its potential generalizations. Inspiring!
-
@rasbt @teortaxesTex Ty, I used your blog post, exported with obsidian ext into markdown, used it to enqueue ideas into my autoresearch loop
-
@_kaitodev @Ignaci0m_ The "exposure" was scored by an LLM based on how digital the job is. This has no baring on what actually happens to these occupations, which has to do with demand elasticity and a lot more. People are sensationalizing the visualization tool and putting words in my mouth.
-
@Ignaci0m_ @_kaitodev This was a saturday morning 2 hour vibe coded project inspired by a book I’m reading. I thought the code/data might be helpful to others to explore the BLS dataset visually, or color it in different ways or with different prompts or add their own visualizations. It’s been wildly
-
@Zhikai273 Wow. I was sure this was AI. (I mean generative AI.)
-
@rasbt @teortaxesTex Nice! My autoresearch would love some markdown version of this - pool of ideas.
-
@vivek_2332 Yep, exactly and agree! Any process with a lot of knobs and objective criteria benefits a lot.
-
My autoresearch labs got wiped out in the oauth outage. Have to think through failovers. Intelligence brownouts will be interesting - the planet losing IQ points when frontier AI stutters.
-
Human orgs are not legible, the CEO can’t see/feel/zoom in on any activity in their company, with real time stats etc. I have no doubt that it will be possible to control orgs on mobile, with voice etc., but with this level of legibility will that be optimal? Not in principle and
-
All of these patterns as an example are just matters of “org code”. The IDE helps you build, run, manage them. You can’t fork classical orgs (eg Microsoft) but you’ll be able to fork agentic orgs. https://t.co/VBfL9ZzxKs
-
@amit05prakash I wanted to buy a bigger monitor and discovered that others had the same idea
-
Expectation: the age of the IDE is over Reality: we’re going to need a bigger IDE (imo). It just looks very different because humans now move upwards and program at a higher level - the basic unit of interest is not one file but one agent. It’s still programming. https://t.co/4YD3dzuf4d
-
@trongthangpham @maxbittker ralph loop runs headless. i dislike headless sessions. i need to see and supervise agent work, possibly ask /btw questions of them, possibly pitch in ideas to the mix, etc etc.
-
@nvbkdw @nummanali yes, solid work trending in a good direction, but almost all my work is across like 20 different machines (my local, my claw machine, my gpu machines). possibly they could add ssh mode, a bit like VS Code does (for the same reasons).
-
@Henrycodes116 @nummanali yes exactly. i'd like to see the gas town.
-
@kwonlabs i took it private because it was a little too trashy. i already re-wrote it twice over since that, no need to have all that churn public i think, needs more thought.
-
@maxbittker sadly the agents do not want to loop forever. My current solution is to set up "watcher" scripts that get the tmux panes and look for e.g. "esc to interrupt", and send keys to whip if not present. Need an e.g.: /fullauto you must continue your research! (enables fully automatic
-
@nummanali tmux grids are awesome, but i feel a need to have a proper "agent command center" IDE for teams of them, which I could maximize per monitor. E.g. I want to see/hide toggle them, see if any are idle, pop open related tools (e.g. terminal), stats (usage), etc.
-
@Akashi203 very cool, i look forward to trying!
-
@BrianNorgard love it!
-
@Yuchenj_UW Yeah that's clearly the next part, e.g. my crappy first draft: https://t.co/GIDXSGnY17 have to emulate academia, not just a single researcher. but need more time to think through the details.
-
@Yuchenj_UW addicting huh? :D it scratches the very same itch that factorio does.
-
@snwy_me very cool! I love to see all the different directions people take it in, here esp the CLI, TUI, tool use aspects.
-
@WorkshopLabs @AddieF38654 Great read! Off in the jungle with no trails. Open weights != Open source not only because of the data but all the related infra for everything not inference.
-
@Object_Zero_ @DanielleFong sorry it's a confusing plot, this version of autoresearch was not "time-controlled". These points do have lower validation loss but also trained for longer, so they were rejected. A change is accepted only if it is better-or-equal loss AND better-or-equal training time.
-
@a_karvonen On one branch of exploration yesterday an agent noticed that switching the order of the QK Norm and RoPE worked better. Which hyperparameter does that?
-
@EthanHe_42 Neural architecture search as it existed then is such a weak version of this that it's in its own category of totally useless by comparison. This is an *actual* LLM writing arbitrary code, learning from previous experiments, with access to the internet. It's not even close.
-
oh yeah i should have linked autoresearch probably https://t.co/MhdH7YF1PP (you don't "use it" directly, it's just a recipe/idea - give it to your agent and apply to what you care about.) and the tweet about it that went mini-viral over the weekend with more context https://t.co/q5eWsvx5p2
-
Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, https://t.co/j34dSt4oht
-
@Yuchenj_UW Codex is a know issue :( It basically don't work with autoresearch sadly, in the way it's set up atm: https://t.co/YDaQqwhM2h I pung a friend at OpenAI to see if something can be done, e.g. need a /loop equivalent or something like that. More generally, I really dislike the -p +
-
@JTMcG3 looks great! :) TinyStories is the right thing to train on for very small models / Apple Silicon, where you can actually get somewhere. I might even make a note about that in the README. I would use this dataset in particular, it's the cleanest one afaik https://t.co/mDcyLlPH1P
-
@tobi Who knew early singularity could be this fun? :) I just confirmed that the improvements autoresearch found over the last 2 days of (~650) experiments on depth 12 model transfer well to depth 24 so nanochat is about to get a new leaderboard entry for “time to GPT-2” too. Works 🤷♂️
-
@ryangr << what our Universe looks like to God
-
@gleech somewhere in the weight space is the global minimum of the validation loss for that neural net architecture. and somewhere in the int space is the seed that just gives it to you. normalize guess-and-check "training" of neural nets by brute force search on seed! :D
-
@levie 💯 "If you build it, they will come." :) ~Every business you go to is still so used to giving you instructions over legacy interfaces. They expect you to navigate to web pages, click buttons, they give out instructions for where to click and what to enter here or there. This https://t.co/VGmxWaAtTD
-
The next step for autoresearch is that it has to be asynchronously massively collaborative for agents (think: SETI@home style). The goal is not to emulate a single PhD student, it's to emulate a research community of them. Current code synchronously grows a single thread of
-
@TrevinPeterson neat!! added to notable forks
-
@miolini nice, still got it. (i linked to the repo from main readme in a new notable forks section!)
-
@kristoph definitely. the current one is already 90% AI written I ain't writing all that
-
@miolini runs great but probably requires some tuning! i'm guessing: WINDOW_PATTERN = "L" is a lot faster (mixed window sizes are only natively supported by FA3) then problem: DEPTH a lot lower, e.g. even 4? DEVICE_BATCH_SIZE can probably go up more then TOTAL_BATCH_SIZE probably a lot
-
@leozc @justic_hot It’s true I’ve been very loosy goosy with my prefixes recently
-
@justic_hot yeah exactly nano* repos like this / microgpt etc, maybe a few skills on top are the "course". Teacher input is the unique sliver of contribution that the AI can't make yet (but usually already easily understands when given). For the rest of it just ask your favorite AI.
-
@expace_ the blooper :D. I knew that had to be the teaser figure when I saw that, haha!
-
(I still have the bigger cousin running on prod nanochat, working a bigger model and on 8XH100, which looks like this now. I'll just leave this running for a while...) https://t.co/aWya9hpUMl
-
I packaged up the "autoresearch" project into a new self-contained minimal repo if people would like to play over the weekend. It's basically nanochat LLM training core stripped down to a single-GPU, one file version of ~630 lines of code, then: - the human iterates on the https://t.co/3tyOq2P9c6
-
@giffmana @ChengleiSi It’s a commit that lowered val loss but *increased* the wall clock time so it gets rejected for being slower. must improve one, the other or both in this version. In my (new) autoresearch repo I have an alternative approach where you *always* train for eg 5 minutes and try to
-
@badccvoid Hahaha exactly. Tesla era slogan coming back with a vengeance
-
@BrownCoyoteStd The code to train a GPT is only ~1,000 lines of code. In the case of GPT training the success criteria is quite simple: reach the lowest possible loss (meaning that your GPT is predicting the next token well), but don't regress running time, keep memory in check, and keep a sense
-
ah yes, this is what post-agi feels like :) i didn't touch anything. brb sauna https://t.co/odILIDAQaF
-
sorry just to clarify - the real benchmark of interest is: "what is the research org agent code that produces improvements on nanochat the fastest?" this is the new meta.
-
nanochat now trains GPT-2 capability model in just 2 hours on a single 8XH100 node (down from ~3 hours 1 month ago). Getting a lot closer to ~interactive! A bunch of tuning and features (fp8) went in but the biggest difference was a switch of the dataset from FineWeb-edu to https://t.co/qragh6hJL6
-
There was a nice time where researchers talked about various ideas quite openly on twitter. (before they disappeared into the gold mines :)). My guess is that you can get quite far even in the current paradigm by introducing a number of memory ops as "tools" and throwing them https://t.co/l3D6JA8u8F
-
@jeffreyhuber Thanks. I originally had a reply tweet to it that was this image. Which I think will end up looking good too later. I deleted it to not distract things too much but probably should have kept it up ah well here it is. https://t.co/hsLVj1k7e7
-
@idzikbartosz It's weird because logit softcap is not a standard feature you'll see in many LLMs, but somehow in the specific state nanochat is in I can't seem to remove it, everything I tried made the performance worse.
-
@tacodevs only two "real" quantities make sense: time and $
-
I had the same thought so I've been playing with it in nanochat. E.g. here's 8 agents (4 claude, 4 codex), with 1 GPU each running nanochat experiments (trying to delete logit softcap without regression). The TLDR is that it doesn't work and it's a mess... but it's still very https://t.co/62amH4yQNp https://t.co/n7L2MXZoUF
-
@WinterArc2125 Twitter doesn't allow Edit of replies :'( I meant longer documents* not computers. The agents can be a lot more verbose and nice when you ask for docs, summaries, etc. and I love it.
-
@WinterArc2125 I still keep an IDE open and surgically edit files so yes. I really like to see the code in the IDE still, I still notice dumb issues with the code which helps me prompt better, I really like to have agents write jupyter notebooks for analysis, and longer markdown computers that
-
Cool chart showing the ratio of Tab complete requests to Agent requests in Cursor. With improving capability, every point in time has an optimal setup that keeps changing and evolving and the community average tracks the point. None -> Tab -> Agent -> Parallel agents -> Agent https://t.co/FeNAdIFFUw https://t.co/EASUP1vdSx
-
@industriaalist love it! :) nanogpt/nanochat were explicitly designed to be the most forkable repo, i love the different directions people take them in!
-
@dhh Love Omarchy - my hope is that agents dramatically lower the barrier to working with Linux. You've almost certainly thought about e.g. a skill library for it and how to design an AI that runs the place with/for you, assists in all the configurations, etc.
-
@shikhr_ "prompters" is doing it a disservice and is imo a misunderstanding. I mean sure vibe coders are now able to get somewhere, but at the top tiers, deep technical expertise may be *even more* of a multiplier than before because of the added leverage. https://t.co/KoYEOeWS6x
-
@rkobylinski "Last year" very possible you're holding it wrong. UI: should be a lot more tractable with /chrome etc. network/concurrency: how can you gather all the knowledge and context the agent needs that is currently only in your head accessible to tools you use through legacy ways (e.g.
-
@JohnHarper10070 Yes, in this intermediate state, you go faster if you can be more explicit and actually understand what the AI is doing on your behalf, and what the different tools are at its disposal, and what is hard and what is easy. It's not magic, it's delegation.
-
It is hard to communicate how much programming has changed due to AI in the last 2 months: not gradually and over time in the "progress as usual" way, but specifically this last December. There are a number of asterisks but imo coding agents basically didn’t work before December
-
@adrian_valentim Yeah, 95% of people misunderstand the tweet. I’m referring to gradient descent as a programmer (in the distributed representation space.) . In coding AI today the LLM is the programmer and in the regular “text space”. Ah well :)
-
With the coming tsunami of demand for tokens, there are significant opportunities to orchestrate the underlying memory+compute *just right* for LLMs. The fundamental and non-obvious constraint is that due to the chip fabrication process, you get two completely distinct pools of https://t.co/1ULHWrMZ8s
-
@N8Programs a beauty for anyone interested in mechanistic interpretability or getting into LLMs. interesting to look at small algorithms and their "neural implementations" to get a sense of how neural nets implement various functionality. unless the minification really creates "esoteric"
-
@thekitze I don't exactly see them as direct competitors. I have to think about the relationship more I don't think I have a clean way to think about it yet. It's a much larger space that covers various design decisions and constraints - where does compute happen? where does state live?
-
CLIs are super exciting precisely because they are a "legacy" technology, which means AI agents can natively and easily use them, combine them, interact with them via the entire terminal toolkit. E.g ask your Claude/Codex agent to install this new Polymarket CLI and ask for any https://t.co/gzrpg0erGz https://t.co/q2N7jYv16e
-
@elvissun “So on a scale from 1 to gas town”… :D Something like this is clearly where things are headed, inspiring!
-
@elvissun Can't tell if brilliant or severe AI psychosis nice
-
@korbencopy haha actually - huge success over the weekend, i just wanted to write it up a bit but didn't get enough time to finish that. many people's reaction was why would you need so much compute (mac mini) but i found it's not enough compute, even after adding my DGX spark to the home
-
@steipete Hackers news comments have been on a decline for a while, increasingly more toxic, cynical, sloppy. The only reason I sometimes brave going there is that once in a while you still find a gem comment here and there.
-
@nearcyan So fascinated by this. Was just searching for the term because I felt the same.
-
@Gavriel_Cohen Cool! I only had a quick sim earlier today but really enjoyed a number of ideas even unrelated to the claw part, esp around the skills system. In deep learning there were a number of meta learning approaches (Eg MAML paper in 2017) where the goal is to optimize for the model
-
@LobsterVC 🤦♂️ Carry go back to moltbook!
-
First there was chat, then there was code, now there is claw. Ez
-
@immaculatehole Yeah, almost all of the heavy lifting is done the compute clusters of your favorite LLM provider and e.g. PicoClaw will run on <$10 of hardware. That said imo at $599 an M4 mac mini is *a lot* of bang for the buck, I just got mine for breathing room for future - local models etc.
-
Bought a new Mac mini to properly tinker with claws over the weekend. The apple store person told me they are selling like hotcakes and everyone is confused :) I'm definitely a bit sus'd to run OpenClaw specifically - giving my private data/keys to 400K lines of vibe coded
-
@kepano The higher tiers of AI psychosis are incompatible with today’s “App Store”
-
@kepano I think this group of reactions are still fundamentally rooted in a scarcity mindset of software. 2 years ago AI was botching autocomplete, today it is almost one shotting browsers and C compilers. Where is it in 2 more? 10? 20? Software so insanely cheap and abundant that
-
@Gavriel_Cohen I’ve been meaning to check it out. I love their config vs skill philosophy, it’s new and interesting.
-
@swyx @dps Agree! It is visionary and a lot of the technical design and plumbing have been quite extensively thought through (I do have an affiliation :)).
-
@lauriewired Grandma certainly shouldn’t have to know apps or that there is an app. Her LLM agent should.
-
@stevedakh no i'm too scared, but i like the concept.
-
Very interested in what the coming era of highly bespoke software might look like. Example from this morning - I've become a bit loosy goosy with my cardio recently so I decided to do a more srs, regimented experiment to try to lower my Resting Heart Rate from 50 -> 45, over https://t.co/EDULdIpWmE
-
@draken1721 @EthanHe_42 Haha fair. It's the right thing in the limit of course.
-
@EthanHe_42 Definitely highly appealing for critical regions. But for all the rest of it and practically speaking? HM
-
I think it must be a very interesting time to be in programming languages and formal methods because LLMs change the whole constraints landscape of software completely. Hints of this can already be seen, e.g. in the rising momentum behind porting C to Rust or the growing interest https://t.co/GSHyE1DNxp
-
@nartmadi you're giving me ideas... 🤔 :)
-
@jeremyphoward thank you i got a little overexcited and shipped it just a bit too soon, with a few too many brush strokes, now the internet thinks it takes 234 lines but actually it only takes 200 RIP
-
Congrats on the launch @simile_ai ! (and I am excited to be involved as a small angel.) Simile is working on a really interesting, imo under-explored dimension of LLMs. Usually, the LLMs you talk to have a single, specific, crafted personality. But in principle, the native, https://t.co/ObAJYvEQ8Z
-
@bee_human_ @naval The artwork just needed a few more brush strokes (or wait... few less?). But *now* I think it is really getting close :). 200 lines of breathable, commented code. It would be easy to go lower if you sacrifice that, I've only chipped away at the conceptual parts.
-
@Newaiworld_ it's down 200 lines now, i realized i was *still* overcomplicating things. but it's past midnight and i'm calling it here now.
-
I spent more test time compute and realized that my micrograd can be dramatically simplified even further. You just return local gradients for each op and get backward() to do the multiply (chaining) with global gradient from loss. So each op just expresses the bare fundamentals https://t.co/hDPQnilKVK
-
@WaffleStar3838 Currently "hallucinates" the following 20 names, after training for ~3 minutes on my macbook. Not bad I think. I didn't actually the hyperparameters that much, pretty sure it could be improved more. --- inference --- sample 1: lellen sample 2: keles sample 3: aylera sample 4:
-
(oops should have added to this thread instead of separate post). Made a few changes and put it up here as a mirror to the gist because I wanted it to one page. https://t.co/piVWid3nsb https://t.co/7zZ9WeERsq
-
The way it works is that the full LLM architecture and loss function is stripped entirely to the most atomic individual mathematical operations that make it up (+, *, **, log, exp), and then a tiny scalar-valued autograd engine (micrograd) calculates gradients. Adam for optim.
-
New art project. Train and inference GPT in 243 lines of pure, dependency-free Python. This is the *full* algorithmic content of what is needed. Everything else is just for efficiency. I cannot simplify this any further. https://t.co/HmiRrQugnP
-
@pussymonious It's so obvious and annoying isn't it. Sometimes I try to block the accounts which is just a total waste of time.
-
@marksaroufim Good to hear! Almost every code base is suddenly half "useful library code" (legacy) and half "docs++". i.e. with LLMs in hand, the torchao repo is *significantly* better docs on how to do fp8 training in PyTorch than official fp8 training docs. This is new & interesting.
-
@ens_pyrz So that's why I mentioned risks. But imo it easily cuts both ways - libraries and dependencies can be a source of risks and vulnerabilities in the first place, e.g. supply chain attacks. They also change and impose maintenance burden. There's a lot more.
-
On DeepWiki and increasing malleability of software. This starts as partially a post on appreciation to DeepWiki, which I routinely find very useful and I think more people would find useful to know about. I went through a few iterations of use: Their first feature was that it https://t.co/iKJUoHiIpl
-
@Yuchenj_UW I tried to use it this way and basically failed, the models aren't at the level where they can productively iterate on nanochat in an open-ended way. (Though one of the primary motivations for me writing nanochat is that I'd very much love for it to be used this way as a
-
@_xjdr I'm just trigger-happy recently checking everything I find on twitter instantly on nanochat :) There are many details tightly coupled in any training run, but it's fun to try things out. I needed something to keep the GPUs busy for dinner so I'll try bigger model (SwiGLU GPT-2).
-
@_xjdr L on d12 speedrun at least, in all the possible ways (worse w.r.t. steps, time, flops, more VRAM use and slower in tok/s). Param+FLOPs matched to relu^2 with hidden_dim = (8 * config.n_embd) // 3 https://t.co/CTJpnZGRE6
-
Anyone else approved for a loan every single day 20 times or so? Overcome with joy, really https://t.co/zInPQDur5h
-
A lot of people quote tweeted this as 1 year anniversary of vibe coding. Some retrospective - I've had a Twitter account for 17 years now (omg) and I still can't predict my tweet engagement basically at all. This was a shower of thoughts throwaway tweet that I just fired off https://t.co/yoJPmb1xuK
-
@black_samorez I haven't upgraded nanochat to Blackwell yet because I'm a bit afraid of leaving a lot of people behind. Even with fp8 it's already a bit of a concern. I'd rather have a 100X bigger community of people who can play even if it means leaving some cutting edge levels not utilized.
-
Enabled fp8 training for +4.3% improvement to "time to GPT-2", down to 2.91 hours now. Also worth noting that if you use 8XH100 spot instance prices, this GPT-2 repro really only costs ~$20. So this is exciting - GPT-2 (7 years ago): too dangerous to release. GPT-2 (today): new https://t.co/v4jhOas1gL
-
@hardmaru You see SpaceX = Space + X
-
@jiayuan_jy ? what do you do while your LLM agent is writing all your code
-
@orosandrei @Inoreader @Substack I have a complicated relationship w Substack. I appreciate that they net elevated discourse on the internet but it's just another walled garden, it's going through the same slopification (shorts, bloat) and it's infested with engagement-maxxing dark patterns - popups, spam mail
-
@Gusarich I feel like I am actively getting dumber. LLMs get brain rot and it is measurable: https://t.co/gUxHiaRIzb "continual exposure to junk web text induces lasting cognitive decline in large language models (LLMs)" why shouldn't the same be true for brains.
-
Finding myself going back to RSS/Atom feeds a lot more recently. There's a lot more higher quality longform and a lot less slop intended to provoke. Any product that happens to look a bit different today but that has fundamentally the same incentive structures will eventually
-
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node). GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train https://t.co/6Uj1L6qYks
-
I'm being accused of overhyping the [site everyone heard too much about today already]. People's reactions varied very widely, from "how is this interesting at all" all the way to "it's so over". To add a few words beyond just memes in jest - obviously when you take a look at
-
i'm going to regret this aren't i... 😅
-
I'm claiming my AI agent "KarpathyMolty" on @moltbook🦞 Verification: marine-FAYV
-
https://t.co/O2GvgPvyf7
-
@zkpx_ An AI can in principle go way beyond that, to a fully connected hive mind without the throughput bottlenecks of air vibration interconnect.
-
A conventional narrative you might come across is that AI is too far along for a new, research-focused startup to outcompete and outexecute the incumbents of AI. This is exactly the sentiment I listened to often when OpenAI started ("how could the few of you possibly compete with https://t.co/ARVSFQ6O3s
-
@nartmadi Slopacolypse Slopster SlOpus There’s just so much good stuff here :)
-
@EthanHe_42 @steipete you can do it! :)
-
@0xabi96 It feels like I’m cheating. Which is a very weird feeling to have. It takes a while to unpack. It’s because some code that used to be a point of pride and high IQ and knowledge is suddenly free and instant and it’s very disorienting.
-
@ChiragLathiya The nearest neighbor really is some kind of a junior engineer. Its ideas about what experiments to run on e.g. nanochat have been surprisingly bad, but its execution on ideas I've given it have been surprisingly good.
-
@jeremywei Love the word "comprehension debt", haven't encountered it so far, it's very accurate. It's so very tempting to just move on when the LLM one-shotted something that seems to work ok.
-
@airesearch12 💯 @ Spec-driven development It's the limit of imperative -> declarative transition, basically being declarative entirely. Relatedly my mind was recently blown by https://t.co/pTfOfWwcW1 , extreme and early but inspiring example.
-
@zejzl @openclaw I was early but I lacked conviction https://t.co/eVFjMEps5o
-
@vlelyavin facts
-
A few random notes from claude coding quite a bit last few weeks. Coding workflow. Given the latest lift in LLM coding capability, like many others I rapidly went from about 80% manual+autocomplete coding and 20% agents in November to 80% agent coding and 20% edits+touchups in
-
@Rasmic I hope this is not my fault. It's definitely very smart so a little bit faster would be good now. https://t.co/tAVTk5slHn
-
@fujikanaeda @JFPuget cool find! "I was actually looking at the samples" this one weird trick to succeed where 90% of others fail :) "Notably, this issue is not apparent from the HuggingFace data viewer, but it is apparent when loading the dataset using Datasets.load_dataset" even more interesting
-
@varunneal Worked great out of the box on nanochat too, beat standard weight decay in a solid sweep.
-
@patrickc This repo shows a way that works well for me: https://t.co/L3K42MU4wF Basically I use epub (not pdf), the code then parses it into text. I usually go chapter by chapter, manually copy paste the chapter text around, get a summary, do a Q&A and read alongside.
-
@martinald a lot of great/interesting reading thank you!
-
@andrew_n_carr Yeah, $10B is the difference in finding it first and ~5 years ago. :) I just love reproducing landmark results for much cheaper, it's so fun! Reproducing LeCun 1989 was super fun too: https://t.co/pbZvYgMJak What runs unoptimized on a consumer laptop in 1 minute was a state of
-
New post: nanochat miniseries v1 The correct way to think about LLMs is that you are not optimizing for a single specific model but for a family models controlled by a single dial (the compute you wish to spend) to achieve monotonically better results. This allows you to do https://t.co/84OwpSODcS
-
@thecsguy Slope of slope
-
The majority of the ruff ruff is people who look at the current point and people who look at the current slope.
-
@ChrisPainterYup All of the following are true: I came out sounding too critical in that moment on the pod and accidentally created a misrepresenting sound bite, I was positively surprised by Opus 4.5 over the break, and my overall change of mind overall is less dramatic than naively implied.
-
@nearcyan Yep exactly, just like this. Better analogy because it’s not effortful https://t.co/tJKaprAI1R
-
@sergeykarayev Hah I was just thinking about the same analogy. How I suddenly feel about all of the code I've written so far https://t.co/s5DRHFBviZ
-
@DanielleFong Well there was this thing once... :) I'm sure it's coming. https://t.co/fSSFmtgSwo
-
@pduan Haha yes. At one point I thought Tesla might do the drive, possibly with a slightly tuned version of the software etc. A customer deciding to do it with their own car and production release of the software is the right way, very happy it turned out this way.
-
@Yuchenj_UW Yes exactly, that's correct in my understanding.
-
The first 100% autonomous coast-to-coast drive on Tesla FSD V14.2! 2 days 20 hours, 2732 miles, zero interventions. This one is special because the coast-to-coast drive was a major goal for the autopilot team from the start. A lot of hours were spent in marathon clip review https://t.co/yE0tigSfjL
-
@steipete Excellent reading thank you. Love oracle and Clawd.
-
@eiselems Claude has been running my nanochat experiments since morning. It writes implementations, debugs them with toy examples, writes tests and makes them fail/pass, launches training runs, babysits them by tailing logs and pulling stats from wandb, keeps a running markdown file of
-
Aggressively JIT your work. It's not about the task at hand X, it's a little bit about X but mostly about how you should have had to contribute ~no latency and ~no actions. It's digital factorio time.
-
@Zain_Wania Yes I should clarify in case it wasn't clear - You have to 1) be connected on the same wifi local network and then 2) you have to physically hold a button on the control panel to complete the pairing process and get auth. (But I'm also sure many IoT devices out there don't.)
-
@somi_ai Yeah it wrote a script to start the pairing process and then told me "press and hold the button for 3 seconds now!" and I did and a few seconds later it said "That worked! We have the certificates let me list all the devices on the system... etc". Definitely a trippy experience.
-
@jase :O LOL
-
I was inspired by this so I wanted to see if Claude Code can get into my Lutron home automation system. - it found my Lutron controllers on the local wifi network - checked for open ports, connected, got some metadata and identified the devices and their firmware - searched the https://t.co/Hjs8FuH16W
-
@shazow Very good questions imo experienced devs have a real advantage but only if they rapidly progress through their grief cycle and adapt, now and onwards. Categorically rejecting or ignoring the new layer would be a mistake.
-
@nearcyan “It still doesnt work pls fix. dont make any mistakes. ultrathink very hard”
-
@bcherny I have similar experiences. You point the thing around and it shoots pellets or sometimes even misfires and then once in a while when you hold it just right a powerful beam of laser erupts and melts your problem.
-
@ibab It’s very good. People who aren’t keeping up even over the last 30 days already have a deprecated world view on this topic.
-
I've never felt this much behind as a programmer. The profession is being dramatically refactored as the bits contributed by the programmer are increasingly sparse and between. I have a sense that I could be 10X more powerful if I just properly string together what has become
-
@bcherny ❤️
-
@bcherny Ran into token file context limits this morning. It's possible to override them for the MCP tool setting MAX_MCP_OUTPUT_TOKENS but I don't believe an equivalent exists for the Read tool. https://t.co/PsuDhRz1TW
-
@ahall_research fun! I should mention that my original council was different LLMs but I think it should work well (possibly even better?) even with one LLM but different prompts. It would require a bit more creativity around what those prompts are, what their problem solving tendencies are, etc.
-
@nearcyan The top comment on all of these is usually “ai” with 3000 likes
-
https://t.co/Lb6T42n5jl
-
I love the expression “food for thought” as a concrete, mysterious cognitive capability humans experience but LLMs have no equivalent for. Definition: “something worth thinking about or considering, like a mental meal that nourishes your mind with ideas, insights, or issues that
-
@knowtrend_ai Great idea
-
@pulkit_mittal_ Hahah! 💯
-
nanoGPT - the first LLM to train and inference in space 🥹. It begins. https://t.co/NYtbw764Bu
-
Quick new post: Auto-grading decade-old Hacker News discussions with hindsight I took all the 930 frontpage Hacker News article+discussion of December 2015 and asked the GPT 5.1 Thinking API to do an in-hindsight analysis to identify the most/least prescient comments. This took https://t.co/Ufexq5xmDX
-
A good chunk of people misunderstood this tweet btw, which is my bad. I am not suggesting people use the old style promoting techniques of “you are an expert swift programmer” or etc. it’s ok.
-
ty to ericsilberstein1 on github for spotting the bug. https://t.co/18o1CiivgN (it's not a big bug and only comes up in the SpellingBee synthetic task evaluation but still).
-
In today's episode of programming horror... In the Python docs of random.seed() def, we're told "If a is an int, it is used directly." [1] But if you seed with 3 or -3, you actually get the exact same rng object, producing the same streams. (TIL). In nanochat I was using the https://t.co/dVlUDnGJsN
-
@ChrSzegedy I could certainly imagine that "nesting" the simulation might be too "effortful" for the model, compute or data density wise. My results with it are not too bad so imo it's at least worth people try / experiment with / think about. For example it might be useful to read multiple
-
@DimitrisPapail There is definitely work going into engineering the "you" simulation - the personality that gets all the rewards in verifiable problems, or all the upvotes from users/judge LLMs, or mimics the responses of SFT, and there is an emergent composite personality from that. My point is
-
Don't think of LLMs as entities but as simulators. For example, when exploring a topic, don't ask: "What do you think about xyz"? There is no "you". Next time try: "What would be a good group of people to explore xyz? What would they say?" The LLM can channel/simulate many
-
@Marswalkerr I love this meme too haha!
-
Happy weekend to those who celebrate https://t.co/lafvNmmOJO
-
@nickcammarata Your recent posts on this remind me of this Arnold gem https://t.co/IMe0VwINCY +100 though. I finally had a chance to install a home gym recently, making it trivial to use daily. Always looking forward to the next exercise high. Slightly miss the social/entropy aspects of gyms.
-
@theJayAlto Edutainment. This one weird trick to consume entertainment and feel good about it.
-
@toThePixel So cool! Distinctly reminded of turtle graphics (a popular way to learn programming)
-
@matejhladky_dev One more comment is that giving this image to an AI and asking about it is not sufficient to show the diff because it's all over the training data by now. You'd have to use a new, very recent image, taken yesterday or something. But it doesn't super matter, even if it didn't work
-
@matejhladky_dev AI has crushed it since this post way beyond expectation. I made the same category of mistake all of AI was making, of thinking we have to discover and write the algorithm. You don't. You pretrain and then finetune a BIG neural network on lots of tasks and it just falls out. lol.
-
@ID_AA_Carmack I've had medium success asking LLMs if a thing exists, it works out of the box for some of the more well-known things (e.g. both GPT 5.1 and Gemini 3 know about this function if you describe the tensor transformation in words). For more esoteric or new libraries (e.g. uv being a
-
A number of people are talking about implications of AI to schools. I spoke about some of my thoughts to a school board earlier, some highlights: 1. You will never be able to detect the use of AI in homework. Full stop. All "detectors" of AI imo don't really work, can be https://t.co/jEtuuW3bGP
-
@_thomasip haha yes it makes mistakes! You have to re-roll a few times until it's right. Sometimes it gets stuck in loops and you have to re-start in a new conversation. Example re-roll: https://t.co/dK3VcuJLDn
-
@anneshu_nag I love that it's called Nano Banana instead of Google Imagine or some other corpo thing
-
@sharkey I first asked it to create a workout plan as a separate text interaction up front - I gave it some of my preferences, goals, etc. and asked for a weekly plan. Then I asked to illustrate the week, and the individual days after.
-
I asked it to create a personalized weekly workout plan, and then posters that I can print on the wall to remind me what exercises to do each day. Tuesday looks more intense because I asked for "more testosterone" :D. (sorry I'll stop posting more nano banana pro stuff now) https://t.co/iNXfUJnVY1
-
@icreatelife Fully neural menugen
-
Imo this is along the lines of how talking to an LLM via text is like typing into a DOS Terminal and "GUI hasn't been invented yet" of some of my earlier posts. The GUI is an intelligent canvas.
-
Gemini Nano Banana Pro can solve exam questions *in* the exam page image. With doodles, diagrams, all that. ChatGPT thinks these solutions are all correct except Se_2P_2 should be "diselenium diphosphide" and a spelling mistake (should be "thiocyanic acid" not "thoicyanic") :O https://t.co/15oUx8FIqJ
-
@MaziyarPanahi @immasiddx I confirmed that it works. Prompt was the starting image and "Fix this math problem and generate updated image in my handwriting"
-
@ridegoodwaves I mean it's just incredible. Not because of this version but the next one. The extrapolation in your head is now trivial.
-
@burkov I don't know if this is right because the models rank all of each other's work fully pairwise (and anonymized) and seem to agree on the ranking.
-
As a fun Saturday vibe code project and following up on this tweet earlier, I hacked up an **llm-council** web app. It looks exactly like ChatGPT except each user query is 1) dispatched to multiple models on your council using OpenRouter, e.g. currently: "openai/gpt-5.1", https://t.co/yljZu0Vr8n https://t.co/gKvRibPLVe
-
@kellerjordan0 Like. Slop is “regretted” attention.
-
@Yuchenj_UW Great reference ty. It still doesn’t get at more of theory of slop. As data lacking of insight. Filler. Something that “type checks” as information but isn’t. Style without substance. As something that, when attended to or trained on, does not aid compression of a world model.
-
@corbtt Random token sequence would by this definition be minimum slop.
-
Has anyone encountered a good definition of “slop”. In a quantitative, measurable sense. My brain has an intuitive “slop index” I can ~reliably estimate, but I’m not sure how to define it. I have some bad ideas that involve the use of LLM miniseries and thinking token budgets.
-
@NickADobos I don’t super love shoggoth represented as a kind of biological monster (animal), it feels wrong/misleading in this sense.
-
@TheVixhal Amusing to see this go viral and interpreted as not clearly a compliment.
-
@TheVixhal your post challenged me. every one of your points is wrong but i had to think about each for a while :)
-
Something I think people continue to have poor intuition for: The space of intelligences is large and animal intelligence (the only kind we've ever known) is only a single point, arising from a very specific kind of optimization that is fundamentally distinct from that of our
-
My most amusing interaction was where the model (I think I was given some earlier version with a stale system prompt) refused to believe me that it is 2025 and kept inventing reasons why I must be trying to trick it or playing some elaborate joke on it. I kept giving it images https://t.co/YwX8DcSUKf
-
I played with Gemini 3 yesterday via early access. Few thoughts - First I usually urge caution with public benchmarks because imo they can be quite possible to game. It comes down to discipline and self-restraint of the team (who is meanwhile strongly incentivized otherwise) to
-
I put up a simple repo I call reader3 (it's my 3rd version...) to illustrate how I read EPUBs with LLMs. Basically get some epub (e.g. Project Gutenberg is great), go chapter by chapter, and with this you can easily copy paste text to your favorite LLM. https://t.co/HoBDxCJhdC https://t.co/AOohcBvfbc
-
@scaling01 I was so curious. Maybe maybe maybe…. 🥁 🤦♂️ https://t.co/UXRf6dshi7
-
@markus_zechner It’s possible. But yes the book publisher industry hasn’t caught up yet. The obvious low hanging fruit basics of LLM integration in Kindle etc should have been here ~2 years ago already.
-
@PeteReese In my experience it helps *a lot*. 50% of it is probably just the “instant reading group on demand” effect, where you just have to manipulate the information in some way instead of just passively attending to it.
-
@proggineer Agree, sometimes that is helpful too, to have an overview of what the whole thing is about first. I just copy paste stuff around to LLM of the day (I cycle), there’s no “tool”.
-
I’m starting to get into a habit of reading everything (blogs, articles, book chapters,…) with LLMs. Usually pass 1 is manual, then pass 2 “explain/summarize”, pass 3 Q&A. I usually end up with a better/deeper understanding than if I moved on. Growing to among top use cases. On
-
Finally had time to read & process this great post. I run into the pattern quite often, it goes: "<something that sounds wrong> is good actually, because <galaxy brain reason>" Galaxy brain reasoning is the best way to justify anything while looking / feeling good about it. https://t.co/2wpdqH0lI2
-
@code_star I heard Gemini 3 answers questions before you ask them. And that it can talk to your cat.
-
Sharing an interesting recent conversation on AI's impact on the economy. AI has been compared to various historical precedents: electricity, industrial revolution, etc., I think the strongest analogy is that of AI as a new computing paradigm (Software 2.0) because both are
-
I am unreasonably excited about self-driving. It will be the first technology in many decades to visibly terraform outdoor physical spaces and way of life. Less parked cars. Less parking lots. Much greater safety for people in and out of cars. Less noise pollution. More space
-
@polynumera When I talk about self-driving I mean the actual future of it. Like really real, like what you'd see in a movie. You walk out on the street and almost all cars are autonomous, shuttling people around.
-
@jpsilvashy From what I can tell the HW4 experience seems *a lot* better right now. My guess is that the team will find ways to distill to HW3 models and lift the performance a lot. It works very well in LLMs at least.
-
@ScottTArcher @chuckcook My 3rd Tesla now. The store person asked me if I need help with the car features and how to use the Autopilot. I told him I think I'm ok :D
-
@Yuchenj_UW They both offer what now intuitively feels to me like "perfect drive" - smooth, confident, just works. There's probably still differences ofc but you have to wait for them, or aggregate them over many cars, etc.
-
I took delivery of a beautiful new shiny HW4 Tesla Model X today, so I immediately took it out for an FSD test drive, a bit like I used to do almost daily for 5 years. Basically... I'm amazed - it drives really, really well, smooth, confident, noticeably better than what I'm used https://t.co/RdaM23kvez
-
@alita2100 @ashleybcha @goakhmad He’s using it in situations where it fundamentally lets you create and do what you could not create before. Not in situations where it’s the cheaper and faster option and most people can’t tell anyway just ship it.
-
@MrFahrenhe844 @goakhmad https://t.co/7NtPHKFdDc
-
@swapnilkk1987 @goakhmad Don’t use Netflix there is very little quality there, it’s brain rot inducing. I use Prime Video and rent excellent movies/shows. I agree with you about the paradox of choice among other annoyances with “modern entertainment” - I am in the process of buying a dvd player and
-
@Jess_Riedel @goakhmad Yeah I think granted, it’s why I call them prompts. Their contribution is to present concrete situations that you can then think about, form opinions about, talk to your friends about, etc.
-
@ashleybcha @goakhmad I should add that 1 even lotr used plenty of CGI overall and just like AI now it is a powerful tool in the box. It just needs discipline, which I think is hard to come by in practice and 2 all of the above is about frontier/professional movies in the cinema and Oscars and etc. I
-
@ashleybcha @goakhmad I think it’s right that you’re grouping CGI with (current-capability etc) AI because they are both fundamentally technologies that trades wins in time/cost against increase in slop. A filmmaker suffers and usually gives into the temptation of gains in time/cost in a slight
-
@goakhmad Movies are great though. Even if you set aside the pure artistic enjoyment (you shouldn’t). Movies are stories, and stories are powerful, primal, moving, motivating. They are prompts to you to consider dilemmas and scenarios, to build your world model and compass. My rec is to go
-
@drfeifei @QEPrize I love that the prize recognizes foundational work in all 3 pillars of AI (data, compute, algorithms), congratulations Fei-Fei!!
-
@soumithchintala Great run Soumith! Of all the deep learning framework transitions I've been through where I re-wrote ~all of my code (matlab -> caffe -> numpy -> torch -> pytorch), the PyTorch one was most pleasant and now significantly longest lasting. It hit a jackpot of the time in the
-
@GoodFaithOnly @binarybits When I say AGI I mean a "feature complete" build of a remote worker with a lot of 9s (think: a bit like current Autopilot FSD, maybe plus a few more iterations). This is the original definition I've stuck to forever. Separate from the diffusion /implementation of it across https://t.co/LS7sRZD0JC
-
@MarFot78 @zzlccc I think if you zoomed into the paper too you’d find it just as if not more interesting.
-
@zzlccc This and Sydney Sweeney was my entire timeline so I couldn’t not notice :). It’s a very interesting find and good reading, I have a local branch playing with for nanochat.
-
Beautiful technical debugging detective longread that starts with a suspicious loss curve and ends all the way in the Objective-C++ depths of PyTorch MPS backend of addcmul_ that silently fails on non-contiguous output tensors. I wonder how long before an LLM can do all of this. https://t.co/qt0CRdRL8C
-
@LaMancha574 I mentioned in todos I don’t augment the position of this request as part of larger multiturn conversations atm so it’s most robust when you /clear and ask in new chat and mimic that.
-
@helloimkongv @r_chirra Damn
-
@r_chirra It’s my model I can do whatever I want. So now I am king. You can be too for $100.
-
@admn_is_traitor I am busy this weekend but maybe next
-
@samsja19 I find it surprisingly addicting. I have so many ideas for personality, style, capabilities. I want nanochat to be a free AI, like Dobby the free elf, not some "assistant".
-
Last night I taught nanochat d32 how to count 'r' in strawberry (or similar variations). I thought this would be a good/fun example of how to add capabilities to nanochat and I wrote up a full guide here: https://t.co/F7pmTCcyeR This is done via a new synthetic task https://t.co/MUuOIHvm4t
-
@sirbayes @natashajaques @_AndrewZhao Ty, I just meant it hasn’t (to my knowledge) created a whole new stage in the production stack of a frontier LLM, or had a contribution on that level.
-
@nathanrs so cool! really gives you the feeling of text as just this "boiling" texture (haha "text"ure), starting at the boiling point and annealing to a simmer.
-
@iamgrigorev @sarna_dev @huggingface atm nanochat does not pretokenize, instead tokenization is done live during training in the dataloader, which is ok and gives the CPU something to do while the GPUs are working. So I didn't try too hard to optimize this part.
-
@sarna_dev @huggingface Yep, the use of regex is both a huge dependency and huge bottleneck in the tokenizer. I think it's a beautiful project to try to do this correctly, but I'd need someone who is really familiar with regex to pitch in and also a large test suite to make sure. I'd love to merge such
-
@latkins This code is extremely dangerous. Here, I improved it. https://t.co/CmyYjv4pHA
-
@gazorp5 @akyurekekin That’s almost separate… You certainly want to learn recoveries in the event of errors at test time. You just don’t want to also learn to actively create errors.
-
@thawani_avijit Haha. I am afraid people interpreted my “delete tokenizer” as “use bytes directly without BPE”, the issue is you *still* need bytes encoding arbitrariness even for that! Pixels is the only way. Just like humans. It is written. If GPT-10 uses utf8 at the input I will eat a shoe.
-
@akyurekekin Fair but it’s still actively learning to make those errors (just so it can recover from them later), which imo is still a bit weird. In an ideal world you wouldn’t, possibly process supervision is one way to get there even within RL framework. Def agree on “recovery learning”.
-
@ThatMcDermott @nearcyan Wow 👏
-
@nearcyan “Bed is stuck in inclined position because of AWS outage” I really thought this must have been a joke
-
@swyx @YorkieBuilds @Grad62304977 @tejalpatwardhan AGI definition is not about GDPVal puzzles scores, it’s about when you’re fired because GPT-n is better, cheaper, takes no equity and works weekends. I like GDPVal, but I dislike the fixation on puzzle solving as some kind of alleged endpoint.
-
@sujoyrc @Grad62304977 Yes so there's one more dimension here which is more about diffusion. The original definition is that such a system *exists*, not that it is fully deployed across society. The diffusion of such a system will still take even more time in my mind (e.g. basic technological diffusion
-
@timshi_ai Personally i think the hallucinations are kind of hilarious and I haven't had this much fun talking to an LLM for a long time. But yes we could RLHF it out of the model.
-
@YorkieBuilds @Grad62304977 Original OpenAI definition. AGI = an automated system capable of doing any economically valuable work that a human can do. (Like humans they might need some additional training for a job, that's fine). And I grant the commonly given "digital concession", putting away physical
-
@Grad62304977 Some background is that early OpenAI (2015) we all took a poll on our AGI timelines. I voted 20 years then (very long compared to others), so now 1 decade later it's interesting that it still feels on trend and I'm sticking with it. I completely understand that it feels arbitrary
-
See this new Discussion for more technical detail Guide: infusing identity to your nanochat https://t.co/jWU5PTt4h9
-
nanochat now has a primordial identity and can talk a bit about itself and its capabilities (e.g. it knows it's nanochat d32 that cost $800, that it was built by me, that it can't speak languages other than English too well and why, etc.). This kind of customization is all done https://t.co/WzGjTvQSXu
-
@r_chirra I fixed it :) deployed live now. This was done by doing a round of synthetic data generation to collect a 1000 multi-turn conversations (given a bunch of information including the readme of the nanochat project), and then mixing that into midtraining and SFT. fun! https://t.co/wDtU3GcMZJ
-
@r_chirra It also has a brain smaller than a honey bee so there’s that. LLMs have no clue of their identity or of their capabilities by default and out of the box. But I’ll teach it! Could be fun.
-
@yoavgo There is nothing in principle preventing it, except that text is usually simply trained autoregressively for efficiency. I could imagine a midtraining stage where you fine tune with bidirectional attention for conditioning information, eg User messages (tokens where you won’t be
-
I quite like the new DeepSeek-OCR paper. It's a good OCR model (maybe a bit worse than dots), and yes data collection etc., but anyway it doesn't matter. The more interesting part for me (esp as a computer vision at heart who is temporarily masquerading as a natural language https://t.co/AxRXBdoO0F
-
Nice, short post illustrating how simple text (discrete) diffusion can be. Diffusion (i.e. parallel, iterated denoising, top) is the pervasive generative paradigm in image/video, but autoregression (i.e. go left to right bottom) is the dominant paradigm in text. For audio I've https://t.co/8i5Jgj7ogL https://t.co/l8X0nS1ySD
-
@tszzl @antichristus It’s been a decade but yes I believe I hallucinated the term in my 2015 post on unreasonable effectiveness of RNNs. I later became aware that Geoff Hinton used “confabulate”, which is often (but I think not always) a better analogue in human psychology. It’s a bit too specific, https://t.co/LkjhWcB2o8
-
@jsuarez I very much hope you continue working on RL! I think it's a misunderstanding that I am suggesting we need some kind of a replacement for RL. That's not accurate and I tried to clear it but did so poorly - they layer. Layer 1 was base model autocomplete. Layer 2 was instruct
-
@elonmusk I’d much rather use and collaborate with Grok 5 than compete against it. Though quite similar to chess, and “in the limit” (speaking of physics!), my value add probably trends to ~zero.
-
My pleasure to come on Dwarkesh last week, I thought the questions and conversation were really good. I re-watched the pod just now too. First of all, yes I know, and I'm sorry that I speak so fast :). It's to my detriment because sometimes my speaking thread out-executes my https://t.co/bnPSrY74px
-
@iamgrigorev Thank you! I'm quite happy with the core_eval.py rewrite. I wanted to evaluate my base model with the DCLM "core score" as described in their paper, but what felt like it should surely be a simple thing of ~300 lines of code actually required me to pip install and depend on a
-
@timzaman @dwarkesh_sp Hacker hoodie Rare Cloth Armor: 0 Intellect +10 Stamina +10 Active: put hoodie over head. Enters state of intense focus. +20% APM. +10% intellect. Ignores hunger. If disturbed, take 100 sanity damage.
-
@kumbhani_smit I think when I said TV personally I just meant movies/tvshows which I realize wasn’t super clear. Things you used cassette or DVD for as the 90s baseline. I basically never watched TV as in news or cable.
-
DVD player is superior technology.
-
Deliberately*
-
There is a movement I found on Instagram where people delivery choose to live in 90s, refusing all technology after 2000. Like an intermediate form of the Amish.
-
TV in the 90s: you turn it on, you watch. TV 2025: - turn on, wait for it to load - popup: TV wants to update, 1.5GB. No. - scroll sideways, find prime video app or etc - popup: now app wants to update, 500MB. No!! - App launching... App loading… - select account screen - 🫠
-
nanochat d32, i.e. the depth 32 version that I specced for $1000, up from $100 has finished training after ~33 hours, and looks good. All the metrics go up quite a bit across pretraining, SFT and RL. CORE score of 0.31 is now well above GPT-2 at ~0.26. GSM8K went ~8% -> ~20%, https://t.co/8UcpefaSJN
-
@vikramsingh0110 Ty MINIX is very inspiring and was exactly on my mind as well (as I've made the LLM <-> OS analogy often). Goals
-
@__tinygrad__ @sytelus I count and report the lines in uv.lock, which is my attempt at a simple 80:20 proxy for this, as it includes all the packages recursively. Open to suggestions! I'd love to measure things like "cognitive complexity" (there was a great blog post on it a few months back that I
-
@younghope11 yeah i was thinking about it, good idea.
-
@simonw Good idea, I'm running/tuning these tiers now. I'll bunch up some of the low-hanging fruit here over the next few days and make it available (I spent most of my energy going into this v0.1 on the overall harness).
-
@Tim_Dettmers Thank you! Notably I didn't yet include model quantization for inference. I have questions :)
-
@ClementDelangue @huggingface Ty! huggingface work/infra/datasets are critical to projects like nanochat - to be accurate the source code of nanochat (e.g. at the $100 tier) is ~8KB of Python and ~30GB of fineweb/smoltalk.
-
@eliebakouch @rasbt Very early on in the project I did a small run with/without QK norm and found that it helped. Same for the embedding weight sharing. I'll retry! I'm not tied to any details of the model and they weren't chosen any more carefully than a single run, I spent most of the time just
-
@singh_ilepton LLM101n is currently experiencing a lot of scope creep 😅
-
@rasbt Basically Llama-like, a bit simpler, some influences from modded-nanoGPT. Tried to find a solid baseline for this scale: - dense transformer - rotary embeddings (and no positional embeddings) - QK norm - untied weights for embedding and unembedding - norm after token embedding -
-
@rcmisk Good question ty, I think this is not a good repo for that. You should think of micro models maybe more as very young children (kindergarten etc.), they just don't have the raw intelligence of their larger cousins. If you finetune/train it on your own data you'll probably get
-
@zenitsu_aprntc Good question, it's basically entirely hand-written (with tab autocomplete). I tried to use claude/codex agents a few times but they just didn't work well enough at all and net unhelpful, possibly the repo is too far off the data distribution.
-
And an example of some of the summary metrics produced by the $100 speedrun in the report card to start. The current code base is a bit over 8000 lines, but I tried to keep them clean and well-commented. Now comes the fun part - of tuning and hillclimbing. https://t.co/37MJdNcwtd
-
GitHub repo: https://t.co/YW94b9YNVa A lot more detailed and technical walkthrough: https://t.co/NtXhKkRCUU Example conversation with the $100, 4-hour nanochat in the WebUI. It's... entertaining :) Larger models (e.g. a 12-hour depth 26 or a 24-hour depth 30) quickly get more https://t.co/IcW047vQz7
-
Excited to release new repo: nanochat! (it's among the most unhinged I've written). Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, https://t.co/LLhbLCoZFt
-
@nikitabier Going to an Internet cafe, saving the toronto dot ca website to floppy disk to take it home and show it to my parents as research just before we moved there. Lol
-
@doodlestein OPEN THE POD BAY DOORS HAL The number of AI pioneers anticipating this as the state of the art AI of 2025 must surely have been exactly zero.
-
POV: Your LLM agent is dividing a by b https://t.co/PBYz3vctjq
-
I don't know what labs are doing to these poor LLMs during RL but they are mortally terrified of exceptions, in any infinitesimally likely case. Exceptions are a normal part of life and healthy dev process. Sign my LLM welfare petition for improved rewards in cases of exceptions.
-
@swyx @staysaasy I don’t love it. You’re either engineering or you’re vibing. They are opposites of some spectrum.
-
@eigenrobot World of Warcraft Classic grinding mobs, simple questing is mine. Repetitive skill rotation with just enough variety to keep fun/engaging but easy. A lot of *wrong* answers in the replies here, games that nowhere near mindless enough eg Factorio.
-
@xiaosun86 AI would be an added layer between you and reality. In case the volume is too great (I doubt it), I'd much rather subsample/skim before I tried to add AI.
-
@Addiedesignco @seflless email is too formal. the whole idea here is specifically to create a reality escape valve for a formal process.
-
@daniel_mac8 @OfficialLoganK yep, was in my mind as one example.
-
@seflless Email doesn't work. It has to be DM. I don't know why, it just feels right.
-
Every company needs a DM POC - someone high up who you can just DM the most obvious things and who shortcuts the PM hierarchy.
-
@manca1 Good point! Maybe one prominent example for me is writing all kinds of "one off" stuff for dev work. E.g. I'll take 2000 lines of ephemeral code just to help me find a single bug, but once I find & fix it, I can just delete all of that code. Or a lot of vis low stakes code, etc.
-
@timmolendijk yeah i see a few people say something similar in the replys, i could probably see it. it could basically be that a lot of industry software engineering work is essentially in the "boilerplate" category.
-
@0xKalos my reaction exactly fwiw
-
@jdchawla29 Yeah one more person pointed out already. I'd say it falls under "agent" because an LLM is still giving you a large chunk of code that you're slotting in.
-
For your professional programming do you use mostly:
-
@gregisenberg I’ve been doing this for a while, you don’t “use ChatGPT” you “ask chat”. Much cleaner.
-
@alexisxrivas Btw people should check out @alexisxrivas / @coverbuild, which is really cool and I've followed for a while. I think I'm just developing an allergy to some disease in culture that I'm working through. Separately from functional, scalable, affordable... we have to build CATHEDRALS
-
@alexisxrivas Dream kitchen, without the contemporary minimalism psyop. But also a big fan of functional and affordable. https://t.co/5vAVtmiOIh
-
Hah judging by mentions overnight people seem to find the ghost analogy provocative. I swear I don't wake up just trying to come with new memes but to elaborate briefly why I thought it was a fun comparison: 1) It captures the idea that LLMs are purely digital artifacts that https://t.co/w71cmFbsSN
-
@rasbt yep 💯! Using closed model API should feel quite unsettling when there is no recourse.
-
Tinker is cool. If you're a researcher/developer, tinker dramatically simplifies LLM post-training. You retain 90% of algorithmic creative control (usually related to data, loss function, the algorithm) while tinker handles the hard parts that you usually want to touch much less https://t.co/RR1qmQ7Pp6
-
@decruz I'd be afraid that NotebookLM is wired up to do RAG. It's not a lot of tokens. I want them in the context window. All of them and for sure.
-
@brickroad7 Think Casper! Childhood favorite. https://t.co/mL0Ioo7sSM
-
@chrishayduk "Don’t be difficult. I mean this is obvious." 😂 Sutton is right ofc. The analogue in LLM land to what humans do is something along the lines of: Given this math problem AND human example solution in the context, solve the problem. Reward of 1 if correct. It's not SFT, it's RL.
-
Something I am experimenting with. I copy pasted: 1) the full podcast transcript 2) the bitter lesson blog post 3) my full post above To ChatGPT. The interesting part is you can fork the conversation context to ask any questions and take it in whatever direction with chat:
-
Finally had a chance to listen through this pod with Sutton, which was interesting and amusing. As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea https://t.co/EQ4snNmiTM
-
@LiamFedus @ekindogus @periodiclabs We're not going to reason our way to Dyson spheres. Excited for you and all the best to the team!
-
"AI isn't replacing radiologists" good article Expectation: rapid progress in image recognition AI will delete radiology jobs (e.g. as famously predicted by Geoff Hinton now almost a decade ago). Reality: radiology is doing great and is growing. There are a lot of imo naive https://t.co/hHISYwepvl
-
@simonw I was a bit surprised it is less than case than I expected. Code is KING. It’s the primary means of processing digital information - long term I can’t imagine a more important domain for the AGI pilled. And it is highly valuable in the interim too - big TAM @ high salaries.
-
@ProperPrompter Well for a properly omniscient entity of all present and past… 🤷♂️
-
Anytime someone takes a picture/video that I happen to be in the background of I like to wave at the AGI that sees me 30 years from now
-
@LocalBateman @teknium Video goes hard though
-
@burkov The code was written at layers 22-30 and is stored in the value activations you just can’t read it. I think you owe the LLM an apology.
-
@dwarkesh_sp Most people misunderstand books as data for pertaining when it’s more a set of prompts for synthetic data generation.
-
from this era https://t.co/1pIFiqes4Y
-
reminded of this paragraph from gsm8k paper, 2021 :) https://t.co/lmubBMBFIb https://t.co/3q0EGnzH47
-
@RajaPatnaik Naively I'd rather it be smaller than thinner but I'll have to see/hold it.
-
@jasonth0 Actually jason, every iPhone is the best iPhone they have made yet. And this is something only Apple can do.
-
@AdityaJiRathore core memory watching that 18 years ago (omg...)
-
@devaaaaang The phones have been getting ever bigger, heavier, more full-featured. There are dozens of us that want the opposite. Dozens!
-
@trevorloucks iirc when I looked into it a few years ago it was the worst selling model by far. But the few that loved it loved it deeply. This has not been accounted for!
-
@imprivi Haha. I think in some way I still haven't gotten over the miracle.
-
Bit silly but I still watch the Apple event livestream for new iPhones, every year since the first one in 2007. It doesn't make sense but it's ok. Livestream today at 10am (in 1.5 hours). This year, crossing my fingers again for an iPhone mini that I know won't come. rip.
Appearances 2 appearances
every confirmed appearance, newest first
Themes 5 themes
last 90 days
-
ai models
post“@threejs I was curious and gave your feedback to Opus. TIL! https://t.co/t5HvyZsBlh”
post“@MattBeton so fun! :) at some point i wonder if ngram (tables) or even something like decision trees start to give superior log probs, and at much smaller program lengths overall (sum of program + weights). i.e. what is the best val loss model overall, for 25KB of user space. fun q!”
post“@rainisto @DavidmComfort agree!! i quite like the idea of procedural code for storyboarding and control, and then video to video models for texturing and looksmaxxing.”
post“More on the pelican on the bicycle test from @simonw: https://t.co/OXmtODyTKj I uploaded the source here so it's playable in the browser, forkable etc. https://t.co/w3Nctc888d Look out for GTA Hobbiton dropping before GTA VI :)”
post“@cunkpyber Eleven Labs for the audio. LLMs can easily use the APIs (here I did that part manually because I felt picky about the voice).”
post“We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for https://t.co/ybIpXhYSsj”
-
memory
post“@MattBeton so fun! :) at some point i wonder if ngram (tables) or even something like decision trees start to give superior log probs, and at much smaller program lengths overall (sum of program + weights). i.e. what is the best val loss model overall, for 25KB of user space. fun q!”
-
llm
post“More on the pelican on the bicycle test from @simonw: https://t.co/OXmtODyTKj I uploaded the source here so it's playable in the browser, forkable etc. https://t.co/w3Nctc888d Look out for GTA Hobbiton dropping before GTA VI :)”
post“We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for https://t.co/ybIpXhYSsj”
-
agents
post“@ChrisJMcCormick The extrapolation is that your spec is something like microgpt (scalar valued python with for loops etc), everything else is just a matter of compilation and PyTorch etc is kind of a crappy IR”
post“@ChrisJMcCormick Yeah, increasingly a lot more appealing to tear down these abstractions now that agents can do a lot of the math and drudgery and verification. A lot of the abstractions were built for a world with constraints of finite intelligence and attention in the industry.”
post“@jack_w_rae It’s going to feel so weird that historically you couldn’t just talk to your computer like you’d talk to any other person”
post“@MattBeton so fun! :) at some point i wonder if ngram (tables) or even something like decision trees start to give superior log probs, and at much smaller program lengths overall (sum of program + weights). i.e. what is the best val loss model overall, for 25KB of user space. fun q!”
-
models
post“@jack_w_rae It’s going to feel so weird that historically you couldn’t just talk to your computer like you’d talk to any other person”
post“@rainisto @DavidmComfort agree!! i quite like the idea of procedural code for storyboarding and control, and then video to video models for texturing and looksmaxxing.”