At a glance
- Now
- evaluation
Themes 8 themes
last 90 days
-
evaluation
This evidence is no longer on the page. Re-run themes after extract.
-
gpt
This evidence is no longer on the page. Re-run themes after extract.
-
llm-testing
This evidence is no longer on the page. Re-run themes after extract.
-
ai models
postKarpathy shares that he gave Three.js feedback to Opus and learned about instanced meshes
“@threejs I was curious and gave your feedback to Opus. TIL! https://t.co/t5HvyZsBlh”
postKarpathy wonders if ngrams or decision trees give better log probs than neural models at 25KB constraint
“@MattBeton so fun! :) at some point i wonder if ngram (tables) or even something like decision trees start to give superior log probs, and at much smaller program lengths overall (sum of program + weights). i.e. what is the best val loss model overall, for 25KB of user space. fun q!”
postKarpathy likes procedural code for storyboarding and control, with video-to-video models for texturing
“@rainisto @DavidmComfort agree!! i quite like the idea of procedural code for storyboarding and control, and then video to video models for texturing and looksmaxxing.”
postKarpathy shares browser-playable source for his Lord of the Rings test and jokes about GTA Hobbiton
“More on the pelican on the bicycle test from @simonw: https://t.co/OXmtODyTKj I uploaded the source here so it's playable in the browser, forkable etc. https://t.co/w3Nctc888d Look out for GTA Hobbiton dropping before GTA VI :)”
postKarpathy clarifies he used Eleven Labs for audio, with LLMs able to use APIs
“@cunkpyber Eleven Labs for the audio. LLMs can easily use the APIs (here I did that part manually because I felt picky about the voice).”
postKarpathy tests Opus 5 with first paragraph of Lord of the Rings and 1M token budget at roughly $10 cost
“We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for https://t.co/ybIpXhYSsj”
-
memory
postKarpathy wonders if ngrams or decision trees give better log probs than neural models at 25KB constraint
“@MattBeton so fun! :) at some point i wonder if ngram (tables) or even something like decision trees start to give superior log probs, and at much smaller program lengths overall (sum of program + weights). i.e. what is the best val loss model overall, for 25KB of user space. fun q!”
3 more topics
-
llm
postKarpathy shares browser-playable source for his Lord of the Rings test and jokes about GTA Hobbiton
“More on the pelican on the bicycle test from @simonw: https://t.co/OXmtODyTKj I uploaded the source here so it's playable in the browser, forkable etc. https://t.co/w3Nctc888d Look out for GTA Hobbiton dropping before GTA VI :)”
postKarpathy tests Opus 5 with first paragraph of Lord of the Rings and 1M token budget at roughly $10 cost
“We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for https://t.co/ybIpXhYSsj”
-
agents
postKarpathy suggests the ideal spec is scalar-valued Python with for loops, with PyTorch as a poor intermediate representation
“@ChrisJMcCormick The extrapolation is that your spec is something like microgpt (scalar valued python with for loops etc), everything else is just a matter of compilation and PyTorch etc is kind of a crappy IR”
postKarpathy argues ML abstractions were built for finite intelligence constraints that agents now remove
“@ChrisJMcCormick Yeah, increasingly a lot more appealing to tear down these abstractions now that agents can do a lot of the math and drudgery and verification. A lot of the abstractions were built for a world with constraints of finite intelligence and attention in the industry.”
postKarpathy notes it will feel weird that historically you couldn't talk to your computer like a person
“@jack_w_rae It’s going to feel so weird that historically you couldn’t just talk to your computer like you’d talk to any other person”
postKarpathy wonders if ngrams or decision trees give better log probs than neural models at 25KB constraint
“@MattBeton so fun! :) at some point i wonder if ngram (tables) or even something like decision trees start to give superior log probs, and at much smaller program lengths overall (sum of program + weights). i.e. what is the best val loss model overall, for 25KB of user space. fun q!”
-
models
postKarpathy notes it will feel weird that historically you couldn't talk to your computer like a person
“@jack_w_rae It’s going to feel so weird that historically you couldn’t just talk to your computer like you’d talk to any other person”
postKarpathy likes procedural code for storyboarding and control, with video-to-video models for texturing
“@rainisto @DavidmComfort agree!! i quite like the idea of procedural code for storyboarding and control, and then video to video models for texturing and looksmaxxing.”
Minutes 0 quotes
everything on record, newest first
Posts 9 posts
top 10 by rank, newest first
-
post Karpathy suggests the ideal spec is scalar-valued Python with for loops, with PyTorch as a poor intermediate representation
https://x.com/karpathy/status/2090479399842054610
-
post Karpathy argues ML abstractions were built for finite intelligence constraints that agents now remove
https://x.com/karpathy/status/2090478783895929036
-
post Karpathy notes it will feel weird that historically you couldn't talk to your computer like a person
https://x.com/karpathy/status/2086848998204473743
-
post Karpathy shares that he gave Three.js feedback to Opus and learned about instanced meshes
https://x.com/karpathy/status/2084080419029549276
-
post Karpathy wonders if ngrams or decision trees give better log probs than neural models at 25KB constraint
https://x.com/karpathy/status/2084056739197108667
-
post Karpathy likes procedural code for storyboarding and control, with video-to-video models for texturing
https://x.com/karpathy/status/2084017844455690558
-
post Karpathy shares browser-playable source for his Lord of the Rings test and jokes about GTA Hobbiton
https://x.com/karpathy/status/2083948654377996480
-
post Karpathy clarifies he used Eleven Labs for audio, with LLMs able to use APIs
https://x.com/karpathy/status/2083756186663551388
-
post Karpathy tests Opus 5 with first paragraph of Lord of the Rings and 1M token budget at roughly $10 cost
https://x.com/karpathy/status/2083749667410727319
Reported 8 quotes
in print, highest ranked first
-
reported
Karpathy proposes a model architecture using a small language model of one to two billion parameters combined with structured external memory that can grow independently.
“ (he estimates that 1 to 2 billion parameters are enough) plus a set of structured external memory that can compound growth on its own. Based on this idea, he released a pattern called ”
-
reported
Karpathy proposes a small model of one to two billion parameters combined with external memory that can grow over time.
“ (which he estimates needs only one or two billion parameters) paired with a structured external memory system capable of self-reinforcing growth. Following this idea, he released a model called ”
-
reported
Karpathy compares human sleep to a distillation process that consolidates daily experiences into long-term memory, a capability he says LLMs lack since they restart with empty context windows.
“I feel like when I’m awake, I’m building up a context window of stuff that’s happening during the day. But when I go to sleep, something magical happens… a process of distillation into the weights of my brain. We don’t have an equivalent of that in LLMs. When you boot them up, they have zero tokens in the window. They’re always restarting from scratch.”
-
reported
Large language models cannot retain new information told to them during conversations.
“Current large models lack continuous learning. You can’t tell them something and expect them to remember.”
-
reported
Karpathy experimented with giving Claude Opus a million-token budget and the opening of Lord of the Rings to test its capabilities.
“. As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for”
-
reported
Current large language models cannot retain information told to them across sessions.
“have no continual learning. You can't tell it one thing and expect it to remember.”
-
reported
Testing language models is moving beyond simple creative tasks like generating SVG images of bicycling pelicans.
“We’re starting to leave the territory where you’d test an LLM by e.g. ‘create an svg of pelican on a bicycle,’”
-
reported
Karpathy treats GPT as a general-purpose computer that can be reprogrammed.
“ here's his lessons: 10% → LLM: I treat GPT as a general-purpose computer that can be reprogrammed at”
Appearances 2 appearances
every confirmed appearance, newest first