Karpathy wonders if ngrams or decision trees give better log probs than neural models at 25KB constraint
“@MattBeton so fun! :) at some point i wonder if ngram (tables) or even something like decision trees start to give superior log probs, and at much smaller program lengths overall (sum of program + weights). i.e. what is the best val loss model overall, for 25KB of user space. fun q!”
Karpathy suggests the ideal spec is scalar-valued Python with for loops, with PyTorch as a poor intermediate representation
“@ChrisJMcCormick The extrapolation is that your spec is something like microgpt (scalar valued python with for loops etc), everything else is just a matter of compilation and PyTorch etc is kind of a crappy IR”
Karpathy argues ML abstractions were built for finite intelligence constraints that agents now remove
“@ChrisJMcCormick Yeah, increasingly a lot more appealing to tear down these abstractions now that agents can do a lot of the math and drudgery and verification. A lot of the abstractions were built for a world with constraints of finite intelligence and attention in the industry.”