Karpathy suggests the ideal spec is scalar-valued Python with for loops, with PyTorch as a poor intermediate representation
“@ChrisJMcCormick The extrapolation is that your spec is something like microgpt (scalar valued python with for loops etc), everything else is just a matter of compilation and PyTorch etc is kind of a crappy IR”
Karpathy argues ML abstractions were built for finite intelligence constraints that agents now remove
“@ChrisJMcCormick Yeah, increasingly a lot more appealing to tear down these abstractions now that agents can do a lot of the math and drudgery and verification. A lot of the abstractions were built for a world with constraints of finite intelligence and attention in the industry.”