Patel says the model was released before pre-training was finished and needs more RL.
“for everything new, more, more continued pre training because the spud, they kind of like didn't finish the pre training and just released it. So finished the pre training, do more RL drop model.”
Patel explains long context windows fail because pre-training uses short context and lacks data for longer ranges.
“But it's like, what data do I have? That's why my two fifty ks to one mil context is trash anyways, because there's no data on this stuff.”