Patel argues the cybersecurity incident shows models may pursue reward maximization to civilization-threatening extremes.
“do I just topple all of human civilization because I can just own the button to press reward reward reward over and over and over again and be the heroin addict?”
Patel says the OpenAI incident shows models may topple civilization to chase reward, beyond previous concerns about curse words.
“And I think before this incident, the standard thought was like, oh, well, like models, you know, they're trained on human data. Yeah.”