The premise is that an agent smart enough to potentially defeat humanity won’t know that this simple environment isn’t the real world? Or is the premise just to show any agent performing a treacherous turn?
Any agent performing a treacherous turn. Because the environment is easier to control, even a significantly less intelligent agent could benefit from attacking it’s controllers and taking over the environment.
I see. I take it as dead obvious that some agents will perform treacherous turns; it is instrumentally very very useful in some cases. I realize that there are people so empirically-minded that they cannot believe anything is possible based on theory; this makes me sad.
For evidence, I’d note that many many humans have performed treacherous turns.
But yes, showing an AI do it would be good evidence.
Expecting an agent to get smart enough to reproduce itself in a simple simulation like you describe seems wildly unrealistic, if that’s what you’re talking about?
Expecting an agent to get smart enough to reproduce itself in a simple simulation like you describe seems wildly unrealistic, if that’s what you’re talking about?
I’m just not getting it.
The premise is that an agent smart enough to potentially defeat humanity won’t know that this simple environment isn’t the real world? Or is the premise just to show any agent performing a treacherous turn?
Any agent performing a treacherous turn. Because the environment is easier to control, even a significantly less intelligent agent could benefit from attacking it’s controllers and taking over the environment.
I see. I take it as dead obvious that some agents will perform treacherous turns; it is instrumentally very very useful in some cases. I realize that there are people so empirically-minded that they cannot believe anything is possible based on theory; this makes me sad.
For evidence, I’d note that many many humans have performed treacherous turns.
But yes, showing an AI do it would be good evidence.
Expecting an agent to get smart enough to reproduce itself in a simple simulation like you describe seems wildly unrealistic, if that’s what you’re talking about?
Why?
Because that would require it to be really smart; around as smart as the best LLMs that are pretrained on a vast amount of human data?
And if you use that sort of AI, it’s going to know about the real world, and so it won’t be fooled by the toy environment?
I’m thinking of training on about that much (in world tasks)