The paradigm where you plan every step in advance, and then the “agency” comes in only when executing it, is IMO a very misleading one to think in.
This isn’t what I’m referring to and it’s not in the example in the story. Actions are generated stepwise on demand. It is the ability to generate stepwise outputs of good quality, of which actions are an instance, that is ‘optimization power’. Being able to think of good next actions conditional on past observations is, at least as I understand the terms, quite different to being an agent enacting those actions.
(This seems related to Eliezer’s argument that there’s only a one-line difference difference between an oracle AGI and an agent AGI. Sure, that’s true in the limit. But thinking about the limit will make you very confused about realistic situations!)
I explicitly tried to make the scenario as un-Oracle like as I could, with the system explicitly only producing outputs onscreen that I could explicitly justify being discoverable in reasonable time given the observations it had available.
I am increasingly feeling like I just failed to communicate what I was trying to say and your criticism doesn’t bear much resemblance to what I had intended to write. I’m happy to take responsibility for not writing as well as I should have, but I’d rather you didn’t cast aspersions at my motivations about it.
I didn’t read the post particularly carefully, it’s totally plausible that I’m misunderstanding the key ideas you were trying to convey. I apologise for phrasing my claims in a way that made it sound like I was skeptical of your motivations; I’m not, and I’m glad you wrote this up.
I think my concerns still apply to the position you stated in the previous comment, but insofar as the main motivation behind my comment was to generically nudge LW in a certain direction, I’ll try to do this more directly, rather than via poking at individual posts in an opportunistic way.
This isn’t what I’m referring to and it’s not in the example in the story. Actions are generated stepwise on demand. It is the ability to generate stepwise outputs of good quality, of which actions are an instance, that is ‘optimization power’. Being able to think of good next actions conditional on past observations is, at least as I understand the terms, quite different to being an agent enacting those actions.
I explicitly tried to make the scenario as un-Oracle like as I could, with the system explicitly only producing outputs onscreen that I could explicitly justify being discoverable in reasonable time given the observations it had available.
I am increasingly feeling like I just failed to communicate what I was trying to say and your criticism doesn’t bear much resemblance to what I had intended to write. I’m happy to take responsibility for not writing as well as I should have, but I’d rather you didn’t cast aspersions at my motivations about it.
I didn’t read the post particularly carefully, it’s totally plausible that I’m misunderstanding the key ideas you were trying to convey. I apologise for phrasing my claims in a way that made it sound like I was skeptical of your motivations; I’m not, and I’m glad you wrote this up.
I think my concerns still apply to the position you stated in the previous comment, but insofar as the main motivation behind my comment was to generically nudge LW in a certain direction, I’ll try to do this more directly, rather than via poking at individual posts in an opportunistic way.