A surprisingly large fraction of people I talk to try to convince me that some kind of irrationality is good actually, and my overly strong assumptions about rationality are causes me to expect AI doom. I fairly sure this is false and I’m making relatively weak assumptions about what it means to be rational. One assumption I am happy to make is that an agent will try to avoid shooting itself in the foot (by its own lights). What sort of actions count as “shooting itself in the foot” depends on what the goals are about, and what the environment is like, and often while explaining this I reference this post.
Having a simple example of an extremely obvious example of coherence is great.
I meant to write a longer review but have run out of time. I’ll try to add it later.
A surprisingly large fraction of people I talk to try to convince me that some kind of irrationality is good actually, and my overly strong assumptions about rationality are causes me to expect AI doom. I fairly sure this is false and I’m making relatively weak assumptions about what it means to be rational. One assumption I am happy to make is that an agent will try to avoid shooting itself in the foot (by its own lights). What sort of actions count as “shooting itself in the foot” depends on what the goals are about, and what the environment is like, and often while explaining this I reference this post.
Having a simple example of an extremely obvious example of coherence is great.
I meant to write a longer review but have run out of time. I’ll try to add it later.