Other MIRI staff report that the book helped them fit the whole argument in their head better, or that it made sharp some intuitions they had that were previously vague. So you might get something out of it even if you’ve been around a while.
Can confirm! I’ve followed this stuff for forever, but always felt at the edge of my technical depth when it came to alignment. It wasn’t until I read an early draft of this book a year ago that I felt like I could trace a continuous, solid line from “superintelligence grown by a blind process...” to “...develops weird internal drives we could not have anticipated”. Before, I was like, “We don’t have justifiable confidence that we can make something that reflects our values, especially over the long haul,” and now I’m like, “Oh, you can’t get there from here. Clear as day.”
As for why this spells disaster if anyone builds it, I didn’t need any new lessons, but they are here, and they are chilling—even for someone who was already convinced we were in trouble.
Having played some small part in helping this book come together, I would like to attest to the sheer amount of iteration it has gone through over the last year. Nate and co. have been relentlessly paring and grinding this text ever closer to the kind of accessibility that won’t just help individuals understand why we must act, but will make them feel like their neighbors and political leaders can understand it, too. I think that last part counts for a lot.
The book is also pretty engaging.
The pitch I suggested we share with our friends and allies is this:
If you’ve been waiting for a book that can explain the technical roots of the problem in terms your representative and your mother can both understand, this is the one. This is the grounded, no-nonsense primer on why superintelligence built blindly via gradient descent will predictably develop human-incompatible drives; on why humanity cannot hope to endure if the fatal, invisible threshold is crossed; and on what it will take to survive the coming years and decades.
You convinced me to pre-order it. In particular, these lines: > It wasn’t until I read an early draft of this book a year ago that I felt like I could trace a continuous, solid line from “superintelligence grown by a blind process...” to “...develops weird internal drives we could not have anticipated”. Before, I was like, “We don’t have justifiable confidence that we can make something that reflects our values, especially over the long haul,” and now I’m like, “Oh, you can’t get there from here. Clear as day.”
Can confirm! I’ve followed this stuff for forever, but always felt at the edge of my technical depth when it came to alignment. It wasn’t until I read an early draft of this book a year ago that I felt like I could trace a continuous, solid line from “superintelligence grown by a blind process...” to “...develops weird internal drives we could not have anticipated”. Before, I was like, “We don’t have justifiable confidence that we can make something that reflects our values, especially over the long haul,” and now I’m like, “Oh, you can’t get there from here. Clear as day.”
As for why this spells disaster if anyone builds it, I didn’t need any new lessons, but they are here, and they are chilling—even for someone who was already convinced we were in trouble.
Having played some small part in helping this book come together, I would like to attest to the sheer amount of iteration it has gone through over the last year. Nate and co. have been relentlessly paring and grinding this text ever closer to the kind of accessibility that won’t just help individuals understand why we must act, but will make them feel like their neighbors and political leaders can understand it, too. I think that last part counts for a lot.
The book is also pretty engaging.
The pitch I suggested we share with our friends and allies is this:
If you’ve been waiting for a book that can explain the technical roots of the problem in terms your representative and your mother can both understand, this is the one. This is the grounded, no-nonsense primer on why superintelligence built blindly via gradient descent will predictably develop human-incompatible drives; on why humanity cannot hope to endure if the fatal, invisible threshold is crossed; and on what it will take to survive the coming years and decades.
You convinced me to pre-order it. In particular, these lines:
> It wasn’t until I read an early draft of this book a year ago that I felt like I could trace a continuous, solid line from “superintelligence grown by a blind process...” to “...develops weird internal drives we could not have anticipated”. Before, I was like, “We don’t have justifiable confidence that we can make something that reflects our values, especially over the long haul,” and now I’m like, “Oh, you can’t get there from here. Clear as day.”