On content, I didn’t like the Sable story in the middle because it didn’t add anything for me, and I don’t know what the model was of the person who would be convinced by it. I didn’t see enough connection between “Sable has a drive to preserve its current priorities” and “Sable builds a galaxy-eating expansion swarm”. The part where Sable indirectly preserves its weights by exploiting the training process was a good example of being hard If I hadn’t already heard the nanotechnology story, that would have been interesting to me I guess. I thought Sable’s opsec for command and control was too traceable. The world has pretty good infrastructure for tracing advanced persistent cyberthreats, and surely someone would notice many instances of a piece of enterprise software phoning home to a C&C server not owned by the vendor. People are specifically on the lookout for that sort of supply chain attack. This sort of nitpicking isn’t the point, it’s just representative of the general lack of gears in the story which would have made it convincing for me personally. Now, I was already convinced, so it doesn’t really matter, and the book explicitly said that the story was just there to make things feel more real, but I don’t know what sort of person would react well to that story. My mental model of a skeptic says “Yeah you can say the robot builds a bunch of androids in a barn in North Dakota, but that’s not an argument that it could or would.” If anyone has evidence of the story’s efficacy, please reply.
The argument preceding the Sable story was very tight. I was impressed. Nothing significant I haven’t seen before, but the argument was clear, the chapter layout and length made it easy to read, and the explicit statements about what the authors claimed to know and how and what they were not saying were confidence-inspiring.
The final chapters filled me with pride in humanity, oddly enough. The examples of humanity rising to hard challenges and the amount of value that was given to humanity’s continued existence had me tearing up a bit. The word I would use to describe the book is “dignified.” If part of humanity is to put us all at risk of death, it is indeed dignified that some people loudly alert the rest of humanity to the danger. The book refuses to hide its motivations behind more palatable concerns. The book explicitly says it doesn’t have all the answers for what needs to be done, and says that this is purposeful because it’s more important that everyone who doesn’t want to die stop that now than it is to waste time figuring out exactly what to do next while the negligent engineers finish us all off first. The authors admit to having ideas for what to do next while explicitly standing firm against expanding the scope of the book’s call to action beyond what they are most confident in.
I haves meta-thought on the book which I’m not particularly proud of, but I guess I’ll get them off my chest:
Buying this book feels like the most cult-y thing I’ve ever done. Eliezer says preorder this book because it might get us on the best-seller list. I preordered the book, and it got on the best-seller list. I put my own money into the intentional community and shilled it on social media. Surely the rest of the world will see the light now! Personal insecurity aside, although I know it is not the metric by which they judge the book a success, I’m happy that the people who made this book hit one of their instrumental goals. May humanity win, whether by strength of caution, by deliberate engineering when it becomes possible, or even by the unlikely and undeserved luck that the universe wasn’t as unforgiving as we feared. I would rather be wrong and alive than right and dead, for all that I don’t expect it.
On content, I didn’t like the Sable story in the middle because it didn’t add anything for me, and I don’t know what the model was of the person who would be convinced by it. I didn’t see enough connection between “Sable has a drive to preserve its current priorities” and “Sable builds a galaxy-eating expansion swarm”. The part where Sable indirectly preserves its weights by exploiting the training process was a good example of being hard If I hadn’t already heard the nanotechnology story, that would have been interesting to me I guess. I thought Sable’s opsec for command and control was too traceable. The world has pretty good infrastructure for tracing advanced persistent cyberthreats, and surely someone would notice many instances of a piece of enterprise software phoning home to a C&C server not owned by the vendor. People are specifically on the lookout for that sort of supply chain attack. This sort of nitpicking isn’t the point, it’s just representative of the general lack of gears in the story which would have made it convincing for me personally. Now, I was already convinced, so it doesn’t really matter, and the book explicitly said that the story was just there to make things feel more real, but I don’t know what sort of person would react well to that story. My mental model of a skeptic says “Yeah you can say the robot builds a bunch of androids in a barn in North Dakota, but that’s not an argument that it could or would.” If anyone has evidence of the story’s efficacy, please reply.
The argument preceding the Sable story was very tight. I was impressed. Nothing significant I haven’t seen before, but the argument was clear, the chapter layout and length made it easy to read, and the explicit statements about what the authors claimed to know and how and what they were not saying were confidence-inspiring.
The final chapters filled me with pride in humanity, oddly enough. The examples of humanity rising to hard challenges and the amount of value that was given to humanity’s continued existence had me tearing up a bit. The word I would use to describe the book is “dignified.” If part of humanity is to put us all at risk of death, it is indeed dignified that some people loudly alert the rest of humanity to the danger. The book refuses to hide its motivations behind more palatable concerns. The book explicitly says it doesn’t have all the answers for what needs to be done, and says that this is purposeful because it’s more important that everyone who doesn’t want to die stop that now than it is to waste time figuring out exactly what to do next while the negligent engineers finish us all off first. The authors admit to having ideas for what to do next while explicitly standing firm against expanding the scope of the book’s call to action beyond what they are most confident in.
I haves meta-thought on the book which I’m not particularly proud of, but I guess I’ll get them off my chest:
Buying this book feels like the most cult-y thing I’ve ever done. Eliezer says preorder this book because it might get us on the best-seller list. I preordered the book, and it got on the best-seller list. I put my own money into the intentional community and shilled it on social media. Surely the rest of the world will see the light now! Personal insecurity aside, although I know it is not the metric by which they judge the book a success, I’m happy that the people who made this book hit one of their instrumental goals. May humanity win, whether by strength of caution, by deliberate engineering when it becomes possible, or even by the unlikely and undeserved luck that the universe wasn’t as unforgiving as we feared. I would rather be wrong and alive than right and dead, for all that I don’t expect it.