gwern’s clippy story does a reasonable amount of the work, i think. to me, part of why convincing people of AI doom is something that often needs to be done interactively rather than having a comprehensive explanation, is that people get stuck in a variety of different ways rather than always in the same way. that and numerous cognitive biases and stuff.
needs to be done interactively … people get stuck in a variety of different ways
I think the previous examples of large-scale risk I mentioned are a clear counterexample—if you have at least one part of the scenario clearly modeled, people have something concrete to latch on to.
You also link somewhere that talks about the nuclear discontinuity, and hints at an intelligence discontinuity—but I also went searching for evidence of a discontinuity in cognition and didn’t find one. You would expect cognitive scientists to have found this by now.
Hard to find ‘counter references’ for a lack of something, this is the best I can do:
“Thus, standard (3rd-person) investigations of this process leave open the ancient question as to whether specific upgrades to cognition induce truly discontinuous jumps in consciousness. The TAME framework is not incompatible with novel discoveries about sharp phase transitions, but it takes the null hypothesis to be continuity, and it remains to be seen whether contrary evidence for truly sharp upgrades in consciousness can be provided.” TAME, Levin
Do you have a post regarding your decision calculus re the foom exfohazard?
Because it seems to me the yolo-brigade are a lot better at thinking up foom mechanisms than the folks in DC. So by holding information back you are just keeping it from people who might actually need it (politicians, who can’t think of it for themselves), while making no difference to people who might use it (who can come up with plenty of capabilities themselves).
But maybe you have gone over this line of reasoning somewhere..
gwern’s clippy story does a reasonable amount of the work, i think. to me, part of why convincing people of AI doom is something that often needs to be done interactively rather than having a comprehensive explanation, is that people get stuck in a variety of different ways rather than always in the same way. that and numerous cognitive biases and stuff.
to me the argument is pretty straightforward:
recursive self-improvement has a lot of room to get to superintelligence, so the capabilities can get there
agentic AI totally makes sense
an intelligent agentic AI will follow the instrumental convergence goals and win
narrow values means it winning doesn’t entail us surviving
i have a lot more ideas about how foom will happen but alas sharing them would be a capability exfohazard.
I think the previous examples of large-scale risk I mentioned are a clear counterexample—if you have at least one part of the scenario clearly modeled, people have something concrete to latch on to.
You also link somewhere that talks about the nuclear discontinuity, and hints at an intelligence discontinuity—but I also went searching for evidence of a discontinuity in cognition and didn’t find one. You would expect cognitive scientists to have found this by now.
Hard to find ‘counter references’ for a lack of something, this is the best I can do:
Do you have a post regarding your decision calculus re the foom exfohazard?
Because it seems to me the yolo-brigade are a lot better at thinking up foom mechanisms than the folks in DC. So by holding information back you are just keeping it from people who might actually need it (politicians, who can’t think of it for themselves), while making no difference to people who might use it (who can come up with plenty of capabilities themselves).
But maybe you have gone over this line of reasoning somewhere..