I feel like this is giving way too little weight/salience to humans being really bad at strategy and philosophy in general, and in particular MIRI being bad at strategy and philosophy. You talk about Holden being misled about AGI for 8 years, but don’t mention MIRI planning to build recursively improving Friendly AI with a small team and potentially just 1 philosopher, for a comparable amount of time. If OpenPhil had funded MIRI more in 2016, it would have been funding them to attempt this!
Just in course of searching for my name in the comments section of Holden’s Thoughts on the Singularity Institute (SI), I came across more examples (of humans being bad at strategy and philosophy):
Eliezer banning Roko’s Basilisk post (and Roko posting it in the first place)
Eliezer apparently forgetting or disregarding most previous critics of MIRI (e.g. me): “Nonetheless, it already has a warm place in my heart next to the debate with Robin Hanson as the second attempt to mount informed criticism of SIAI.”
Eliezer forgetting about banning Roko’s post, writing in all caps “Once again: ROKO DELETED HIS OWN POST. NO OUTSIDE CENSORSHIP WAS INVOLVED.”, and komponisto chickening out of reminding Eliezer about it.
cousin_it avoiding the debate because “I realized that talking about saving the world makes me really upset and I’m better off avoiding the whole topic”
MIRI hiring Ben Goertzel as Research Director even though Eliezer thought his AGI project would kill everyone if it succeeded (“And if Novamente should ever cross the finish line, we all die.”), because “Ben Goertzel’s projects are knowably hopeless, so I didn’t too strongly oppose Tyler Emerson’s project from within SIAI’s then-Board of Directors; it was being argued to have political benefits, and I saw no noticeable x-risk so I didn’t expend my own political capital to veto it, just sighed. Nowadays the Board would not vote for this.”
I think Roko posting the thing was ok. It was in the same class of things being discussed at the time, like Rolf Nelson’s AI deterrence and so on. Eliezer overreacted and caused a Streisand effect, without it only a few of us would even remember it today.
To add color to the point about me: I was a research associate at MIRI then (then called SI). I’d joined in the hope of doing decision theory math, but found that there wasn’t as much math I liked happening inside. However, there were many email discussions about saving the world, which I first tried hard to follow, but then they became just really overwhelming for me. That’s the background of my remark. Later that year I left the program. Maybe you’re right and this all was a failure of strategy on my part :-)
Thanks for the backstory! (Don’t know if you remember, but I asked you back then and you didn’t want to talk on the meta level either.) I think what I was trying to say by citing you is that humans can get emotionally overwhelmed just by thinking about the world ending or saving the world, which probably isn’t great for their strategic competence when dealing with this topic.
Yeah. Maybe it wasn’t even due to that specific topic, could’ve been anything else, like knitting. There was just a lot of emails about it (several every day for months?) and for some reason it felt really hard to follow for me, on top of my work at Google at the time. So then it flipped around to “don’t wanna talk about it, don’t wanna meta-talk about it, just make it go away”. I’m sorry the backstory isn’t more dignified.
MIRI’s attempt to solve technical alignment didn’t make the global situation significantly worse whereas OpenPhil’s funding of OpenAI and other “safety” work did.
Do you really think we’d be better off if no competent team with funding had made a sustained attempt to solve technical alignment? Alternatively, do you believe that there have been other competently-led sustained attempts to solve it outside of MIRI?
Eliezer said in an interview published on Youtube that “I did my crying in 2015” or words very similar to that, by which he meant that that is when (presumably after the founding of OpenAI) he realized the global situation is hopeless, which makes me wonder how you come to believe that “if OpenPhil had funded MIRI more in 2016, it would have been funding them to attempt” “to build recursively improving Friendly AI with a small team”.
It may be worth noting that the basilisk incident also prefigured the FTX fraud (which became a massive crisis for EA) by way of the “quantum billionaire trick” that Roko described in the same post.
What’s the main difference between “Humans are bad at strategy” vs. “Strategy is difficult, if not outright impossible”? The two toy models of the latter are my other comment and titotal’s experiment where, without a queen, not even Stockfish managed to outperform the human at chess, even though the human supposedly had the Elo rating of 1100.
I feel like this is giving way too little weight/salience to humans being really bad at strategy and philosophy in general, and in particular MIRI being bad at strategy and philosophy. You talk about Holden being misled about AGI for 8 years, but don’t mention MIRI planning to build recursively improving Friendly AI with a small team and potentially just 1 philosopher, for a comparable amount of time. If OpenPhil had funded MIRI more in 2016, it would have been funding them to attempt this!
Just in course of searching for my name in the comments section of Holden’s Thoughts on the Singularity Institute (SI), I came across more examples (of humans being bad at strategy and philosophy):
Eliezer banning Roko’s Basilisk post (and Roko posting it in the first place)
Eliezer apparently forgetting or disregarding most previous critics of MIRI (e.g. me): “Nonetheless, it already has a warm place in my heart next to the debate with Robin Hanson as the second attempt to mount informed criticism of SIAI.”
Eliezer forgetting about banning Roko’s post, writing in all caps “Once again: ROKO DELETED HIS OWN POST. NO OUTSIDE CENSORSHIP WAS INVOLVED.”, and komponisto chickening out of reminding Eliezer about it.
cousin_it avoiding the debate because “I realized that talking about saving the world makes me really upset and I’m better off avoiding the whole topic”
MIRI hiring Ben Goertzel as Research Director even though Eliezer thought his AGI project would kill everyone if it succeeded (“And if Novamente should ever cross the finish line, we all die.”), because “Ben Goertzel’s projects are knowably hopeless, so I didn’t too strongly oppose Tyler Emerson’s project from within SIAI’s then-Board of Directors; it was being argued to have political benefits, and I saw no noticeable x-risk so I didn’t expend my own political capital to veto it, just sighed. Nowadays the Board would not vote for this.”
I think Roko posting the thing was ok. It was in the same class of things being discussed at the time, like Rolf Nelson’s AI deterrence and so on. Eliezer overreacted and caused a Streisand effect, without it only a few of us would even remember it today.
To add color to the point about me: I was a research associate at MIRI then (then called SI). I’d joined in the hope of doing decision theory math, but found that there wasn’t as much math I liked happening inside. However, there were many email discussions about saving the world, which I first tried hard to follow, but then they became just really overwhelming for me. That’s the background of my remark. Later that year I left the program. Maybe you’re right and this all was a failure of strategy on my part :-)
Thanks for the backstory! (Don’t know if you remember, but I asked you back then and you didn’t want to talk on the meta level either.) I think what I was trying to say by citing you is that humans can get emotionally overwhelmed just by thinking about the world ending or saving the world, which probably isn’t great for their strategic competence when dealing with this topic.
Yeah. Maybe it wasn’t even due to that specific topic, could’ve been anything else, like knitting. There was just a lot of emails about it (several every day for months?) and for some reason it felt really hard to follow for me, on top of my work at Google at the time. So then it flipped around to “don’t wanna talk about it, don’t wanna meta-talk about it, just make it go away”. I’m sorry the backstory isn’t more dignified.
MIRI’s attempt to solve technical alignment didn’t make the global situation significantly worse whereas OpenPhil’s funding of OpenAI and other “safety” work did.
Do you really think we’d be better off if no competent team with funding had made a sustained attempt to solve technical alignment? Alternatively, do you believe that there have been other competently-led sustained attempts to solve it outside of MIRI?
Eliezer said in an interview published on Youtube that “I did my crying in 2015” or words very similar to that, by which he meant that that is when (presumably after the founding of OpenAI) he realized the global situation is hopeless, which makes me wonder how you come to believe that “if OpenPhil had funded MIRI more in 2016, it would have been funding them to attempt” “to build recursively improving Friendly AI with a small team”.
It may be worth noting that the basilisk incident also prefigured the FTX fraud (which became a massive crisis for EA) by way of the “quantum billionaire trick” that Roko described in the same post.
What’s the main difference between “Humans are bad at strategy” vs. “Strategy is difficult, if not outright impossible”? The two toy models of the latter are my other comment and titotal’s experiment where, without a queen, not even Stockfish managed to outperform the human at chess, even though the human supposedly had the Elo rating of 1100.