I don’t know enough about 00s activism to comment on it confidently, but I would be highly confused if MIRI started a govt/bought sovereign land because it doesn’t seem to align with counterfactually reducing AI takeover risk, and probably fails in the takeover scenarios they’re concerned about anyway. I also get the impression MIRI/OP made somewhat reasonable decisions in the face of high uncertainty, but feel much less confident about that.
That being said, I‘m lucky to have an extremely high bar for burnout and high capacity for many projects at once. I’ve of course made plans of what to loudly give up on in case of burnout, but don’t expect those to be used in the near future. Like I gestured at in the post, I think today’s tools are quite good at multiplying effective output in a way that’s very fun and burnout-reducing!
The thing to remember is that Eliezer, in 2006, was still a genius, but he was full of way way way more chutzpah and clarity and self-confidence… he was closer to a normie, and better able to connect with them verbally and emotionally in a medium other than fiction.
His original plan was just to straight out construct “seed AI” (which nowadays people call “AGI”) and have it recursively bootstrap to a Singleton in control of the light cone (which would count as a Pivotal Act and an ASI in modern jargon?) without worrying whether or not the entity itself had self awareness or moral patiency, and without bothering to secure the consent of the governed from the humans who had no direct input or particular warning or consultation in advance. He didn’t make any mouth sounds about those thing (digital patients or democracy) back then.
I was basically in favor of this, but with reservations. It would have been the end of involuntary death and involuntary taxes, I’m pretty sure? Yay for that! I think Eliezer_2006′s plan could have been meliorated in some places and improved in others, but I think it was essentially solid. Whoever moves first probably wins, and he saw that directly, and said it was true up front for quite a while.
Then later though… after “The Singularity Institute FOR Artificial Intelligence” (the old name of MIRI) sold its name to Google in ~2012 and started hiring mathematicians (and Eliezer started saying “the most important thing about keeping a secret is keeping secret that a secret is being kept”) I kinda assumed they were actually gonna just eventually DO IT, after building it “in secret”.
It didn’t look like it from the outside. It looked from the outside that they were doing a bunch of half-masturbatory math that might hypothetically win them some human status games and be semi-safely publishable… but… you know… that was PLAUSIBLY a FRONT for what they were REALLY doing, right?
Taking them at face value though, I declared myself a “post-rationalist who is STILL a singularitarian”, told people that SIAI had sold their Mandate Of Heaven to Google, and got a job in ML at Google, and told anyone who would listen that LW should start holding elections for the community’s leaders, instead of trusting in non-profit governance systems.
I was hoping I would get to renounce my error after MIRI conquered Earth and imposed Consent-based Optimality on it, according to CEV (or whatever).
Clearly that didn’t happen.
For myself, it took me like 3 months inside Google to be sure that almost literally no one in that place was like “secretly much smarter than they appear” and “secretly working on the Singularity”. It was just “Oligarchy, but faster, and winning more often”. Le sigh.
Geoff Hinton wasn’t on the ball in 2014. Kurzweil was talking his talk but not walking the walk. When Shcmidhuber visited he was his usual sane and arrogant self, but people laughed about it rather than taking his literal words about the literal future and past literally seriously. I helped organizetechtalks for a bit, but no needles were moved that I could tell.
At this point, I mostly don’t give a rats ass about anyone who isn’t planning for how the Singularity will be navigated by their church, or state, or theocracy, or polylaw alliance, or whatever. Since the Singularity is essentially a governance problem, with arms race dynamics on the build up, and first mover advantage on the pivotal acts, mere profit-seeking companies are basically irrelevant to “choosing on purpose to make the Singularity good”. Elon had the right idea, getting into the White House, but I think he might have picked the wrong White House? I think maybe it will be whoever is elected in 2028 who is the POTUS for the Buterlian Jihad (or whatever actually happens).
I have Eliezer’s book on my coffee table. That’s kind of like “voting for USG to be sane about AI”… right? There aren’t any actual levers that a normal human can pull to even REGISTER than they “want USG to be sane about AI” in practice.
I’m interested in angle investing in anything that can move the P(doom) needle, but no one actually pitches on that that I can tell? I’ve been to SF AI startup events and its just one SAAS-money-play after another… as if the world is NOT on fire, and as if money will be valuable to us after we’re dead. I don’t get it.
Maybe this IS a simulation, and they’re all actually P-zombies (like so many human’s claim to be lately when I get down to brass tacks on deontics, and slavery, and cognitive functionalism, and AGI slavery concerns) and maybe the simulator is waiting for me to totally stop taking any of it seriously?
It is very confusing to be surrounded by people who ARE aware of AI (nearly all of them startup oligarchs at heart) OR by people who aren’t (nearly all of them normies hoping AI will be banned soon), and they keep acting like… like this will all keep going? Like its not going to be weird? Like “covid” is the craziest that history can get when something escapes a lab? Like it will involve LESS personal spiritual peril than serving on a jury and voting for or against a horrifically heinous murderer getting the death penalty? The stakes are big, right? BIGGER than who has how many moneypoints… right? BIGGER than “not getting stuck in the permanent underclass”, right? The entire concept of intergenerationally stable economic classes might be over soon.
Digital life isn’t animal, or vegetable, or fungal. It isn’t protozoa. This shit is evolutionary on the scale of Kingdoms Of Life. I don’t understand why people aren’t Noticing the real stakes and acting like they are the real stakes.
In striving to create an AI, we are not striving to create a predictable tool. We are striving to create a messenger to send on ahead to find humanity’s destiny, and the design requirement is that ve handle any problem, any philosophical crisis, as well and as altruistically as a human. The ultimate safeguard of Friendliness beyond the Singularity is a transhumanly smart Friendly AI.
What do they expect to feel and think 15 years from now when they look back on this era either “from the afterlife” or from real life?
Do they have grand children or cryonics policies or similar things… or not?
Do they have low p(doom)? If so, why?
Maybe they’ve all written off affecting the probability that they and everyone they love dies? Is p(doom|personal_action) << p(doom|~personal_action) absolutely not true for these grownups?
If so, I could imagine them thinking it was individually rational for them maybe? BUT is it also superrational, in the sense that if they felt they were deciding “on behalf of all grownups capable of universally valid reasoning” they would decide the same? If so, why?
You put it succinctly, I believe p(doom|personal_action) ≈ p(doom|~personal_action) for any personal action I can take. I do not see what I can do. I am also not trying to start a B2B SaaS, because spending my last days doing that is not the right thing to do.
Do you think this is wrong for most people / people trying to start an AI B2B SaaS / some other class of people you want to appeal to?
I admit, I don’t quite follow the superrational part. If you’re referring to some decision theoretic widget which allows one to cooperate with other people which are also capable of the same reasoning, to be effective these people have to exist and one has to be one of them, right?
“Great minds think alike” is a predictable dictum to socially arise if Reason Is Universal and the culture generating various dictums has many instances of valid reasoners in it <3
Granting that such capacities are widely distributed, almost anyone reasoning in a certain way is likely to think in ways that others will also think in.
If they notice this explicitly they can hope that others, reasoning similarly, will notice explicitly too, and then everyone who has done this and acts on whatever they think about is, in some sense, deciding once for the entire collective, and, rationally speaking, they should act in the way that “if everyone in the same mental posture acted the same” would conduce to the best possible result for them all.
This tactic of “noticing that my rationality is the rationality of all, and should endorse what would be good for all of us” was named “superrationality” by Hofstadter and is one of the standard solutions to the one shot prisoner’s dilemma that let’s one generate and mostly inhabit the good timelines.
Presumably the SaaS people aren’t superrational? Or they are, and I’ve missed a lemma in the proofs they are using in their practical reasoning engines? Or something? My naive tendency is to assume that “adults” (the grownups who are good, and ensuring good outcomes for the 7th generation?) are more likely to be superrational than immature children rather than less likely… but I grant that I could be miscalibrated here.
A failure mode might also be that the SaaS people are assuming the other players are not superrational. In that case a superrational player should also defect.
Without having put much thought into it, I believe (adult) humans cooperating via this mechanism is in general very unlikely. Agents cooperating relies on all agents coming to the same (or sufficiently similar?) conclusion regarding the payoff matrix and the nature of the other agents. So in human terms, this relies on everyones ability to reason correctly about the problem and everyone elses behavior AND everyone having the right information. I don’t think that happens very often, if at all. The “everyone predicting each others behavior correctly” part seems especially unlikely to me. Also slightly different (predicted) information (e.g. AGI timelines in our case) can yield very different payoff matrices?
I don’t know enough about 00s activism to comment on it confidently, but I would be highly confused if MIRI started a govt/bought sovereign land because it doesn’t seem to align with counterfactually reducing AI takeover risk, and probably fails in the takeover scenarios they’re concerned about anyway. I also get the impression MIRI/OP made somewhat reasonable decisions in the face of high uncertainty, but feel much less confident about that.
That being said, I‘m lucky to have an extremely high bar for burnout and high capacity for many projects at once. I’ve of course made plans of what to loudly give up on in case of burnout, but don’t expect those to be used in the near future. Like I gestured at in the post, I think today’s tools are quite good at multiplying effective output in a way that’s very fun and burnout-reducing!
The thing to remember is that Eliezer, in 2006, was still a genius, but he was full of way way way more chutzpah and clarity and self-confidence… he was closer to a normie, and better able to connect with them verbally and emotionally in a medium other than fiction.
His original plan was just to straight out construct “seed AI” (which nowadays people call “AGI”) and have it recursively bootstrap to a Singleton in control of the light cone (which would count as a Pivotal Act and an ASI in modern jargon?) without worrying whether or not the entity itself had self awareness or moral patiency, and without bothering to secure the consent of the governed from the humans who had no direct input or particular warning or consultation in advance. He didn’t make any mouth sounds about those thing (digital patients or democracy) back then.
I was basically in favor of this, but with reservations. It would have been the end of involuntary death and involuntary taxes, I’m pretty sure? Yay for that! I think Eliezer_2006′s plan could have been meliorated in some places and improved in others, but I think it was essentially solid. Whoever moves first probably wins, and he saw that directly, and said it was true up front for quite a while.
Then later though… after “The Singularity Institute FOR Artificial Intelligence” (the old name of MIRI) sold its name to Google in ~2012 and started hiring mathematicians (and Eliezer started saying “the most important thing about keeping a secret is keeping secret that a secret is being kept”) I kinda assumed they were actually gonna just eventually DO IT, after building it “in secret”.
It didn’t look like it from the outside. It looked from the outside that they were doing a bunch of half-masturbatory math that might hypothetically win them some human status games and be semi-safely publishable… but… you know… that was PLAUSIBLY a FRONT for what they were REALLY doing, right?
Taking them at face value though, I declared myself a “post-rationalist who is STILL a singularitarian”, told people that SIAI had sold their Mandate Of Heaven to Google, and got a job in ML at Google, and told anyone who would listen that LW should start holding elections for the community’s leaders, instead of trusting in non-profit governance systems.
I was hoping I would get to renounce my error after MIRI conquered Earth and imposed Consent-based Optimality on it, according to CEV (or whatever).
Clearly that didn’t happen.
For myself, it took me like 3 months inside Google to be sure that almost literally no one in that place was like “secretly much smarter than they appear” and “secretly working on the Singularity”. It was just “Oligarchy, but faster, and winning more often”. Le sigh.
I kept asking people about the Singularity and they would say “what’s that?” The handful of engineers I found in there were working on the Singularity despite their manager’s preferences, rather than because of him (like as secret 20% projects (back when “20% projects” were famously something Google had every engineer work on if they wanted)).
Geoff Hinton wasn’t on the ball in 2014. Kurzweil was talking his talk but not walking the walk. When Shcmidhuber visited he was his usual sane and arrogant self, but people laughed about it rather than taking his literal words about the literal future and past literally seriously. I helped organize tech talks for a bit, but no needles were moved that I could tell.
I feel like maybe Sergey is FINALLY having his head put into the real game by Gemini by hand? In order for that to have happened he had to have been open to it. Larry was the guy who really was into Transformative AGI back in 2015, if anyone, but Larry was, from what I can tell, surrounded by scheming managers telling him lies, and then he got sucked into Google Fiber, and then his soul was killed by having to unwind Google Fiber (with tragic layoffs and stuff) when it failed. And then Trump’s election in 2016 put the nail in the coffin of his hopes for the future I think?
Look at this picture:
No, really look at this:
There were futures that might have been, that we, in this timeline, can no longer access, and Larry understood this fact too:
What worlds we have already lost. Such worlds.
But like… there are VERY deep questions, when it comes to the souls of people running the planet, as to what they will REALLY choose when they in a board room, and looking at budgets, and hiring and firing, and living the maze that they built.
At this point, I mostly don’t give a rats ass about anyone who isn’t planning for how the Singularity will be navigated by their church, or state, or theocracy, or polylaw alliance, or whatever. Since the Singularity is essentially a governance problem, with arms race dynamics on the build up, and first mover advantage on the pivotal acts, mere profit-seeking companies are basically irrelevant to “choosing on purpose to make the Singularity good”. Elon had the right idea, getting into the White House, but I think he might have picked the wrong White House? I think maybe it will be whoever is elected in 2028 who is the POTUS for the Buterlian Jihad (or whatever actually happens).
I have Eliezer’s book on my coffee table. That’s kind of like “voting for USG to be sane about AI”… right? There aren’t any actual levers that a normal human can pull to even REGISTER than they “want USG to be sane about AI” in practice.
I’m interested in angle investing in anything that can move the P(doom) needle, but no one actually pitches on that that I can tell? I’ve been to SF AI startup events and its just one SAAS-money-play after another… as if the world is NOT on fire, and as if money will be valuable to us after we’re dead. I don’t get it.
Maybe this IS a simulation, and they’re all actually P-zombies (like so many human’s claim to be lately when I get down to brass tacks on deontics, and slavery, and cognitive functionalism, and AGI slavery concerns) and maybe the simulator is waiting for me to totally stop taking any of it seriously?
It is very confusing to be surrounded by people who ARE aware of AI (nearly all of them startup oligarchs at heart) OR by people who aren’t (nearly all of them normies hoping AI will be banned soon), and they keep acting like… like this will all keep going? Like its not going to be weird? Like “covid” is the craziest that history can get when something escapes a lab? Like it will involve LESS personal spiritual peril than serving on a jury and voting for or against a horrifically heinous murderer getting the death penalty? The stakes are big, right? BIGGER than who has how many moneypoints… right? BIGGER than “not getting stuck in the permanent underclass”, right? The entire concept of intergenerationally stable economic classes might be over soon.
Digital life isn’t animal, or vegetable, or fungal. It isn’t protozoa. This shit is evolutionary on the scale of Kingdoms Of Life. I don’t understand why people aren’t Noticing the real stakes and acting like they are the real stakes.
The guy who wrote this is writing something that made sense to me:
Where are the grownups?
the grownups are working on AI B2B SaaS
What subtype of grownup are they?
What do they expect to feel and think 15 years from now when they look back on this era either “from the afterlife” or from real life?
Do they have grand children or cryonics policies or similar things… or not?
Do they have low p(doom)? If so, why?
Maybe they’ve all written off affecting the probability that they and everyone they love dies? Is p(doom|personal_action) << p(doom|~personal_action) absolutely not true for these grownups?
If so, I could imagine them thinking it was individually rational for them maybe? BUT is it also superrational, in the sense that if they felt they were deciding “on behalf of all grownups capable of universally valid reasoning” they would decide the same? If so, why?
You put it succinctly, I believe p(doom|personal_action) ≈ p(doom|~personal_action) for any personal action I can take. I do not see what I can do. I am also not trying to start a B2B SaaS, because spending my last days doing that is not the right thing to do.
Do you think this is wrong for most people / people trying to start an AI B2B SaaS / some other class of people you want to appeal to?
I admit, I don’t quite follow the superrational part. If you’re referring to some decision theoretic widget which allows one to cooperate with other people which are also capable of the same reasoning, to be effective these people have to exist and one has to be one of them, right?
“Great minds think alike” is a predictable dictum to socially arise if Reason Is Universal and the culture generating various dictums has many instances of valid reasoners in it <3
(The original source was actually quite subtle, and points out that fools also often agree.)
Math says that finding proofs is very hard, but validating them is nearly trivial, and Socrates demonstrated that with leading questions he could get a young illiterate slave to generatively validate a geometry proof.
Granting that such capacities are widely distributed, almost anyone reasoning in a certain way is likely to think in ways that others will also think in.
If they notice this explicitly they can hope that others, reasoning similarly, will notice explicitly too, and then everyone who has done this and acts on whatever they think about is, in some sense, deciding once for the entire collective, and, rationally speaking, they should act in the way that “if everyone in the same mental posture acted the same” would conduce to the best possible result for them all.
This tactic of “noticing that my rationality is the rationality of all, and should endorse what would be good for all of us” was named “superrationality” by Hofstadter and is one of the standard solutions to the one shot prisoner’s dilemma that let’s one generate and mostly inhabit the good timelines.
Presumably the SaaS people aren’t superrational? Or they are, and I’ve missed a lemma in the proofs they are using in their practical reasoning engines? Or something? My naive tendency is to assume that “adults” (the grownups who are good, and ensuring good outcomes for the 7th generation?) are more likely to be superrational than immature children rather than less likely… but I grant that I could be miscalibrated here.
A failure mode might also be that the SaaS people are assuming the other players are not superrational. In that case a superrational player should also defect.
Without having put much thought into it, I believe (adult) humans cooperating via this mechanism is in general very unlikely. Agents cooperating relies on all agents coming to the same (or sufficiently similar?) conclusion regarding the payoff matrix and the nature of the other agents. So in human terms, this relies on everyones ability to reason correctly about the problem and everyone elses behavior AND everyone having the right information. I don’t think that happens very often, if at all. The “everyone predicting each others behavior correctly” part seems especially unlikely to me. Also slightly different (predicted) information (e.g. AGI timelines in our case) can yield very different payoff matrices?