Why are you asking AIs overwhelmingly sympathetic to FDT whether they favor EDT or CDT? Why are you benchmarking them on only problems labeled EDT or CDT and not Parfit’s Hitchhiker or whether to vote in elections, as discriminate EDT/CDT from FDT?
Eliezer Yudkowsky
Karma: 165,476
I rather expected from that time that Aschenbrenner would be trouble for EA, if they just couldn’t stop themselves from promoting Harvard blondes; but I did not say anything against him then on the basis of that conversation, and only marked him down as actively harmful once I saw him starting to oppose all good ideas during his brief tenure at FTX philanthropy. He did of course further go on to actively stoke arms races with China.
The relevance to the Wei Dai part is that Wei Dai seemed to me to be asking, “Aren’t you deciding post facto who is ‘EA’ and who is ‘rationalist’?” and my reply to Wei Dai is implicitly, “My memory claims that it is really really not hard to tell, in advance, early on.”
I remember the first time I met Leopold Aschenbrenner. He questioned me about why my timelines weren’t longer and finally said, “Why aren’t you updating at all in the direction of all the smart people saying longer timelines?” (aka: OpenPhil doctrine of 2050), “At this point you’re not even pretending to be rational.” I hesitated briefly, because I knew it would not end well to answer honestly, but there wasn’t any good ending past there; and then I told him honestly that I did not consider those people to be my epistemic peers.
I don’t usually consider myself good at reading faces, but wow, the sheer LOOK of contempt and scorn that crossed his face; before he turned away and walked off without another word, dismissing me from further consideration. I will never forget that moment—because of it being such a LOUD face and my actually being able to read it, not because it had any influence on my estimates, of course.
Amusingly, I have later heard from OpenPhil people telling me very earnestly that they think OpenPhil avoided groupthink on AGI timelines because there was dissent allowed inside OpenPhil. Well, sure there was allowed a pretense of dissent inside the OpenPhil Overton window, unless you said that you weren’t updating your opinion in favor of deferring to the people whom OpenPhil considered locally high-status enough that deferring to them was rational, and then you would get frozen out and dismissed from all consideration. I can only imagine the Look that Aschenbrenner would have given somebody who’d said the same thing while being an intern rather than Yudkowsky, and while other OpenPhil personnel might’ve been more polite, I don’t really see that intern being hired.
OpenPhil and EAs may consider themselves to be terribly modest and outside-viewing, but of course that’s all an utter sham given their actual skill levels, and they have no concept of what it would actually look like for them to be humble.
It doesn’t surprise me in the least what Aschenbrenner went on to do. But yes, he is definitely a “Modesty!! Outside view!!!” sort of person, or knows that is what he is supposed to perform. I don’t expect there will ever be any contradiction that he understands, between “modesty”, and whatever the hell thought he decides to take into his head. That would take skill, and he is not aware that he is unskilled.
Luke Mueuhlhauser is actually modest. I don’t think he would ever blow up a hedge fund. It also didn’t surprise me at all when he moved on to OpenPhil, and I was glad for him, because I knew MIRI had not fit him and he would be happier at OpenPhil, and Luke had worked hard at MIRI and maybe saved the organization while he lasted. In related news, EAs picked Aschenbrenner rather than Mueuhlhauser to run their hedge fund.
Carl Shulman and Luke Mueulhauser were very Modesty-Argument / “outside view” people, which tightly corresponds to getting sucked in by the EA cluster once it exists / makes them an offer.
Thank you for the pointer; I have now tweeted about this and made the uncertain guess you did not particularly want a “H/T Wei Dai” about it.
“What if we’re in a simulation that I can discern to be visibly unusual” discourse popping up again.
And whatever the utility function of gods, you imagine (1) there’s not much they could possibly find any more entertaining than that, and (2) there’s not that many other possibilities for things that would be equally entertaining?
The conceit of imagining yourself to be Top-Tier Entertainment for a god. Why, there’s not just many possible things to imagine, so why wouldn’t it imagine you, amirite?
Do you believe that God intervened in the 2016 election? All these arguments carry over identically to God.
I’m renaming “XOR Blackmail” to “XOR Demand”. Does anyone want to object to that?
One can try to build AIs in correspondence with deontology (corrigibility), consequentialism (value-aligned sovereigns), or virtue ethics (worthy successors).
Corrigibility is of these the easiest to define, easiest to test, easiest to build, and the easiest to check for the first worrisome signs that not all is going great—though of course if you wait long enough in capabilities escalation to test, everything will look great to you.
Any sane person who was not allowed to back off the problem would try for corrigibility, by far the easiest of the three.
Worthy successors are the hardest to build, the least well-defined, the hardest to verify that you are doing correctly; the easiest place to leap on little local events that you can convince yourself are signs of hope, because you have no framework to tell you what more is required; hence, the craziest thing to attempt, beyond even a nice Sovereign ASI. So of course all the fools, having failed at corrigibility, will convince themselves they are building worthy successors instead; migrating, with the inevitability of an amoeba following an agar trail, to wherever it is hardest for fools to be persuaded (with a fool’s demanded certainty) that whatever is currently happening is not all according to their plan.
Or legalize childcare. There’s no need to subsidize it or exempt it from taxes, just make it legal for anyone I trust to sell me some childcare in exchange for money.
If I were doing it all over, I would be much less terrified of something called “cultishness”, would have installed something more like an accreditation or licensing system to be considered a member in good standing, and then a requirement of being in good standing would be that you were either raising kids or contributing money to those who did, located somewhere it was legal to build housing and where they could have houses in close proximity. (Some huge amount of modern pain of childrearing is downstream of having no burden-sharing on babysitting and nowhere to put a bunch of kids where they can play together instead of with you.) I expect that’s how Civilization does it in dath ilan; you either make babies yourself or you fund the sort of babies you want to see in the world.
To state the obvious:
Many of the same people who publicly make a great fuss of this don’t seem to care that Nvidia is selling GPUs to China by way of Singapore. Their purpose is to drag up China as an excuse for continuing what they are doing. If China has more GPUs, all the better for them in the future; China will be a greater threat that they can use to continue what they are doing.
Their model hasn’t chosen an explanation like “China will play the role of the movie bad guy at any cost to their own selfish interests”. They aren’t reasoning at GPT 3.5-level, only GPT-2 level. China is a scary word-vector, so they can argue that if the USA tries to slow down against extinction risk, then China won’t slow down, because that’s a scary thing so “China” is a word-vector that the scary thing can be attributed to.
4. The main argument against abolition was that if the British didn’t do it, other countries would.
Suggest rephrase to “if the British didn’t trade slaves”. I read this as “if the British didn’t do [abolition]” at first, and was confused.
(Posted to a Discord, and they said, “Shortform that on LW”.)
I’d say optimize some around “how much would I regret dying in a few months”, eg by taking your kids on long-imagined trips (as an old old friend recently asked me about). But the constraint you should obey is “don’t spend down resources to the point where it would feel bad to hear the world was not ending”. Eg, if it became scary rather than joyous to hear that an international treaty was signed against AI escalation and suddenly three more decades was a real possibility.
Similarly with not pressuring your kids to get a horrible job immediately. Yeah, it makes sense not to press, because there might not be a payoff for them. But don’t corner or depressure your kids to the point where they become dysfunctional (on the margins where they were previously scheduled to be functional), such that you or they would be sad to hear the world was not ending (where you wouldn’t previously have been thus sad).
Maybe tldr: “Reduce your regrets if the world were to end shortly, but not in ways that would make you regret to hear that the world wasn’t ending.”
Rowling once stated in an interview that magical genes were “strongly dominant” in which case two marrying Muggleborns would constantly be throwing non-magical offspring. There are entire papers trying to reconstruct how magical genes could realistically work instead, albeit I think they tend to come up with answers that make bad fiction compared to HPMOR. Eg https://pmc.ncbi.nlm.nih.gov/articles/PMC2151141/. But ultimately it is a truth of the final origins of the observed phenomena of Harry Potter that Rowling did not understand genes and was running on Folk Heredity.
The HPMOR notion here might be that Seamus’s dad was legally a Muggle in the eyes of the legal system since he has no magical parents and they barely acknowledge the notion of “Muggleborn”; but that, whaddaya know, he was the sort of “Muggle” whose eyes don’t slide off a magical chest (Squib phenotype).
It’s the difference between recessive genes and dominant genes. In HPMOR there is an allele controlling the degree of magical ability, with a Magical and a Mundane genotype. Having one copy of the Magical genotype and one copy of the Mundane genotype gives you what HPMOR takes to be a Squib phenotype, no longer defined as “the non-magical child of a magical couple” per Rowling canon, but instead defined to be a distinct phenotype that can perceive magical events without them being blotted out by Muggle-confusing charms, but not cast magical spells. Then Muggleborn wizards may be born to two parents each with the Squib phenotype, whose chance recombination of chromosomes yields two copies of the Magical genotype creating the full Wizard phenotype.
If that sounded weird you might wanna go read up on “Mendelian genetics”.
If it didn’t sound weird, the genetically literate mind informed by canon will now inquire how two Wizard phenotypes ever yield a Squib phenotype when Wizards are Magical-Magical homozygous. The answer is not fully given in HPMOR proper, but the author notes that to him the answer forced by inference from HPMOR’s postulates is that (rot13):
Gurer vf abg n Zntvpny trar pbzcyrk gung perngrf zntvp, ohg n Zhaqnavgl trar pbzcyrk gung fhccerffrf vg. Nsgre nyy, sebz cher culfvpf jr pna arire trg gb zntvp, ohg va na vaangryl zntvpny havirefr fbzrobql pbhyq pnfg n fcryy gb perngr gur ybpny nccrnenapr bs zbfgyl culfvpf. Fvzvyneyl vs nyy orvatf fgneg bhg zhaqnar gura ab trar pbzcyrk pna cbffvoyl perngr zntvp va gurz, ohg vs nyy orvatf fgneg bhg zntvpny fbzr trar pbzcyrk pbhyq fhccerff gurve zntvp. Fdhvof naq hygvzngryl Zhttyrobeaf gura nevfr anghenyyl sebz n cbchyngvba bs Zhttyr curabglcrf nf zhgngvbaf qnzntr gur Zhaqnavgl trar pbzcyrk, naq va ghea Jvmneq curabglcrf fbzrgvzrf lvryq Fdhvof nf gur Zhaqnavgl trar pbzcyrk fbzrgvzrf ercnvef vgfrys ol puebzbfbzny pebffbire, jvgu gur curabzraba orvat zber pbzzbayl bofreirq va jvmneqvat snzvyvrf jvgu erprag Zhttyr naprfgel, gb gur nynez bs trargvpf-vyyvgrengr cheroybbqf.
Sebz gur Zraqryvna cnggrea vzcylvat gur Zhaqnavgl trar pbzcyrk vf nyy ba bar puebzbfbzr, gur trar pbzcyrk pna or vasreerq gb or negvsvpvny va angher.
Does training to endorse CDT make the agent generally stupider? (Larger models with more inference time move more toward FDT behaviors and explicitly endorsing FDT, so it’s an obvious thing to check.)