I’ve been a LessWrong organizer since 2011, with roughly equal focus on the cultural, practical and intellectual aspects of the community. My first project was creating the Secular Solstice and helping groups across the world run their own version of it. More recently I’ve been interested in improving my own epistemic standards and helping others to do so as well.
Raemon(Raymond Arnold)
Karma: 33,010
This statement is kinda opaque and I’d like it if you spelled out your arguments more. (I realize it’s not always worth the effort to wade into the fully argumentation, but a point of the review is to more-fully hash out arguments for posts. Road To Mazedom ranked at #19 during the preliminary voting, so if there’s disagreement about it I think it’s good to spell out).
(I don’t necessarily disagree with your claim, but as worded it doesn’t really convey anything beyond “Romeo thinks it’s misleading”)
I might be conflating Richard, Paul, and my own guesses here. But, I think part of the argument here is about what can happen before AGI, that gives us lines of hope to pursue.
Like, my-model-of-Paul wants various tools for amplifying his own thought to (among other things) help think about solving the longterm alignment problem. And the question is whether there are ways of doing that that actually help when trying to solve the sorts of problems Paul wants to solve. We’ve successfully augmented human arithmetic and chess. Are there tools we actually wish we had, that narrow AI meaningfully helps with,
I’m not sure if Richard has a particular strategy in mind, but I assume he’s exploring the broader question of “what useful things can we build that will help navigate x-risk”
The original dialogs were exploring the concept of pivotal acts that could change humanity’s strategic position. Are there AIs that can execute pivotal acts that are more like calculators and Deep Blue than like autonomous moon-base-builders? (I don’t know if Richard actually shares the pivotal act / acute risk period frame, or was just accepting it for sake of argument)
But I think Richard’s point is ‘but we totally built AIs that defeated grand chess masters without destroying the world. So, clearly it’s possible to use tool AI to do this sort of thing. So… why do think various domains will reliably output horrible outcomes? If you need to cure cancer, maybe there is an analogous way to cure cancer that just… isn’t trying that hard?’
Richard is that what you were aiming at?
Curious if you have particular exemplar posts from sociology.
Periodically I see people make the “I’d like to see you engaging with the mainstream literature” comment, but then, well, there’s a whole lot of mainstream literature and I’m not sure how to sift through it for the parts that are relevant to me. Do you actually have experience with the literature or just an assumption that there’s stuff there? (if you are familiar, I think it’d be great to have an orientation post giving an overview of it)
This was on the long side, covering a lot of points. I’m curious to get feedback from people who ended up skimming and bounced off about where they did so.
Implications of Civilizational Inadequacy (reviewing mazes/simulacra/etc)
does it have a name, or just a vaguely amorphous concept blob?
While writing this, I was reminded of an older (2017) conversation between Eliezer and Paul on FB. I reread it to see whether Paul seemed like he’d be making the set of mistakes this post is outlining.
It seems like Paul acknowledges the issues here, but his argument is that you can amplify humans without routing through “the hard parts” that are articulated in this post. i.e. it seems like you can use current ML to build something that helps a human effectively “think longer”, and he thinks one can do this without routing through the dangerous-plan-searchspace. I don’t know if there’s much counterargument beyond “no, if you’re building an ML system that helps you think longer about anything important, you already need to have solved the hard problem of searching through plan-space for actually helpful plans.”
But, here were some comments from that thread I found helpful to reread.
Eliezer:
I have a new oversimplied straw way to misdescribe the disagreement between myself and Paul Christiano, and I’m interested to hear what Paul thinks of it:
Paul sees “learn this function” or “learn and draw from this probability distribution” as a kind of primitive op that modern machine learning is increasingly good at and can probably give us faithful answers to. He wants to figure out how to compose these primitives into something aligned.
Eliezer thinks that what is inside the black box inexorably kills you when the black box is large enough, like how humans are cognitive daemons of natural selection (the outer optimization process operating on the black box of genes accidentally constructed a (sapient) inner optimization process inside the black box) and this is chicken-and-egg unavoidable whenever the black box is powerful enough to do something like predict complicated human judgments, since in this case the outer optimization was automatically powerful enough to consider and select among multiple hypotheses the size of humans, and the inner process is automatically as powerful as human intelligence.
Paul thinks it may be possible to do something with this anyway given further expenditures of cleverness and computation, like taking a million such black boxes and competing them to produce the best predictions.
Eliezer expects an attempt like this to come with catastrophically exponential sample-complexity costs in theory, and to always fail in practice because you are trying to corral hostile superintelligences which means you’ve already lost. E.g. we can tell stories about how inner predictors could take over AIXI-tl by using bad off-policy predictions to manipulate AIXI-tl into a position where only that predictor (or LDT-symmetrized predictor class) can predict the answer to a one-way-hash problem it set up; and this isn’t an isolated flaw, it faithfully reflects the fact that once you are trying to outwit a hostile superintelligence you’re already screwed. Plus maybe it can do the equivalent of Rowhammering you, or using a bad but “testable” prediction just once that gets somebody to let it out of the box, etcetera. Only it doesn’t do any of those things, it does something cleverer, etcetera. Eliezer thinks that once there’s a hostile superintelligence running anywhere inside the system you are almost surely screwed *in practice*, which means Eliezer thinks Paul never gets to the point of completing one primitive op of the system before the system kills him.
Paul:
I think this is a mostly correct picture of the disagreement. I *would* agree with “what is inside the black box inexorably kills you when the black box is large enough,” if we imagine humans interacting with an arbitrarily large black box. This is a key point of agreement.
I am optimistic anyway because before humanity tries to produce “aligned AI with IQ 200” we can produce “aligned AI with IQ 199.” Indeed, if we train our systems with gradient descent then the intelligence will almost necessarily increase continuously. The goal is to maintain alignment as an inductive invariant, not to abruptly inject it into an extremely powerful system. So the gap between “smartest aligned intelligence we have access to” and “AI we are currently trying to train” is always quite small. This doesn’t make the problem easy, but I do think it’s a central feature of the problem that isn’t well accounted for in your arguments for pessimism.
Buck:
My guess of Eliezer’s reply is:
If we had an IQ 199 aligned AGI, that would indeed be super handy for building the IQ 200 one. But that seems quite unlikely.
Firstly, the black box learner required to build an AI that is aligned at all (eg the first step of capability amplification), even if that learned AI is very dumb, must itself be a really powerful learner, powerful enough that it is susceptible to scary internal unaligned optimizers.
Secondly, building an IQ 200 aligned agent via imitation of a sequence of progressively smarter aligned agents seems quite unlikely to be competitive, so without unlikely amounts of coordination someone will just directly build the IQ 200 agent.
Paul:
The first step of capability amplification is a subhuman AI similar in kind to the AI we have today; so if this is someone’s objection then they ought to be able to stick their neck out today (e.g. by saying that we can’t solve the alignment problem for systems we build today, or by saying that systems we can build today definitely won’t be able to participate in amplification).
The AlphaGo Zero example really seems to take much of the wind out of the concerns about feasibility. It’s the most impressive example of RL to date, and it was literally trained as a sequence of increasingly good go players learning to imitate one another.
I think the worst concerns are daemons (which are part of the unreasonably-innocuous-sounding “reliability” in my breakdown) and the impossibility alignment-preserving amplification. Setting up imitation / making informed oversight work also seems pretty hard, but I think it’s less related to Eliezer’s concerns.
Buck:
My Eliezer-model says that systems we can build today definitely won’t be able to participate that helpfully in amplification. Amplification of human tasks seems like an extremely hard problem—our neural nets don’t seem like they’d be up to it without adding in a bunch of features like attention, memory, hierarchical planning systems, and so on. The daemons come in once you start adding in all of that, and if you don’t add in all that, your neural nets aren’t powerful enough to help you.
Skipping ahead a bit, to the next Paul rejoinder that felt relevant:
Paul
I think the most important general type of amplification is “think longer.” I think breaking a question down into pieces is a relatively easy case for amplification that can probably work with existing models. MCTS is a lot easier to get working than that.
> My Eliezer-model says that systems we can build today definitely won’t be able to participate that helpfully in amplification.
To the extent that this is actually anyone’s view, it would be great if they could be much clearer about what exactly they think can’t be done.
I personally didn’t feel confused, so I think I mostly turn that question around to you? (i.e. it seemed natural to me to use “consequentalist” in this way, and insofar as any confusion came up, specifying ‘oh, no I didn’t mean it as an ethical theory’ seems like it should address it. But, you might disagree)
I’m pretty sure “consequentialism” here wasn’t meant to mean anything to do with ethics in this case (which I acknowledge as confusing)
I think consequentialism-as-ethics means “the right/moral thing to do is to choose actions that have good consequences.”
I think consequentialism as Eliezer/John meant here is more like “the thing to do is choose actions that have the consequences you want.”
A consequentialist is something that thinks, predicts, and plans (and, if possible, acts) in such a way as to bring about particular consequences.
(I think it’s plausible that we want different words for these things, but I think this use of the word consequentialism is fairly natural, and makes sense to see “moral consequentalism” as a subset of consequentialism.)
What’s Up With Confusingly Pervasive Consequentialism?
I think this would make a good top-level post.
(I’m copying this into my original review comment so it’s easily viewable on the /reviewVoting page, but posting here since it’s a new set of thoughts)
A followup question I found myself thinking is: “What do Simulacrum levels actually add to the conversation?” We knew that sometimes people lie. We already knew about Beliefs as Attire, Belief-in-Belief, Professing and Cheering, etc. We knew that social reality was pervasive and politics make us go funny in the head. Do Simulacra add anything?
After thinking a bit, here are my answers:
First, Baudrillard-style simulacrum levels point at a particular progression/mechanism that’s interesting You have object level truth that becomes distorted, then masking, then uncoupled completely from reality. This seems relevant to some subsets of Social Reality Woes (the original example of “Vice President of Drudgery” seemed to capture a real phenomenon of a thing that happened to business titles), but it’s not obvious to me that this is usually what’s going on. Zvi argues that the Baudrillard definition is still fairly intertwined with the Lion definition, but I didn’t personally find it that persuasive. (I also don’t find it cruxy for anything other than ‘should we still call these simulacrum levels?’ so I’m not that worried about it).
Second, Simulacrum Level 4 exists. This is mabe vaguely alluded to in the original LW sequences and HPMOR, but I don’t think it’s really been spelled out. “Belief in Belief” covers Level 3, but it doesn’t really address the sort the level of cynicism that goes into someone who is dishonest about their beliefs-in-belief, and what that dishonesty feels like from the inside. This increases my desire to hear Zvi, Benquo or others weigh in about how they think about level 4 these days.
This was probably a reference to In My Culture.
I recommend reading Logan Strohl’s posts on Noticing. I don’t know how well organized they are but I suggest starting here: http://agentyduck.blogspot.com/p/noticing.html
lol j/k I’m bad at understanding timezones. Turns out it launches in ~23 hours. Happy last day of reviewing.
There’s two hours left in the Review Phase. If you got to lesswrong.com/reviewVoting, you’ll see posts sorted to the top that you highly upvoted, which you may want to review so that they can get into the final voting phase.
On one hand, AFAICT the math here is pretty fuzzy, and one could have written this post without it, instead just using the same examples to say “you should probably be less risk averse.” I think, in practice for most people, the math is a vague tribal signifier that you can trust the post, to help the advice go down.
But, I see this post in a similar reference class to Bayes’ Theorem. I think most people don’t actually need to know Bayes Theorem. They need to remember a few useful heuristics like “remember the base rates, not just the salient evidence you came across”, and “think about relative likelihood ratios between multiple hypothesis.”
Nonetheless, an important aspect of The LessWrong Project™ is building a unified worldmap, looking to understand the lawfulness that underlies our heuristics, and when our cleanest laws don’t quite apply to our messy situations, keeping an eye out for how we could someday later better integrate everything into a single unified model.
My understanding of the Kelly Criterion makes it something like “idealized Bayes Theorem but for instrumental rationality instead of epistemic.”
I’d kinda like there to be a version of this post that’s more rigorous, that attempts to take some of Jacob’s fuzzy examples and do more math to them (maybe just a couple, to fully illustrate the idea). I confess this is the sort of thing I don’t expect to really be directly helpful to anyone – probably most people read this post, absorb the general lesson of “you should be less risk averse” and move on with their day. But it’s nice to have more details available when you peer under the hood.
I haven’t had time to reread this sequence in depth, but I wanted to at least touch on how I’d evaluate it. It seems to be aiming to be both a good introductory sequence, while being a “complete and compelling case I can for why the development of AGI might pose an existential threat”.
The question is who is this sequence for, what is it’s goal, and how does it compare to other writing targeting similar demographics.
Some writing that comes to mind to compare/contrast it with includes:
Scott Alexander’s Superintelligence FAQ. This is the post I’ve found most helpful for convincing people (including myself), that yes, AI is just actually a big deal and an extinction risk. It’s 8000 words. It’s written fairly entertainingly. What I find particularly compelling here are a bunch of factual statements about recent AI advances that I hadn’t known about at the time.
Tim Urban’s Road To Superintelligence series. This is even more optimized for entertainingness. I recall it being a bit more handwavy and making some claims that were either objectionable, or at least felt more objectionable. It’s 22,000 words.
Alex Flint’s AI Risk for Epistemic Minimalists. This goes in a pretty different direction – not entertaining, and not really comprehensive either . It came to mind because it’s doing a sort-of-similar thing of “remove as many prerequisites or assumptions as possible”. (I’m not actually sure it’s that helpful, the specific assumptions it’s avoiding making don’t feel like issues I expect to come up for most people, and then it doesn’t make a very strong claim about what to do)
(I recall Scott Alexander once trying to run a pseudo-study where he had people read a randomized intro post on AI alignment, I think including his own Superintelligence FAQ and Tim Urban’s posts among others, and see how it changed people’s minds. I vaguely recall it didn’t find that big a difference between them. I’d be curious how this compared)
At a glance, AGI Safety From First Principles seems to be more complete than Alex Flint’s piece, and more serious/a-bit-academic than Scott or Tim’s writing. I assume it’s aiming for a somewhat skeptical researcher, and is meant to not only convince them the problem exists, but give them some technical hooks of how to start thinking about it. I’m curious how well it actually succeeds at that.
I meant to be referring to “I think Moral Mazes is a misleading meme that itself contributes to the problem”. Why is it misleading? Why does it contribute to the problem? What evidence or reasoning leads you to believe that?