Communications lead at MIRI. Unless otherwise indicated, my posts and comments here reflect my own views, and not necessarily my employer’s.
Rob Bensinger
Karma: 10,711
There’s also an ambiguity here about “how much can LWers discuss particular important questions or issues that are informed by stuff they learned from the meetup, or from p.b. and others’ notes?”.
I assume Sam’s views are now generally ‘in the water’ on LW, and that it would be pretty costly for the whole community to permanently self-censor about them in all non-private discussion.
(Both because the info is pretty important and discussion-worthy, and because keeping long-term secrets tends to be very cognitively taxing. And people weren’t warned in advance that they might have to permanently tag a bunch of complicated facts in their head as ‘secret’ and avoid mentioning them at all—either at the meetup, AFAICT, or before reading p.b.’s post.)
I think it makes more sense for the norm to be something like ‘don’t make a big public list of Things Sam Said At The Meetup’, ‘don’t go giving interviews to reporters about it’, and ‘don’t title your posts “Sam Altman thinks X” if your only evidence is from the meetup’, but for discussion of this stuff to otherwise be fair game going forward. (Maybe after a one-week moratorium or something, I dunno.)
Good to know!
Sam apparently said at the meetup that it was fine to share notes with friends, so I assume it’s fine to pass around private copies of p.b.‘s notes (and Gwern’s LW comments, etc.) in a Google Doc—manually adding each individual who wants to see it, rather than setting it to ‘anyone with the link can view’.
If rationalists think they’re right n% of the time and they’re not, then that’s condemnable in its own right, regardless of whether there’s a motte-and-bailey involved.
If rationalists think they’re right n% of the time and they are right n% of the time, but you aren’t allowed to be honest about that kind of thing while also being humble, the so much the worse for humility. There are good forms of humility, but the form of ‘humility’ that’s about lying or deceiving yourself about your competence level is straightforwardly bad.
Regardless, I don’t think there’s any inconsistency with being an ‘aspiring rationalist’. Even if you’re the most rational human alive, you probably still have enormous room to improve. Humans just aren’t that good at reasoning and decision-making yet.
It would be more honest and objective for the argument to stop at the very first doubtful point and leave it there with a CI for future discussion.
This seems fine until you have to make actual decisions under uncertainty. Most decisions have multiple uncertain factors going into them, and I think it’s genuinely useful to try to quantify your uncertainty in such cases (even if it’s very rough, and you feel the need to re-run the analysis in several different ways to check how robust it is, etc.).
What would you propose doing in such cases? I’d be interested to see an example of how you’d go about it.
One option might be ‘do the rationalist-ish thing when you’re forced to because it’s decision-relevant; but when you’re just analyzing an interesting intellectual puzzle for fun, don’t do the rationalist-ish thing’. My main worry there would be that only using a skill when you’re forced to gives you less practice with the skill. Sharing quantitative arguments online also makes it easy for others to express disagreement, point out errors you made, etc., which I think is important for improving and getting more calibrated (and for figuring out what’s true in the first place—but it sounds like we disagree there).
Apologies if I misunderstood what you’re recommending somewhere—an example or two of blog posts you think are making this mistake might help. Possibly I’d agree if I saw the actual cases you had in mind!
I’m not sure what you mean by “has all the answers”. I could imagine a rationalist thinking they’re n standard deviations above the average college-educated human on some measure of ‘has accurate beliefs about difficult topics’, and you disagreeing and think they’re average, or thinking their advantage is smaller. But that just seems like an ordinary disagreement to me, rather than a motte-and-bailey.
It seems at odds with rationalist ideas to assume you’re unusually knowledgeable, but not to conclude you’re unusually knowledgeable. ‘I’m average’ is just as much a claim about the world as ‘I’m exceptional’, and requires you to stick your neck out just as much—if you underestimate your ability, you’re making just as much of an epistemic mistake as if you overestimate your ability.
Thinking about it more: I can imagine a group that tries to become unusually good at learning true things in a pretty domain-general way, so they call themselves the Learnies, or the Discoverers, or the Trutheteers.
If a group like that succeeds in advancing its art, then it should end up using that art to discover at least some truths about the world at large that aren’t widely known. This should be a reliable consequence of ‘getting good at discovering truths’. And in many such worlds, it won’t be trivial for the group to then convince the entire rest of the world to believe the things they learned overnight. So five years might pass and the Learnies are still the main group that is aware of, say, the medical usefulness of some Penicillium molds (because they and/or the world they’re in is dysfunctional in some way).
It seems natural to me that the Learnies’ accumulated learnings get mentally associated with the group, even though penicillin doesn’t have any special connection to the art of learning itself. So I don’t think there’s anything odd about rationalists being associated with ‘knowledge we accumulated in the process of applying rationality techniques’, or about my wanting a way to figure out to what extent someone at a party has acquired that knowledge. I think this example does illustrate a few issues, though:
First, obviously, it’s important to be able to distinguish ‘stuff the Learnies learned that’s part of the art of learning’ from ‘other stuff the Learnies learned’. Possibly it would help here to have more names for the clusters of nonstandard beliefs rationalists tend to have, so I’m somewhat less tempted to think ‘is this person familiar with the rationalist-ish content on ems?’ vs. ‘is this person familiar with the techno-reductionist content on ems?’ or whatever.
Second, I’ve been describing the stuff above as “knowledge”. But what if there’s a dispute about whether the Learnies’ worldly discoveries are true? In that case, maybe it’s logically rude to call yourself ‘Learnies’, because (a) it maybe rhetorically tilts the debate in your favor, and (b) it’s just not very informative. (Example: A world where pro-penicillin people call themselves Penicillists and anti-penicillin people call themselves Anti-Penicillists is better than a world where pro-penicillin people call themselves Accuracyists and anti-penicillin people call themselves Realityists.)
Considerations like this update me toward using the term ‘aspiring rationalist’ more, because it hopefully makes it tilts debate less toward the presupposition that ‘whoever identifies themselves as a truth-seeker’ is correct about whatever thing they think their rationality techniques have helped them learn.
(Though I think ‘aspiring rationalist’ still tilts the debate some. But, thinking about it again, maybe I actually prefer the world where more people make one flavor or another of ‘truth-seeking’ part of their identity? It might be at least somewhat easier to call people out on bullshit and make them reflect and update, if it were more common for people to take pride in their truth-seeking and knowledgeableness, however unwarranted.)
Then there’s the question of whether the Learnies’ commitment to the art of learning obliges them to never Glomarize or stay-out-of-the-fray about politically controversial things they learn. I take it that something along these lines is your main criticism of rationalists calling themselves ‘rationalists’, but I’m not sure exactly what norm you’re advocating. Currently I don’t find this compelling—like, I do think it’s epistemically risky for groups to selectively avoid talking about important politicized topics, and I think it’s important to try to find ways to counter resultant epistemic distortions in those cases (and to seriously consider whether staying-out-of-the-fray is just a dumb strategy). But I guess I just disagree with some of the examples you’ve provided, and agree with others but don’t think they’re as serious or deeply-epistemic-corrupting as you do? Mostly, I just don’t think I’ve read enough of the stuff you’ve written to understand your full argument; but I can at least give my current epistemic state.
Yeah, this already bothered me some, but your way of putting it here makes it bother me more.
I think the motte/bailey often runs in the other direction, though, for modesty-ish reasons: there’s a temptation to redefine ‘rationalist’ as a social concept, because it looks more humble to say ‘I’m in social circle X’ than to say ‘I’m part of important project X’ or ‘I’m a specialist in X’, when you aren’t doing X-stuff as part of a mainstream authority like academia.
I think there are two concepts I tend to want labels for, which I sometimes use ‘rationalist’ to refer to (though I hope I’m not switching between these in a deceptive/manipulative way!):
‘One who successfully develops and/or applies the methods of getting systematically better at mapping reality (and, optionally, steering reality) from inside a human brain.’
‘One who is highly acquainted with the kinds of ideas in the Sequences, and with related ideas that have been a major topic on LW (e.g., the availability heuristic and reductionism and conservation of expected evidence, but also ems, tractability/importance/neglectedness, updateless decision theory, ugh fields, ideas from Alicorn’s Twilight fanfic...).’
I think the latter concept is pretty useful and natural, though I could maybe be convinced that ‘rationalist’ is a bad name for it.
I think it’s mainly a memetic or cultural concept in my mind, like ‘postmodernist’ or ‘Chicago economist’ or ‘the kind of person who’s been to a coding boot camp’: shibboleths are a big deal for who my brain tags this way, but mere social adjacency isn’t. It’s closer to ‘how much background knowledge can I assume if we start talking about simulation shutdown at a party?‘, rather than ‘how much does e.g. Scott Alexander like this person, spend time with them, etc.?’.
(Disclaimer: I haven’t read DiAngelo’s book, and I know very little about her as a person. I’m curious about the question ‘what’s the nearest reasonable version of this strategy, and what would a conversation look like with a reasonable proponent of that strategy?’.
I can’t speak to whether any of that resembles how an actual conversation with DiAngelo would go, and I don’t mean to vouch for DiAngelo’s overall epistemics in any of the following. I expect the epistemics are pretty bad. But my experience has been that the flip side of ‘motte and bailey is popular’ and ‘pop culture makes good ideas memetically evolve into terrible ideas’ is ‘there’s often a much more reasonable version of a thought pattern that’s conceptually close to the unreasonable version’.)
The ‘everything you do except agree with me shows how wrong you are’ thing is really scary, because it can create a situation where confirmation bias has complete dominion and ~no evidence can allow you to update away from your original assumption (it reminds me of outgroup-Bingo).
But it also reminds me a bit of New Atheists’ frustrations with arguing with religious people. New Atheists ended up generating long lists of names for various wrong patterns of reasoning and argument, in part because they were exasperated at how religionists’ confirmation bias caused them to just keep generating more and more new Objectively Terrible rationalizations for the same old falsified claims.
The human mind is a bottomless well of rationalization, when it wants to be. Trying to refute every single object-level argument is a hopeless task (especially when you’re arguing against a religionist who’s smarter than you, or has spent more time studying the topic). But you might hope to categorize a certain kind of mistake, and/or recognize a root cause underlying many different specific mistakes. And this might indeed start looking like a Bingo card, after the fiftieth argument by a religionist gets shot down with yet another canned Fallacy Name; but that doesn’t mean the New Atheist is wrong.
So if DiAngelo is convinced enough in her view of X, and is frustrated by what appears to her to be an endless series of easily-refuted rationalizations for not-X, then I can understand why she might want to have Extremely General Counter-Arguments she can use to defeat a wide variety of objections in one fell swoop.
That still doesn’t address the problem of ‘what if you’re really sure X is true, but in fact X is false? how do you ever escape from believing X?’.
I’m not sure how to address that problem in general, other than:
Being generally epistemically cautious and rigorous, so you end up highly confident of fewer false things.
Being cautious about the strategies ‘round an opponent’s argument off to a general category and refute the category’ and ‘identify the root cause of an opponent’s argument and show the cause is epistemically suspect’; use them relatively sparingly, and put unusual effort into testing your hypotheses about such categories and causes.
Trying to stick to the object level where possible, and having some conversations where the ground rules are ‘we’re not going to use those Extremely General Counter-Arguments’.
Doing conversation in a way that causes people to rationalize and dissemble less / causes them to be more reflective about what’s really in their head. (Circling?)
When I imagine a DiAngelo-Kennaway conversation going really well, I think I imagine a version of Kennaway who clearly signals in the conversation that he’s open to criticism and admitting fault when it comes to his judgment/impartiality/character/virtue, so DiAngelo has Bayesian evidence that she’s not engaging with the equivalent of an Imperviously Rationalizing Religionist. Then DiAngelo acknowledges the Epistemic Trap problem, and explicitly notes examples in the conversation where she updated both upward and downward in her subjective estimate of ‘how racially defensive/biased Kennaway is’ (or something like that).
Then the conversation can perhaps proceed productively from there, even though both parties think the other side is totally wrong and have various theories about why they’re so wrong.
That’s not my ideal version of how this kind of conversation goes; it’s just the thing my brain tags ‘least unlikely’ when I condition on the conversation being substantive, honest, friendly, and productive.
That seems better to me, though I don’t know what the technical options are. I didn’t realize this was being shared so clearly against Sam’s wishes; seems pretty bad.
Some options I like that come to mind include:
p.b. deleting the post (ideally followed up by someone posting a short thing on LW summarizing what went down in this thread, so we have a record of that much).
p.b. blanking the notes in the post, so it can only be viewed in History.
p.b. doing one of the above, plus making a private Google Doc that contains the notes (and sharing it with people who request access).
(Obviously p.b. can do what they want, assuming the moderators don’t think an intervention makes sense here. But the above is what I’d request, if it’s true that Sam wanted notes to not be shared publicly.)
(Also, obviously, the cat’s fairly out of the bag in terms of this post’s contents spreading elsewhere on the Internet. But LW is still a pretty high-traffic site, so we can still do stuff to not add fuel to the fire. And there’s value in clearly signaling our moral position here regardless.)
Thanks for making various edits to the post, pb.
I’d suggest making such edits in a way that makes the original version stay visible (e.g., as struck text). As is, the discussion is confusing because we’ve all read different versions. (I told Oliver a part I objected to—“Merging via CBI most likely path to a good outcome”—and he had seen a later version and hadn’t been aware of what the original said.)
Clearly and very-visibly noting ‘this was false/misleading’ is also pretty important IMO, because I think it often takes unusual effort on the part of individuals and groups to properly un-update after hearing a falsehood. If we forget the falsehood, it can often just keep spreading through the social network side-by-side with the true, fixed info.
I didn’t take notes, and I don’t want to try to correct the record from memory and plausibly make things worse.
Maybe some attendees could make a private Google Doc that tries to be a little more precise about the original claims/context/vibe, then share it after enough attendees have glanced over it to make you confident in the summary?
I don’t expect this would be a huge night-and-day difference from the OP, but it may matter a fair bit for a few of the points. And part of what bothers me right now is that I don’t know which parts to be more skeptical of.
If it’s true that p.b. said some pretty false things about what was stated at the meet-up (per Lanrian), and that James Miller did the same in the comments (per Gwern), then I think the OP (and overall discussion) probably shouldn’t be on the LW frontpage, at least until there’s been more fact-checking and insertion of corrections.
I think some of the info in the OP (if true!) is pretty valuable to discuss, but I really hate the idea of LW spreading false gossip and rumors about someone’s views. (Especially when it would have been relatively easy to check whether others at the meetup had the same recollection before posting.)
Timestamp / excerpt? As described, I think you need to be missing a lot of facts (or be reasoning very badly) in order to reach a conclusion like that.
This post is excellent, and should be part of the ‘Welcome to LessWrong’ introductory content IMO. :)
OK, thanks for the clarifications!
Still, I always had the impression that this line of work focused more on how to build a perfectly rational AGI than on building an aligned one. Can you explain me why that’s inaccurate?
I don’t know what you mean by “perfectly rational AGI”. (Perfect rationality isn’t achievable, rationality-in-general is convergently instrumental, and rationality is insufficient for getting good outcomes. So why would that be the goal?)
I think of the basic case for HRAD this way:
We seem to be pretty confused about a lot of aspects of optimization, reasoning, decision-making, etc. (Embedded Agency is talking about more or less the same set of questions as HRAD, just with subsystem alignment added to the mix.)
If we were less confused, it might be easier to steer toward approaches to AGI that make it easier to do alignment work like ‘understand what cognitive work the system is doing internally’, ‘ensure that none of the system’s compute is being used to solve problems we don’t understand / didn’t intend’, ‘ensure that the amount of quality-adjusted thinking the system is putting into the task at hand is staying within some bound’, etc.
These approaches won’t look like decision theory, but being confused about basic ground-floor things like decision theory is a sign that you’re likely not in an epistemic position to efficiently find such approaches, much like being confused about how/whether chess is computable is a sign that you’re not in a position to efficiently steer toward good chess AI designs.
Maybe what I want is a two-dimensional “prosaic AI vs. novel AI” and “whiteboards vs. code”. Then I can more clearly say that I’m pretty far toward ‘novel AI’ on one dimension (though not as far as I was in 2015), separate from whether I currently think the bigger bottlenecks (now or in the future) are more whiteboard-ish problems vs. more code-ish problems.
Cool, that makes sense!
I abused the hyperbole in that case. What I was pointing out is the impression that old-school MIRI (a lot of the HRAD work) thinks that solving the alignment problem requires deconfusing every related philosophical problem in terms of maths, and then implementing that. Such a view doesn’t seem shared by many in the community for a couple of reasons:
I’m still not totally clear here about which parts were “hyperbole” vs. endorsed. You say that people’s “impression” was that MIRI wanted to deconfuse “every related philosophical problem”, which suggests to me that you think there’s some gap between the impression and reality. But then you say “such a view doesn’t seem shared by many in the community” (as though the “impression” is an actual past-MIRI-view others rejected, rather than a misunderstanding).
HRAD has always been about deconfusion (though I agree we did a terrible job of articulating this), not about trying to solve all of philosophy or “write down a perfectly aligned AGI from scratch”. The spirit wasn’t ‘we should dutifully work on these problems because they’re Important-sounding and Philosophical’; from my perspective, it was more like ‘we tried to write down a sketch of how to align an AGI, and immediately these dumb issues with self-reference and counterfactuals and stuff cropped up, so we tried to get those out of the way fast so we could go back to sketching how to aim an AGI at intended targets’. As Eliezer put it,
It was a dumb kind of obstacle to run into—or at least it seemed that way at that time. It seemed like if you could get a textbook from 200 years later, there would be one line of the textbook telling you how to get past that.
From my perspective, the biggest reason MIRI started diversifying approaches away from our traditional focus was shortening timelines, where we still felt that “conceptual” progress was crucial, and still felt that marginal progress on the Agent Foundations directions would be useful; but we now assigned more probability to ‘there may not be enough time to finish the core AF stuff’, enough to want to put a lot of time into other problems too.
Actually, I’m not sure how to categorize MIRI’s work using your conceptual vs. applied division. I’d normally assume “conceptual”, because our work is so far away from prosaic alignment; but you also characterize applied alignment research as being about “experimentally testing these ideas [from conceptual alignment]”, which sounds like the 2017-initiated lines of research we described in our 2018 update. If someone is running software experiments to test ideas about “Seeking entirely new low-level foundations for optimization” outside the current ML paradigm, where does that fall?
If AGI turns out to be prosaic AGI (which sounds like one of the best bet to make now), then what matters is aligning neural nets, not finding a way of write down a perfectly aligned AGI from scratch
Prosaic AGI alignment and “write down a perfectly aligned AGI from scratch” both seem super doomed to me, compared to approaches that are neither prosaic nor perfectly-neat-and-tidy. Where does research like that fall?
Nowadays, the vast majority of the field disagree that there’s any hope of formalizing all of philosophy and then just implementing that to get an aligned AGI.
What do you mean by “formalizing all of philosophy”? I don’t see ‘From Philosophy to Math to Engineering’ as arguing that we should turn all of philosophy into math (and I don’t even see the relevance of this to Friendly AI). It’s just claiming that FAI research begins with fuzzy informal ideas/puzzles/goals (like the sort you might see philosophers debate), then tries to move in more formal directions.
I imagine part of Luke’s point in writing the post was to push back against the temptation to see formal and informal approaches as opposed (‘MIRI does informal stuff, so it must not like formalisms’), and to push back against the idea that analytic philosophers ‘own’ whatever topics they happen to have historically discussed.
Conceptual alignment research isn’t just turning philosophy into mathematics. This is a failure mode I warned against recently: what matters is deconfusion, not formalization.
Pearl’s causality (the main example of “turning philosophy into mathematics” Luke uses) was an example of achieving deconfusion about causality, not an example of ‘merely formalizing’ something. I agree that calling this deconfusion is a clearer way of pointing at the thing, though!
I haven’t read Brian Tomasik’s thoughts on this, so let me know if you think I’m misunderstanding him / should read more.
The hard problem of consciousness at least gives us a prima facie reason to consider panpsychism. (Though I think this ultimately falls apart when we consider ‘we couldn’t know about the hard problem of consciousness if non-interactionist panpsychism were true; and interactionist panpsychism would mean new, detectable physics’.)
If we deny the hard problem, then I don’t see any reason to give panpsychism any consideration in the first place. We could distinguish two panpsychist views here: ‘trivial’ (doesn’t have any practical implications, just amounts to defining ‘consciousness’ so broadly as to include anything and everything); and ‘nontrivial’ (has practical implications, or at least the potential for such; e.g., perhaps the revelation that panpsychism is true should cause us to treat electrons as moral patients, with their own rights and/or their own welfare).
But I see no reason whatsoever to think that electrons are moral patients, or that electrons have any other nontrivial mental property. The mere fact that we don’t fully understand how human brains work is not a reason to ask whether there’s some new undiscovered feature of particles times smaller than a human brain that explains the comically larger macro-process—any more than limitations in our understanding of stomachs would be a reason to ask whether individual electrons have some hidden digestive properties.
(If others disagree, I’m probably easy to convince on this.)