I’m an admin of this site; I work full-time on trying to help people on LessWrong refine the art of human rationality. (Longer bio.)
I generally feel more hopeful about a situation when I understand it better.
Ben Pace(Ben)
Karma: 30,512
Someone in the community told me that for me to think AGI probably won’t be developed soon, I must think I’m better at meta-rationality than Eliezer Yudkowsky, a massive claim of my own specialness.
Just zooming in on this, which stood out to me personally as a particular thing I’m really tired of.
If you’re not disagreeing with people about important things then you’re not thinking. There are many options for how to negotiate a significant disagreement with a colleague, including spending lots of time arguing about it, finding a compromise action, or stopping collaborating with the person (if it’s a severe disagreement, which often it can be). But telling someone that by disagreeing they’re claiming to be ‘better’ than another person in some way always feels to me like an attempt to ‘control’ the speech and behavior of the person you’re talking to, and I’m against it.
It happens a lot. I recently overheard someone (who I’d not met before) telling Eliezer Yudkowsky that he’s not allowed to have extreme beliefs about AGI outcomes. I don’t recall the specific claim, just that EY’s probability mass for the claim was in the 95-99% range. The person argued that because EY disagrees with some other thoughtful people on that question, he shouldn’t have such confidence.
(At the time, I noticed I didn’t have to be around or listen to that person and just wandered away. Poor Eliezer stayed and tried to give a thoughtful explanation for why the argument seemed bad.)
Perhaps more important to my subsequent decisions, the AI timelines shortening triggered an acceleration of social dynamics.
I noticed this too. I thought a bunch of people were affected by it in a sort of herd behavior way (not focused so much on MIRI/CFAR, I’m talking more broadly in the rationality/EA communities). I do think key parts of the arguments about how to think about timelines and takeoff are accurate (e.g. 1, 2), but I feel like many people weren’t making decisions because of reasons; instead they noticed their ‘leaders’ were acting scared and then they also acted scared, like a herd.
In both the Leverage situation and the AI timelines situation, I felt like nobody involved was really appreciating how much fuckery the information siloing was going to cause (and did cause) to the way the individuals in the ecosystem made decisions.
This was one of the main motivations behind my choice of example in the opening section of my 3.5 yr old post A Sketch of Good Communication btw (a small thing but still meant to openly disagree with the seeming consensus that timelines determined everything). And then later I wrote about the social dynamics a bunch more 2yrs ago when trying to expand on someone else’s question on the topic.
I feel like one of the most valuable things we have on LessWrong is a broad, shared epistemic framework, ideas with which we can take steps through concept-space together and reach important conclusions more efficiently than other intellectual spheres e.g. ideas about decision theory, ideas about overcoming coordination problems, etc. I believe all of the founding staff of CFAR had read the sequences and were versed in things like what it means to ask where you got your bits of evidence from, that correctly updating on the evidence has a formal meaning, and had absorbed a model of Eliezer’s law-based approach to reasoning about your mind and the world.
In recent years, when I’ve been at CFAR events, I generally feel like at least 25% of attendees probably haven’t read The Sequences, aren’t part of this shared epistemic framework, and don’t have an understanding of that law-based approach, and that they don’t have a felt need to cache out their models of the world into explicit reasoning and communicable models that others can build on. I also have felt this way increasingly about CFAR staff over the years (e.g. it’s not clear to me whether all current CFAR staff have read The Sequences). And to be clear, I think if you don’t have a shared epistemic framework, you often just can’t talk to each other very well about things that aren’t highly empirical, certainly not at the scale of more than like 10-20 people.
So I’ve been pretty confused by why Anna and other staff haven’t seemed to think this is very important when designing the intellectual environment at CFAR events. I’m interested to know how you think about this?
I certainly think a lot of valuable introspection and modelling work still happens at CFAR events, I know I personally find it useful, and I think that e.g. CFAR has done a good job in stealing useful things from the circling people (I wrote about my positive experiences circling here). But my sense for a number of the attendees is that even if they keep introspecting and finding out valuable things about themselves, 5 years from now they will not have anything to add to our collective knowledge-base (e.g. by writing a LW sequence that LWers can understand and get value from), even to a LW audience who considers all bayesian evidence admissible even if it’s weird or unusual, because they were never trying to think in a way that could be communicated in that fashion. The Gwerns and the Wei Dais and the Scott Alexanders of the world won’t have learned anything from CFAR’s exploration.
As an example of this, Val (who was a cofounder but doesn’t work at CFAR any more) seemed genuinely confused when Oli asked for third-party verifiable evidence for the success of Val’s ideas about introspection. Oli explained that there was a lemons problem (i.e. information asymmetry) when Val claimed that a mental technique has changed his life radically, when all of the evidence he offers is of the kind “I feel so much better” and “my relationships have massively improved” and so on. (See Scott’s Review of All Therapy Books for more of what I mean here, though I think this is a pretty standard idea.) He seemed genuinely confused why Oli was asking for third-party verifiable evidence, and seemed genuinely surprised that claims like “This last September, I experienced enlightenment. I mean to share this as a simple fact to set context” would be met with a straight “I don’t believe you.” This was really worrying to me, and it’s always been surprising to me that this part of him fit naturally into CFAR’s environment and that CFAR’s natural antibodies weren’t kicking against it hard.
To be clear, I think several of Val’s posts in that sequence were pretty great (e.g. The Intelligent Social Web is up for the 2018 LW review, and you can see Jacob Falkovich’s review on how the post changed his life), and I’ve personally had some very valuable experiences with Val at CFAR events, but I expect, had he continued in this vein at CFAR, that over time Val would just stop being able to communicate with LWers, and drift into his own closed epistemic bubble, and to a substantial degree pull CFAR with him. I feel similarly about many attendees at CFAR events, although fewer since Val left. I never talked to Pete Michaud very much, and while I think he seemed quite emotionally mature (I mean that sincerely) he seemed primarily interested in things to do with authentic relating and circling, and again I didn’t get many signs that he understood why building explicit models or a communal record of insights and ideas was important, and because of this it was really weird to me that he was executive director for a few years.
To put it another way, I feel like CFAR has in some ways given up on the goals of science, and moved toward the goals of a private business, whereby you do some really valuable things yourself when you’re around, and create a lot of value, but all the knowledge you gain about about building a company, about your market, about markets in general, and more, isn’t very communicable, and isn’t passed on in the public record for other people to build on (e.g. see the difference between how all scientists are in a race to be first to add their ideas to the public domain, whereas Apple primarily makes everyone sign NDAs and not let out any information other than releasing their actual products, and I expect Apple will take most of their insights to the grave).
I’ve found this week’s progress pretty upsetting.
I’m fairly scared that if “the EA community” attempted to pivot to running the fire-alarm, that nothing real would happen, we’d spend our chips, and we’d end up in some complicated plot that had no real chance of working whilst maybe giving up our ability to think carefully any more. Like, there’s no plan stated in the post. If someone has a plan that has a chance of doing any particular thing that’d be more interesting.
I spend various amounts of time in close proximity to a bunch of parts of “EA leadership”, and if you convince me that a strategy will work I could advocate for it.
(Also happy to receive DMs if you want to keep specifics private.)
I think this letter is not robust enough to people submitting false names. Back when Jacob and I put together DontDoxScottAlexander.com we included this section, and I would recommend doing something pretty similar:
I think someone checking some of these emails would slow down high-profile signatories by 6-48 hours, but sustain trust that the names are all real.
I’m willing to help out if those running it would like, feel free to PM me.
I did hear your side for 3 hours and you changed my mind very little and admitted to a bunch of the dynamics (“our intention wasn’t just to have employees, but also to have members of our family unit”) and you said my summary was pretty good. You mostly laughed at every single accusation I brought up and IMO took nothing morally seriously and the only ex ante mistake you admitted to was “not firing Alice earlier”. You didn’t seem to understand the gravity of my accusations, or at least had no space for honestly considering that you’d seriously hurt and intimidated some people.
I think I would have been much more sympathetic to you if you had told me that you’d been actively letting people know about how terrible an experience your former employees had, and had encouraged people to speak with them, and if you at literally any point had explicitly considered the notion that you were morally culpable for their experiences.
Prosaic Alignment is currently more important to work on than Agent Foundations work.
It is occasionally said to me,
“Have you considered meditation and buddhism? Enlightenment is really powerful.”
This feels similar to saying
“Have you considered giving up a massive resource—one of your scarce slots for a life-long habit, a daily time-sink with week-long retreats—to Buddhist meditation? It supposedly makes you feel funny at the end, as though you’ve had a major epistemological insight (but you aren’t able to produce corresponding output as a result).”
Given the amount of people offering me something like the above, my background skepticism is very high.
The thing that will most cause me to believe that kenshŌ is valuable for epistemology, will be some examples of things you have managed to do better as a result. If, for example, you wrote sequence of recognisably useful insights unrelated to enlightenment (example), and then afterwards told me that it was due to your having felt enlightenment, I’d consider that interesting evidence. But I do predict that I find your subsequent post not much evidence either way.
I will mention that I have some notion of a thing you might be pointing towards: I’ve experienced ontological updates—updates to the feelings that make up the building blocks of how I think. And this does require a certain ‘get out of your head / no seriously look up’ motion, which you might be pointing too. Also, Buddhism is an old old religion and there’s this notion that religions do capture truths of human nature, which is why they’re able to be so widespread yet profound. So maybe you’ll manage to capture that.
But sometimes when someone has a hard time explaining something, it’s because they’re just confused. So many people are trying to communicate supposed fundamental truths, and aren’t showing much for it. As I say, my background skepticism is high.
I vouch that this person is both a LW user who has written IMO some good posts and a member of in-person rationalist/longtermist/EA communities who is in good standing.
Edit: This comment is not meant as an endorsement (nor is this a disendorsement) of the content of the post. I generally support LWers and rationalists being able to post pseudonymously and have their identity as longstanding members of the various communities verified.
The most important traits of the new humans are that… they prize rationality under all circumstances—to be accepted by them you have to retain clear thinking and problem-solving capability even when you’re stressed, hungry, tired, cold, or in combat
Interestingly, as a LessWronger, I don’t think of myself in quite this way. I think there’s a key skill a rationalist should attain, which is knowing in which environments you will fail to be rational, and avoiding those environments. Knowing your limits, and using that knowledge when making plans.
One that I’ve dealt with, that I think is pertinent for a lot of people, is being aware of how social media can destroy my attention and leave me feeling quite socially self-conscious. Bringing them into my environment damages my ability to think.
On the one hand, becoming able to think clearly and make good decisions while using social media is valuable and for many necessary. Here are some of the ways I try to do that, in the style of the Homo Novis:
I notice when I’m being encouraged to use the wrong concepts (e.g. PR rather than honor) or believe deeply bad theories of ethics (e.g. the copenhagen theory of ethics)
I keep my identity small / use my identity carefully
I build a better model of my social environment, how knowledge propagates, and the narrative forces pushing me with (especially the forces of blandness) so I can see threats coming
But one of the important tools I have is avoiding being in those environments. I respond with very strict rules around Sabbath/Rest Days so I can clear my head. I also don’t carry a phone in general, and install content blockers on my laptop. I think these approaches are more like “avoiding situations where I cannot think clearly” than “learning to think clearly in difficult situations”.
There’s a balance between the two strategies. “Learn to think clearly in more environments” and “shape your environment to help you think clearly / not hinder your ability to think clearly”. In response to a situation where I can’t think clearly, sometimes I pick the one, and sometimes the other.
All that said, Gulf is totally added to my reading list. I read both The Moon is a Harsh Mistress and Stranger in a Strange Land for the first time this year and that was a thrill.
Comedian Simon Munnery:
Many are willing to suffer for their art; few are willing to learn how to draw.
Academia is sufficiently dysfunctional that if you want to make a great scientific discovery you should basically do it outside of academia.
Brief update: I am still in the process of reading this. At this point I have given the post itself a once-over, and begun to read it more slowly (and looking through the appendices as they’re linked).
I think any and all primary sources that Kat provides are good (such as the page of records of transactions). I am also grateful that they have not deanonymized Alice and Chloe.
I plan to compare the things that this post says directly against specific claims in mine, and acknowledge anything where I was factually inaccurate. I also plan to do a pass where I figure out which claims of mine this post responds to and which it doesn’t, and I want to reflect on the new info that’s been entered into evidence and how it relates to the overall picture.
It probably goes without saying that I (and everyone reading) want to believe true things and not false things about this situation. If I made inaccurate statements I would like to know that and correct them.
As I wrote in my follow-up post, I am not intending to continue spear-heading an investigation into Nonlinear. However this post makes some accusations of wrongdoing on my part, which I intend to respond to, and of course for that it is relevant whether the things I said are actually true.
I hope to write a response sometime this week, but I am not committing to any deadlines.
Not sure if it’s worth mentioning, but I hope that people reading this are aware of what Kat writes at the bottom of the appendices:
A quick note on how we use quotation marks: we sometimes use them for direct quotes and sometimes use them to paraphrase. If you want to find out if they’re a direct quote, just ctrl-f in the original post and see if it is or not.
Many of the things that are quotes next to my name are not things I said and not things that I would endorse, and I believe the same is true of many sentences in quotation marks attributed to Alice/Chloe.
Answered all I could except the digit one because of no access to scanner. Looking forward to the results!
I changed my mind about conflict/mistake theory recently, after thinking about Scott’s comments on Zvi’s post. I previously thought that people were either conflict theorists or mistake theorists. But I now do not use it to label people, but instead to label individual theories.
To point to a very public example, I don’t think Sam Harris is a conflict theorist or a mistake theorist, but instead uses different theories to explain different disagreements. I think Sam Harris views any disagreements with people like Stephen Pinker or Daniel Dennett as primarily them making reasoning mistakes, or otherwise failing to notice strong arguments against their position. And I think that Sam Harris views his disagreements with people like <quickly googles Sam Harris controversies> Glenn Greenwald and Ezra Klein as primarily them attacking him for pushing different goals to their tribes.
I previously felt some not-insubstantial pull to pick sides in the conflict vs mistake theorist tribes, but I don’t actually think this is a helpful way of talking, not least because I think that sometimes I will build a mistake theory for why a project failed, and sometimes I will build a conflict theory.
To push back on this part:
Models like “arguments are soldiers” or “our enemies are evil” are the core of Yudkowsky’s original argument for viewing politics as a mind-killer. But these sort of models are essentially synonymous with conflict theory; if we could somehow have a tribalistic or political discussion without those conflict-theoretic elements, I’d expect it wouldn’t be so mindkiller-ish.
“Arguments are soldiers” and “our enemies are evil” are not imaginary phenomena, they exist and people use such ideas regularly, and it’s important that I don’t prevent myself from describing reality accurately when this happens. I should be able to use a conflict theory.
I have a model of a common type of disagreement where people get angry at someone walking in with a mistake theory that goes like this: Alice has some power over Bob, and kinda self-deceives themselves into a situation where it’s right for them to take resources from Bob, and as Bob is getting angry at Alice and tries to form a small political force to punish Alice, then Charlie comes along and is like “No you don’t understand, Alice just made an error of reasoning and if I explain this to them they won’t make that mistake again!” and Bob gets really angry at Charlie and thinks they’re maybe trying to secretly help Alice or else are strikingly oblivious / conflict averse to an unhealthy degree. (Note this is a mistake theory about the disagreement between Bob and Charlie, and a conflict theory about the disagreement between Bob and Alice. And Charlie is wrong to use a mistake theory.)
I think the reason I’m tempted to split mistake and conflict into tribes, is because I do know people that largely fit into one or the other. I knew people at school who always viewed interpersonal conflict as emanating from tribal self-interest, and would view my attempt to show a solution that didn’t require someone being at fault as me trying to make them submit to some kinda weird technicality, and got justifiably irritated. I also know people who are very conflict averse but also have an understanding of the complexity of reality, and so always assume it is merely a principal-agent problem or information flow problem, as opposed to going “Yeah, Alice is just acting out of self-interest here, we need to let her know that’s not okay, and let’s not obfuscate this unnecessarily.” But I think the goal is to have one’s beliefs correspond to reality—to use a conflict theory when that’s true, a mistake theory when that’s true, and not pre-commit to one side or the other regardless of how reality actually is.
I do think that conflict theories are often pretty derailing to bring up when trying to have a meaningful 1-1 public debate, and that it’s good to think carefully about specific norms for how to do such a thing. I do think that straight-up banning them is likely the wrong move though. Well, I think that there are many places where they have no place, such as a math journal. However the mathematical community will need a place to be able to discuss internal politics + norm-violations where these can be raised.
Also, Oliver Habryka was accused of spending ill-gotten karma on a lavish lifestyle. In response, Habryka says he was not spending other people’s internet points on his extravagant lifestyle and in fact he sleeps on a coffee table in the office because he’s always working, and shared a picture to prove it.
Solid contribution, thank you.
I think that losing your faith in civilization adequacy does feel more like a deconversion experience. All your safety nets are falling, and I cannot promise you that we’ll replace them all. The power that ‘made things okay’ is gone from the world.
I experienced a bunch of those disorientation patterns during my university years. For example:
I would only spend time with people who cared about x-risk as well, because other people seemed unimportant and dull, and I thought I wouldn’t want to be close to them in the long run. I would choose to spend time with people even if I didn’t connect with very much, hoping that opportunities to do useful things would show up (most of the time they didn’t). And yet I wasn’t able to hang out with these people. I went through maybe a 6 month period where when I met up with someone, the first thing I’d do was list out like 10-15 topics we could discuss, and try to figure out which were the most useful to talk about and in what order we should talk. I definitely also turned many of these people off hanging out with me because it was so taxing. I was confused about this at the time. I though I was not doing it well enough or something, because I wasn’t providing enough value to them such that they were clearly having a good time.
I became very uninterested in talking with people whose words didn’t cache out into a gears level model of the situation based in things I could independently confirm or understand. I went through a long period of not being able to talk to my mum about politics at all. She’s very opinionated and has a lot of tribal feelings and affiliations, and seemed to me to not be thinking about it in the way I wanted to think about it, which was a more first-principles fashion. Nowadays I find it interesting to put engage with how she sees the world, argue with it, feel what she feels. It’s not the “truth” that I wanted, I can’t take in the explicit content of her words and just input them into my beliefs, but this isn’t the only way to learn from her. She has a substantive perspective on human coordination, that’s tied up with important parts of her character and life story, that a lot of people share.
Relatedly, I went through a period of not being able to engage with aphorisms or short phrases that sounded insightful. Now I feel more trusting of my taste in what things mean and which things to take with me.
I generally wasn’t able to connect with my family about what I cared about in life / in the big picture. I’d always try to be open and honest, and so I’d say something like “I think the world might end and I should do something about it” and they’d think that sounded mad and just ignore it. My Dad would talk about how he just cares that I’m happy. Nowadays I realise we have a lot of shared reference points for people who do things, not because they make you happy or because they help you be socially secure, but because they’re right, because they’re meaningful and fulfilling, and because it feels like it’s your purpose. And they get that, and they know they make decisions like that, and they understand me when I talk about my decisions through that frame.
I remember on my 20th birthday, I had 10 of my friends round and gave a half-hour power-point presentation on my life plan. Their feedback wasn’t that useful, but I realised like a week later, that the talk only contained info about how to evaluate whether a plan was good, and not how to generate plans to be evaluated. I’d just picked the one thing that people talked about that sounded okay under my evaluation process (publishing papers in ML, which was a terrible choice for me, I interacted very badly with academia). It took me a week to notice that I’d not said how to come up with plans. I then realised that I’d been thinking in a very narrow and evaluative way, and not been open to exploring interesting ideas before I could evaluate whether they worked.
I should say, these shifts have not been anything like an unmitigated failure. I think the whole process was totally worth it, and not just because they caused me to be more socially connected to x-risk people, or because they were worth it in some pascal’s mugging kind of way. Like, riffing off that last example, the birthday party was followed by us doing a bunch of other things I really liked—my friends and I read a bunch of dialogues from GEB after that (the voices people did were very funny) and ate cake, and I remember it fondly. The whole event was slightly outside my comfort zone, but everyone had a great time, and it was also in the general pattern of me trying to more explicitly optimise for what I cared about. A bunch of the stuff above has lead me to form the strongest friendships I had, much stronger than I think I expected I could have. And many other things I won’t detail here.
Overall the effects on me personally, on my general fulfilment and happiness and connection to people I care about, has been strongly positive, and I’m glad about this. I take more small social risks, and they pay off in large ways. I’m better at getting what I want, getting sh*t done, etc. Here, I’m mostly just listing some of the awkward things I did while at university.
I am appreciative of folks like yourself Nora and Quintin building detailed models of the alignment problem and presenting thoughtful counterarguments to existing arguments about the difficulty. I think anyone would consider it a worthwhile endeavor regardless of their perspective on how hard the problem is, and wish you good luck in your efforts to do so.
In my culture, people understand and respect that humans can easily trick themselves into making terrible collective decisions because of tribal dynamics. They respond to this in many ways, such as by working to avoid making it a primary part of people’s work or of people’s attention, and also by making sure to not accidentally trigger tribal dynamics by inventing tribal distinctions that didn’t formerly exist but get picked up by the brain and thunk into being part of our shared mapmaking [edit: and also by keeping their identity small]. It is generally considered healthy to spend most of our attention on understanding the world, solving problems, and sharing arguments, rather than making political evaluations about which group one is a member of. People are also extra hesitant about creating groups that exist fundamentally in opposition to other groups.
My current belief is that the vast majority of the people who have thought about the impacts and alignment of advanced AI (academics like Geoffrey Hinton, forecasters like Phil Tetlock, rationalists like Scott Garrabrant, and so forth) don’t think of themselves as participating in ‘optimist’ or ‘pessimist’ communities, and would not use the term to describe their community. So my sense is that this is a false description of the world. I have a spidey-sense that language like this often tries to make itself become true by saying it is true, and is good at getting itself into people’s monkey brains and inventing tribal lines between friends where formerly there were none.
I think that the existing so-called communities (e.g. “Effective Altruism” or “Rationality” or “Academia”) are each in their own ways bereft of some essential qualities for functioning and ethical people and projects. This does not mean that if you or I create new ones quickly they will be good or even better. I do ask that you take care to not recklessly invent new tribes that have even worse characteristics than those that already exist.
From my culture to yours, I would like to make a request that you exercise restraint on the dimension of reifying tribal distinctions that did not formerly exist. It is possible that there are two natural tribes here that will exist in healthy opposition to one another, but personally I doubt it, and I hope you will take time to genuinely consider the costs of greater tribalism.
Poll For Topics of Discussion and Disagreement
Use this thread to
Upvote topics you’re interested in reading about.
Agree/disagree with positions.
Add new positions for people to vote on.