Coding day in and out on LessWrong 2.0. You can reach me at habryka@lesswrong.com
habryka(Oliver Habryka)
Karma: 15,281
What does “implicit request” mean here?
I don’t know, kind of complicated, enough that I could probably write a sequence on it, and not even sure I would have full introspective access into what I would feel comfortable labeling as an “implicit request”.
I could write some more detail, but it’s definitely a matter of degree, and the weaker the level of implicit request, the weaker the reason for sharing needs to be, with some caveats about adjusting for people’s communication skills, adversarial nature of the communication, adjusting for biases, etc.
I think I could write down a full history of employment for all of these orgs (except maybe FHI, which I’ve had fewer tabs on), in an hour or two of effort. It’s somewhat costly for me (in terms of time), but if lots of people are interested, I would be happy to do it.
My current feelings are a mixture of the following:
I disagree with a lot of the details of what many people have said (both people who had bad experiences and people defending their Leverage experiences and giving positive testimonials), and feel like expressing my take has some chance of making those people feel like their experiences are invalidated, or at least spark some conflict of some type
I know that Geoff and Leverage more broadly in the past have said pretty straightforwardly that they will take pretty adversarial action if someone threatens their reputation or brand, and that makes me both feel like I can trust many fewer things in the discussion, and makes me personally more hesitant to share some things (while also feeling like that’s kind of cowardly, but I haven’t yet had the time to really work through my feelings here, which in itself has some chilling effects that I feel uncomfortable with, etc.)
On the other side, there have been a lot of really vicious and aggressive attacks to anyone saying anything pro-leverage for many years, with a strength that I think is overall even greater and harder to predict than what Geoff and Leverage have been doing. It’s also been more of a crowd-driven phenomenon, which makes it less predictable and more scary.
I feel like it’s going to be really hard to say anything without people pigeonholing me into belonging to some group that is trying to rewrite the rationality social and political landscape some way, and that makes me feel like I have to actively think about how to phrase what I am saying in a way that avoids that pigeonholing effect (as a concrete example, one person approached me who read Ben’s initial comment on the “BayAreaHuman” post that said “I confirm that this is a real person in good standing” as an endorsement of the post, when the comment was really just intended as confirming some facts about the identity of the poster, with basically complete independence from the content of the post)
I myself have access to some sensitive and somewhat confidential information, and am struggling with navigating exactly which parts are OK to share and which ones are not.
I generally feel reasonably comfortable sharing unsolicited emails, unless the email makes some kind of implicit request to not be published, that I judge at least vaguely valid. In general I am against “default confidentiality” norms, especially for requests or things that might be kind of adversarial. I feel like I’ve seen those kinds of norms weaponized in the past in ways that seems pretty bad, and think that while there is a generally broad default expectation of unsolicited private communication being kept confidential, it’s not a particularly sacred protection in my mind (unless explicitly or implicitly requested, in which case I think I would talk to the person first to get a more fully comprehensive understanding for why they requested confidentiality, and would generally err on the side of not publishing, though would feel comfortable overcoming that barrier given sufficient adversarial action)
I think the claim is false, though it’s hard to prove an absence. I have never heard of anyone having a particularly strong emotional reaction to Roko’s basilisk (outside of a “sneering” reaction towards rationalists or something), and almost everyone I’ve met in the community thinks the thought experiment is not particularly interesting or important or dangerous. I do like the general policy of not talking a ton about infohazards of this type, but that’s just a broad reference class, of which Roko’s basilisk seems like a non-central example.
The Roko’s Basilisk wiki article has a lot of detail, and I recommend looking through it. It also includes these paragraphs:
Other sources have repeated the claim that Less Wrong users think Roko’s basilisk is a serious concern. However, none of these sources have yet cited supporting evidence on this point, aside from Less Wrong moderation activity itself. (The ban, of course, didn’t make it easy to collect good information.)
Less Wrong user Gwern reports that “Only a few LWers seem to take the basilisk very seriously,” adding, “It’s funny how everyone seems to know all about who is affected by the Basilisk and how exactly, when they don’t know any such people and they’re talking to counterexamples to their confident claims.”
My current model is that this changed around 2017 or so. At least my sense was that people from before that time often had tons of side-projects and free time, and afterwards people seemed to have really full schedules and were constantly busy.
Whatever is happening, the public forum debate will have no impact on it;
I think this is wrong. I think a lot of people who care about AI Alignment read LessWrong and might change their relationship to Open AI depending on what is said here.
Welcome! I hope you have a good time here!
Fixed
Having specific theistic beliefs seems very different than “having drawn inspiration” from someone.
Ah, didn’t see this before I wrote my comment. This all seems correct to me, my comment has a bit of additional flavor but basically says the same thing.
Here is my current take on security for the site:
We pay enough attention to security that I think it’s unlikely we will accidentally dump our whole database to the internet.
I think any sophisticated attacker who is willing to spend something like 50-100 hours on trying to get into our system, can probably find some way to get access to most things in our database.
We do obvious things like encrypt passwords and salt hashes, so I don’t think there is any way I know of for an attacker to reverse-engineer secure user passwords (though having access to the hash is still often useful).
The javascript ecosystem in general, and the way we have been using it, has a lot of attack surface, and I expect that there are probably some vulnerabilities in libraries we are using on the backend, that allow some kind of code execution, though I do think it would require a good amount of work to find them.
I think it’s reasonable to assume that if any nation state or sophisticated group of attackers wants access to your PMs, votes and drafts, they can probably get it somehow, either by attacking the client, infecting an admin or moderator, or just directly hacking our server somehow.
This overall puts somewhat of an upper bound on how useful additional focus on security would be, since I currently think it would be very hard to harden our system against sophisticated attackers. I currently would not treat LW or the AI Alignment Forum as particularly secure, and if you want to have conversations with individuals that will not be accessible by potential sophisticated attackers, I would use something like Signal. In general, there isn’t a lot of private information on LessWrong (outside of PMs and votes, and maybe drafts), so I don’t tend to think of security as a top priority for the site (though I still think a good amount about security, and am also not treating it as super low priority, but it isn’t one of the things the whole site was architected around).
I am not currently very worried about astroturfing or trolling, mostly because we are currently still capable of reviewing all content posted to the site, and can individually review each, and I think that’s a pretty high barrier for language models to overcome. If even I can’t tell the difference between a bot and a person anymore, then yeah, I am not super sure what to do. We could do some kind of identity verification with passports and stuff, but I would prefer to avoid that.
We run some basic statistics on vote patterns and would notice if suddenly some new set of accounts were starting to vote a lot and influence site content a lot. I would also likely notice myself that something is off with that kinds of things are getting upvoted and downvoted and would likely investigate.
The internet archive tends to have backups of all publicly available pages.
We handle our own backups on AWS, and I tend to download one every few months to some hard drive somewhere.
Changing hosting providers would be annoying, but not super hard. My guess is it would take us less than three days to be back up and running if AWS no longer likes us.
0.7*X
30% of literally the highest offer you can get
This is roughly the sense in which I meant “competitive” (I think there are some edge-cases here, where for example I don’t expect we will be able to fully cover the right tail of outcomes. Like, if Sam Bankman-Fried had decided to work with us instead of found FTX, we of course couldn’t have paid him 10 billion dollars, or similar situations).
Some of the recent edits changed some of the links to no longer have the
?showPostCount=true&useTagName=truequery parameters in the links, which changes how they are displayed and makes the display inconsistent. Seems like we should fix this.
I’m a little curious how that ends up working for senior candidates who could be getting 450k (which is basically standard comp at those tech companies for senior engineers) - do you just assume that they’d be capable of passing an interview at one of those places if they clear your bar, assuming they don’t work somewhere like that already?
I am not fully sure yet what the right algorithm here will be, since we haven’t run into that problem yet. My guess is I would try to call in a third party to give me a guess of how much they could make in industry, or we just negotiate a bit back-and-forth and they just tell me the evidence they have for how much they could make in industry if they tried. I can also imagine this turning out to be harder, and I would have to think more about how to best get a fair assessment here.
I think if you start asking people to, say, provide offer letters demonstrating their “market value”, you run the risk of someone looking at their options and then changing their mind.
This seems like a fine outcome to me. Indeed, in the past I have told past LW/Lightcone employees to really try to look for other options and take them seriously, even after I made them an offer, so that if they do decide to take the offer we both felt confident that working at LW/Lightcone is the best choice for them.
Having an in-person campus that allows people to have really good high-bandwidth communication is a big component that I now think is a really useful thing to have in many worlds.
On a higher level of abstraction, I have an internal model that suggests something like the following three components are things that are quite important for AGI (and some other x-risks) to go right:
The ability to do really good research and be really good at truth-seeking (necessary to solve various parts of the AI Alignment problem, and also in general just seems really important for a community to have for lots of reasons)
The ability to take advantage of crises and navigate really quickly changing situations (as a concrete intuition pump, I currently believe that before something like AGI we will probably have something like 10 more years at least as crazy as 2020, and I have a sense that some of the worlds where things go well, are worlds where a bunch of people concerned about AI Alignment are well set-up to take advantage of them, and make sure to not get wiped out by them)
The ability to have high-stakes negotiations with large piles of resources and people (like, I think it’s pretty plausible that in order to actually get the right AI Alignment solution deployed, and to avoid us getting killed some other way before then, some people who have some of the relevant components of solutions will need to negotiate in some pretty high-stakes situations to actually make them happen. And in a much more coherent way than people are currently capable of.)
Though these are all pretty abstract and high-level, and I have a lot of concrete thoughts that are less abstract, though it would take me a while to write them up.
Also, if I may ask “no longer seems sufficient”. Did you thought it was? The sentence seems really strange to be honest, or otherwise I’d be curious if you have a text where you explained why you thought that, as it seems quite surprising
I do think something like this is kind of correct. It’s not that I thought that nothing else had to happen between now and then for humanity to successfully reach the stars, but I did meaningfully think that there were a good number of universes where my work on LessWrong would make the difference (with everyone else of course also doing things), and that I was really moving probability mass.
I still think I moved some probability mass, but I further updated that in order to realize a bunch of that probability mass that I was hoping for, I need to get some other pieces in place. Which is something I didn’t think was as necessary, and I used to think more that the online component of things would itself be sufficient to realize a lot of that probability mass.
I definitely didn’t believe that if I were to just make LessWrong great, existential risk would be solved in most worlds.
Note: I decided to frontpage this post, despite it being more of an organizational announcement, because it does feel pretty relevant to everyone on the site, and I would feel bad if someone was a regular user of LessWrong and didn’t know about this relatively large change in our operating structure.
The post explicitly calls for thinking about how this situation is similar to what is happening/happened at Leverage, and I think that’s a good thing to do. I do think that I do have specific evidence that makes me think that what happened at Leverage seemed pretty different from my experiences with CFAR/MIRI.
Like, I’ve talked to a lot of people about stuff that happened at Leverage in the last few days, and I do think that overall, the level of secrecy and paranoia about information leaks at Leverage seemed drastically higher than anywhere else in the community that I’ve seen, and I feel like the post is trying to draw some parallel here that fails to land for me (though it’s also plausible it is pointing out a higher level of information control than I thought was present at MIRI/CFAR).
I have also had my disagreements with MIRI being more secretive, and think it comes with a high cost that I think has been underestimated by at least some of the leadership, but I haven’t heard of people being “quarantined from their friends” because they attracted some “set of demons/bad objects that might infect others when they come into contact with them”, which feels to me like a different level of social isolation, and is part of the thing that happened in Leverage near the end. Whereas I’ve never heard of anything even remotely like this happening at MIRI or CFAR.
To be clear, I think this kind of purity dynamic is also present in other contexts, like high-class/low-class dynamics, and various other problematic common social dynamics, but I haven’t seen anything that seems to result in as much social isolation and alienation, in a way that seemed straightforwardly very harmful to me, and more harmful than anything comparable I’ve seen in the rest of the community (though not more harmful than what I have heard from some people about e.g. working at Apple or the U.S. military, which seem to have very similarly strict procedures and also a number of quite bad associated pathologies).
The other biggest thing that feels important to distinguish between what happened at Leverage and the rest of the community is the actual institutional and conscious optimization that has gone into PR control.
Like, I think Ben Hoffman’s point about “Blatant lies are the best kind!” is pretty valid, and I do think that other parts of the community (including organizations like CEA and to some degree CFAR) have engaged in PR control in various harmful but less legible ways, but I do think there is something additionally mindkilly and gaslighty about straightforwardly lying, or directly threatening adversarial action to prevent people from speaking ill of someone, in the way Leverage has. I always felt that the rest of the rationality community had a very large and substantial dedication to being very clear about when they denotatively vs. connotatively disagree with something, and to have a very deep and almost religious respect for the literal truth (see e.g. a lot of Eliezer’s stuff around the wizard’s code and meta honesty), and I think the lack of that has made a lot of the dynamics around Leverage quite a bit worse.
I also think it makes understanding the extent of the harm and ways to improve it a lot more difficult. I think the number of people who have been hurt by various things Leverage has done is really vastly larger than the number of people who have spoken out so far, in a ratio that I think is very different from what I believe is true about the rest of the community. As a concrete example, I have a large number of negative Leverage experiences between 2015-2017 that I never wrote up due to various complicated adversarial dynamics surrounding Leverage and CEA (as well as various NDAs and legal threats, made by both Leverage and CEA, not leveled at me, but leveled at enough people around me that I thought I might cause someone serious legal trouble if I repeat a thing I heard somewhere in a more public setting), and I feel pretty confident that I would feel very different if I had similarly bad experiences with CFAR or MIRI, based on my interactions with both of these organizations.
I think this kind of information control feels like what ultimately flips things into the negative for me, in this situation with Leverage. Like, I think I am overall pretty in favor of people gathering together and working on a really intense project, investing really hard into some hypothesis that they have some special sauce that allows them to do something really hard and important that nobody else can do. I am also quite in favor of people doing a lot of introspection and weird psychology experiments on themselves, and to try their best to handle the vulnerability that comes with doing that near other people, even though there is a chance things will go badly and people will get hurt.
But the thing that feels really crucial in all of this is that people can stay well-informed and can get the space they need to disengage, can get an external perspective when necessary, and somehow stay grounded all throughout this process. Which feels much harder to do in an environment where people are directly lying to you, or where people are making quite explicit plots to discredit you, or harm you in some other way, if you do leave the group, or leak information.
I do notice that in the above I make various accusations of lying or deception by Leverage without really backing it up with specific evidence, which I apologize for, and I think people reading this should overall not take comments like mine at face value before having heard something pretty specific that backs up the accusations in them. I have various concrete examples I could give, but do notice that doing so would violate various implicit and explicit confidentiality agreements I made, that I wish I had not made, and I am still figuring out whether I can somehow extract and share the relevant details, without violating those agreements in any substantial way, or whether it might be better for me to break the implicit ones of those agreements (which seem less costly to break, given that I felt like I didn’t really fully consent to them), given the ongoing pretty high cost.