Running Lightcone Infrastructure, which runs LessWrong. You can reach me at habryka@lesswrong.com. I have signed no contracts or agreements whose existence I cannot mention.
habryka(Oliver Habryka)
Karma: 36,770
Hmm, a bit confused what this means. There is I think a relatively large set of skills and declarative knowledge that is pretty verifiable and objective and associated with AI Alignment.
It is the case that there is no consensus on what solutions to the AI Alignment problem might look like, but I think the basic arguments for why this is a thing to be concerned about are pretty straightforward and are associated with some pretty objective arguments.
Open Thread Fall 2024
Welcome! Hope you have a good time here.
I don’t currently know of any public examples and feel weird publicly disclosing details about organizations that I privately heard about. If more people are interested I can try to dig up some more concrete details (but can’t make any promises on things I’ll end up able sharing).
I am not sure I am understanding your question. Are you asking about examples of left-leaning projects that Dustin is involved in, or right-leaning projects that cannot get funding? On the left, Dustin is one of the biggest donors to the democratic party (with Asana donating $45M and him donating $24M to Joe Biden in 2020).
If a friend calls me and says “how soon can you be in NY?” and I respond with “well, the fastest flight gets there at 5PM” and “the slowest flight gets me there at 2PM”, my friend sure will be confused and sure is not expecting me to talk about the literal relative speeds of the plane.
In-general, I think in the context of timeline discussions, people almost always ask “how soon will AI happen?” and I think a reasonable assumption given that context is that “fast” means “sooner” and “slow” means “later”.
Yep, that hypothesis seems mostly wrong, though I more feel like I received 1-2 bits of evidence against it. If the board had stabilized with Sam being fired, even given all I know, I would have still thought a merger with Anthropic to be like ~5%-10% likely.
I don’t have a long list, but I know this is true for Lightcone, SPARC, ESPR, any of the Czech AI-Safety/Rationality community building stuff, and I’ve heard a bunch of stories since then from other organizations that got pretty strong hints from Open Phil that if they start working in an area at all, they might lose all funding (and also, the “yes, it’s more like a blacklist, if you work in these areas at all we can’t really fund you, though we might make occasional exceptions if it’s really only a small fraction of what you do” story was confirmed to me by multiple OP staff, so I am quite confident in this, and my guess is OP staff would be OK with confirming to you as well if you ask them).
(I think I agree with you. I wasn’t thinking super hard about the full context of the conversation. I was just intrigued by Nate’s challenge. I don’t really think engaging with my comment is going to be a good use of your time)
(we can safely assume that the future civ discards all the AIs that can tell they’re simulated a priori; that’s an easy tell.)
Yeah, that’s fair. It seemed more relevant to this specific hypothetical. I wasn’t really answering the question in its proper context and wasn’t applying steelmans or adjustments based on the actual full context of the conversation (and wouldn’t have written a comment without doing so, but was intrigued by your challenge).
This was close the answer I was going to give. Or more concretely, I would have said (this was written after seeing your answer, but I think is reasonably close to what I would have said independently)
The problem is at the point where god tells them that they are the only two AIs in the universe. There are issues of logical omniscience here, but an AI with a good prior should be able to tell whether it’s the kind of AI that would actually exist in base reality, or the kind of AI that would only exist in a simulation. (also just ‘existing’ is in these situations not a real thing. The question is how much magical reality-fluid have you got)
Basically, the AI will have some probability on it being real, and some probability on it being simulated, based on all the facts it knows about itself, even if you simulate reality perfectly faithfully. That prior determines how the AI will behave. You don’t get to change that prior (or like, it will be very costly for you to overcome that prior since there are a lot of AIs and you can’t simulate that many).
Imo sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade to me, at least depending on the actual numbers. As an extreme case, I would sacrifice all current OpenPhil AI safety funding in exchange for OpenPhil getting to pick which major party wins every US presidential election until the singularity.
Yeah, I currently think Open Phil’s policy activism has been harmful for the world, and will probably continue to be, so by my lights this is causing harm with the justification of causing even more harm. I agree they will probably get the bit right about what major political party would be better, but sadly the effects of policy work are much more nuanced and detailed than that, and also they will have extremely little influence on who wins the general elections.
We could talk more about this sometime. I also have some docs with more of my thoughts here (which I maybe already shared with you, but would be happy to do so if not).
Separately, this is just obviously false. A lot of the old AI safety people just don’t need OpenPhil funding anymore because they’re working at labs or governments, e.g. me, Rohin Shah, Geoffrey Irving, Paul (as you mention), etc.
I genuinely don’t know whether Rohin would get funding to pursue what he thinks is most important, if he wanted it. I agree that some others don’t “need” funding anymore, though as I said, lab incentives are even worse on these dimensions and is of very little solace to me. I agree you might be able to get funding, though also see my other discussion with Eli on the boundaries I was trying to draw (which I agree are fuzzy and up-to-debate).
My sense is unions make sense, but legal protections where companies aren’t allowed to route around unions are almost always quite bad. Basically whenever those were granted the unions quickly leveraged what basically amounts to a state-sponsored monopoly, but in ways that are even worse than normal state-sponsored monopolies, because state-sponsored monopolies at least tend to still try to maximize profit, whereas unions tend to basically actively sabotage almost anything that does not myopically give more resources to its members.
You can see some of the EA Forum discussion here: https://forum.effectivealtruism.org/posts/foQPogaBeNKdocYvF/linkpost-an-update-from-good-ventures?commentId=RQX56MAk6RmvRqGQt
The current list of areas that I know about are:
Anything to do with the rationality community (“Rationality community building”)
Anything to do with moral relevance of digital minds
Anything to do with wild animal welfare and invertebrate welfare
Anything to do with human genetic engineering and reproductive technology
Anything that is politically right-leaning
There are a bunch of other domains where OP hasn’t had an active grantmaking program but where my guess is most grants aren’t possible:
Most forms of broad public communication about AI (where you would need to align very closely with OP goals to get any funding)
Almost any form of macrostrategy work of the kind that FHI used to work on (i.e. Eternity in Six Hours and stuff like that)
Anything about acausal trade of cooperation in large worlds (and more broadly anything that is kind of weird game theory)
I remember running into her a bunch before I ran into Buck. Scott/Abram are also second generation. Overall, seems reasonable to include Buck (but communicating my more complicated epistemic state with regard to him would have been harder).
(I like Buck, but he is one generation later than the one I was referencing. Also, I am currently like 50⁄50 whether Buck would indeed be blacklisted. I agree that Carl is a decent counterexample, though he is a bit of a weirder case)
Huh, can you say more about why you are otherwise pro-union? All unions I have interfaced with were structurally the same as this dockworker’s strike. Maybe there were some mid-20th-century unions that were better, there were a lot of fucked up things then, but at least modern unions seem to be somewhat universally terrible in this way.
I think OP has funded almost everyone I have listed here in 2022 (directly or indirectly), so I don’t really think that is evidence of anything (though it is a bit more evidence for ARC because it means the COI is overcomable).
Yep, agree that this is a commonly overlooked aspect (and one that I think sadly has also contributed to the dominant force in AI Safety researchers becoming the labs, which I think has been quite sad).
I really think if you want to tell a story of AI Control work being good (especially compared to working on preventing AGI from being built in the first place), the important and difficult part is figuring out how to actually use these more powerful AI systems to either achieve some kind of global moratorium, or make unprecedented progress on the hard parts of the AI Alignment problem.
When I see most people start thinking about control, I rarely see them interface with either of these two things, and honestly, I mostly see them come up with cool additional commercial applications of AI, which I think will mostly make the present situation worse. To be clear, I think thinking about how to use AI systems for global coordination or making progress on the hard parts of the AI Alignment problem is tractable, but I haven’t seen much work on it, and I am worried it is not the default thing that people will work on when they start thinking about control.
I think it is unlikely for control work to buy humanity much time until someone builds a very powerful unaligned AI system, at least at our present levels of coordination tools. I think controlled AI systems might help with coordination here, but I do think someone needs to do the actual hard work of thinking about how it might.
(I go into this and various related things in my dialogue with Ryan on control)