Wei_Dai
Karma: 25,383
NewTop
My sense is that building functional institutions is something that has been done in history multiple times, and that the other things you list are extreme tail scenarios.
People are trying to build more functional institutions, or at least unintentionally experimenting with different institutional designs, all the time, so the fact that building functional institutions is something that has been done in history multiple times doesn’t imply that any particular attempt to build a particular kind of functional institutions has a high or moderate chance of success.
For some context, I recommend reading Samo’s blog
Can you recommend some specific articles?
It seems like something that works at a well-funded company (that has long time horizons, like SpaceX) would probably also work for a well-funded nonprofit
Are there any articles about what is different about SpaceX as far as institutional design? I think I tried to find some earlier but couldn’t. In my current state of knowledge I don’t see much reason to think that SpaceX’s success is best explained by it having made a lot of advances in institutional design. If it actually did create a lot of advances in institutional design, and they can be applied to a nonprofit working on AI alignment, wouldn’t it be high priority to write down what those advances are and how they can be applied to the nonprofit, so other people can critique those ideas?
Let’s reorganize the rest of our discussion by splitting your charity suggestion into two parts which we can consider separately:
Use your own or your friends’ evaluations of technical work instead of outsourcing evaluations to bigger / more distant / more formal organizations.
Give awards/tenure instead of grants.
(Is this a fair summary/decomposition of your suggestion?) I think aside from my previous objections which I’m still not sure about, perhaps my true rejection to 1 combined with 2 is that I don’t want to risk feeling duped and/or embarrassed if I help fund someone’s tenure and it turns out that I or my friends misjudged their technical abilities or motivation (i.e., they end up producing low quality work or suffer a large decrease in productivity). Yeah from an altruistic perspective I should be risk neutral but I don’t think I can override the social/face-saving part of my brain on this. To avoid this I could fund an organization instead of an individual but in that case it’s likely that most of the money would go towards expansion rather than increasing slack. I guess another problem is that if I start thinking about what AI alignment work to fund I get depressed thinking about the low chance of success of any particular approach, and it seems a lot easier to just donate to an evaluator charity and make it their problem.
It seems to me that 2 by itself is perhaps more promising and worth a try. In other words we could maybe convince a charity to reallocate some money from grants to awards/tenure and check if that improves outcomes, or create a new charity for this purpose. Is that something you’d support?
In particular, I am not sure how you expect x-risk research/strategy to accomplish its objectives without having at least one functional information-processing institution! If solving AI safety (or somehow preventing AI risk without solving AI safety directly) requires solving hard philosophical problems, comparably hard to or harder than what have been solved in the past, then that requires people to optimize for something other than optics.
I guess it’s a disjunction of various low-probability scenarios like:
Metaphilosophy turns out to be easy.
Another AI alignment approach turns out to be easy (i.e., does not require solving hard philosophical problems)
Existing institutions turn out to be more effective than they appear.
Some kind of historical fluke or close call with x-risk causes world leaders to get together and cooperate to stop AI development.
In my mind “building much better information-processing institutions turns out to be easier than I expect” is comparable in likelihood to these scenarios so I don’t feel like pushing really hard on that at a high cost to myself personally or to my contributions to these other scenarios. But if others want to do it I guess I’m willing to help if I can do so at relatively low cost.
I do think there are institutions that process information well enough to do pretty-hard technical things (such as SpaceX), and some relevant questions are (a) what are they doing right, and (b) how can they be improved on.
It’s not clear to me that whatever institutional innovations SpaceX came up with (if any, i.e., if their success isn’t due to other factors) to be more successful than their competitors on solving technical problems would transfer to the kinds of problems you described for the charity sector. Until I see some analysis showing that, it seems a priori unlikely to me.
It could be that AGI will not be developed for a very long time, due to how dysfunctional current institutions are. That actually seems somewhat likely. But, that would imply that there’s a lot of time to build functional institutions.
Don’t you think it takes institutions that are more functional to build aligned AGI than to build unaligned AGI? I have a lot of uncertainty about all this but I expect that we’re currently near or over the threshold for unaligned AGI but well under the threshold for aligned AGI. (It seems totally plausible that by pushing for more functional institutions you end up pushing them over the threshold for unaligned AGI but still under the threshold for aligned AGI, but that might be a risk worth taking.)
Regarding charities: I think people should often be willing to fund things that they can actually check on and see the results of.
Almost nobody can do this for AI alignment, and it’s really costly even for the people that can. For example I personally don’t understand or can’t motivate myself to learn a lot of things that get posted to AF. Seriously, what percentage of current donors for AI alignment can do this?
You can also use recommendations by people you trust, who themselves evaluate things.
How do you suggest people to find trustworthy advisors to help them evaluate things, if they themselves are not experts in the field? Wouldn’t they have to rely on the “optics” of the advisors? Is this likely to be substantially better than relying on the optics of charities?
(Unfortunately, if you’re one of the only people actually checking, then there might not be strong UDT reasons to donate at all)
(I’m not convinced that UDT applies to humans, so if I was donating, it would be to satisfy some faction of my moral parliament.)
The “generalized MIRI donor strategy” would be to
Why is this called “generalized MIRI donor strategy”?
other people have produced some output you can evaluate (so you believe they are competent), and it seems like they are well-motivated to solve the problem (or otherwise do useful things)
This seems really hard for almost anyone to evaluate, both the technical side (see above), and the motivational side. And wouldn’t this cause a lot of people to optimize for producing output that typical donors could evaluate (rather than what’s actually important), and for producing potentially wasteful signals of motivation?
increase these people’s slack so they can optimize for something other than optics
Can you be more explicit about what you’re proposing? What exactly should one do to give these people more slack? Give unconditional cash gifts? I imagine that cash gifts wouldn’t be all that helpful unless it’s over the threshold where they can just quit their job and work on whatever they want, otherwise they’d still have to try to keep their jobs which means working on whatever their employers tell them to, which means optimizing for optics (since that’s what their employers need them to do to keep the donations coming in).
On the object level, I think MIRI has produced useful output, and that increasing their employees’ slack is a good thing, though obviously I haven’t evaluated other charities nearly as much as I have evaluated MIRI, so combine your information state with mine, and also only trust me as much as makes sense given what you know about me
If I followed this strategy, how would I know that I was funding (or doing the best that I can to fund) the most cost-effective work (i.e., that there isn’t more cost-effective work that I’m not funding because I can’t evaluate that work myself and I don’t know anyone I trust who can evaluate that work)? How would I know that I wasn’t creating incentives that are even worse than what’s typical today (namely to produce what people like me can evaluate and to produce signals of well-motivation)? How do I figure out who I can trust to help me evaluate individuals and charities? Do you think there’s a level of evaluative ability in oneself and one’s trusted advisors below which it would be better to outsource the evaluation to something like BERI, FLI, or OPP instead? If so, how do I figure out whether I’m above or below that threshold?
Instead of replying to me point by point, feel free to write up your thoughts more systematically in another post.
I enjoyed the story and I agree with most of the points you’re making, but I’m not sure calling the situation “morally depraved” is a good strategy. (I’d probably call it “suboptimal” or “dire” depending how strongly I wanted to express my feelings.) I think I’m more skeptical than you are that it’s possible to do much better (i.e., build functional information-processing institutions) before the world changes a lot for other reasons (e.g., superintelligent AIs are invented), so I’d optimize more for not making enemies or alienating people than for making people realize how bad the situation is or joining your cause.
On a practical level, do you think there are any charities existing today that are doing substantially better than others with regard to these problems?
Said, can you make it so that if I make a comment on an AF post through GW or respond to an AF comment, it automatically gets posted to AF? (Or let me know if there’s a setting to control this.) Right now I have to remember to go to that comment in LW and then manually move it to AF.
I agree with habryka that this is a really good explanation. I also agree with most of your pros and cons, but for me another major con is that this decomposition moves some problems that I think are crucial and urgent out of “AI alignment” and into the “competence” part, with the implicit or explicit implication that they are not as important, for example the problem of obtaining or helping humans to obtain a better understanding of their values and defending their values against manipulation from other AIs.
In other words, the motivation-competence decomposition seems potentially very useful to me as a way to break down a larger problem into smaller parts so it can be solved more easily, but I don’t agree that the urgent/not-urgent divide lines up neatly with the motivation/competence divide.
Aside from the practical issue of confusion between different usages of “AI alignment” (I think others like MIRI had been using “AI alignment” in a broader sense before Paul came up with his narrower definition), even using “AI alignment” in a context where it’s clear that I’m using Paul’s definition gives me the feeling that I’m implicitly agreeing to his understanding of how various subproblems should be prioritized.
I miss the days when there were hundreds of comments per day and I had to write my own Javascript code to scrape and display all the recent comments (which I called LessWrong Power Reader) to keep up. It seems really strange that even the AI alignment posts these days are getting so few comments compared to for example the decision theory posts back then, even though there must be a lot more people working on or interested in AI alignment today than decision theory back then?
I think that using a broader definition (or the de re reading) would also be defensible, but I like it less because it includes many subproblems that I think (a) are much less urgent, (b) are likely to involve totally different techniques than the urgent part of alignment.
I think it would be helpful for understanding your position and what you mean by “AI alignment” to have a list or summary of those other subproblems and why you think they’re much less urgent. Can you link to or give one here?
Also, do you have a prefered term for the broader definition, or the de re reading? What should we call those things if not “AI alignment”?
It’s a mindset which makes the following ideas seem natural
I think within “realism about rationality” there are at least 5 plausible positions one could take on other metaethical issues, some of which do not agree with all the items on your list, so it’s not really a single mindset. See this post, where I listed those 5 positions along with the denial of “realism about rationality” as the number 6 position (which I called normative anti-realism), and expressed my uncertainty as to which is the right one.
Questions for people working or thinking of working in this area:
Is there a way to have an AI “understand” that the values it is learning are not terminal values, or even instrumental values, but “interim” values? That is, they are things that humans want, subject to the fact that the humans are still trying to figure out what their real values are so the AI shouldn’t be too attached to those values. Maybe it’s possible to stretch the “utility function + mistakes” model to cover this, but it seems like it would be much better if there was a more natural / elegant way to model these “interim” values.
Relatedly, is there a way to apply value learning to the problem of metaphilosophy? In other words, can an AI, by observing humans try to solve philosophical problems, learn how to solve philosophical problems and exceed human level performance?
If the answer to the above question is “no” or “it’s too hard”, it may seem sufficient that an AI can just learn not to interfere with or manipulate a human’s philosophical and moral deliberations. This may be much easier, but if we’re headed towards a multi-polar world of AIs that are aligned to different users/owners, we also need our AIs to protect us against manipulation from other-aligned AIs. Such an AI would seemingly need to distinguish between attempts of manipulation and helpful (or at least good-faith) discussion (otherwise how would we talk with anyone else in the world without risking AI manipulation). But being able to make such distinctions seems a small step away from the ability to be actively helpful, so this problem doesn’t seem much easier than learning how to do philosophical reasoning. Still, it may be useful to consider this as a separate problem just in case it is much easier.
In your proposal, I think the mistake model is “everything that humans do could be mistaken, but when they talk about how they want their life to go, they are not mistaken about that”.
Ok, this is helpful for making a connection between my way of thinking and the “mistake model” way, but it seems a bit of a stretch, since I almost certainly am mistaken (or suboptimal) about how I want my life to go. I only want autonomy and freedom from manipulation because I don’t know how to let an AI manipulate me (i.e., make arguments to me about my values) in a way that would be safe and lead to good results. If I did, I may well let the AI do that and save myself the trouble and risk of trying to figure out my values on my own.
I think it would be useful to give your sense of how Embedded Agency fits into the more general problem of AI Safety/Alignment. For example, what percentage of the AI Safety/Alignment problem you think Embedded Agency represents, and what are the other major chunks of the larger problem?
But more generally, if you think that a different set of life experiences means that you are a different person with different values, then that’s a really good reason to assume that the whole framework of getting the true human utility function is doomed.
Maybe it’s not that bad? For example I can imagine learning the human utility function in two stages. The first stage uses the current human to learn a partial utility function (or some other kind of data structure) about how they want their life to go prior to figuring out their full utility function. E.g., perhaps they want a safe and supportive environment to think, talk to other humans, and solve various philosophical problems related to figuring out one’s utility function, with various kinds of assistance, safeguards, etc. from the AI (but otherwise no strong optimizing forces acting upon them). In the second stage, the AI use that information to compute a distribution of “preferred” future lives and then learns the full utility function only from those lives.
Another possibility is if we could design an Oracle AI that is really good at answering philosophical questions (including understanding what our confused questions mean), we can just ask it “What is my utility function?”
On second thought, even if you assume the latter, the humans you’re learning from will themselves have problems with distributional shifts. If you give someone a different set of life experiences, they’re going to end up a different person with different values, so it seems impossible to learn a complete and consistent utility function by just placing someone in various virtual environments with fake memories of how they got there and observing what they do. Will this issue be addressed in the sequence?
Can you clarify, do “queries to a human” and “data about human behavior” mean things like asking humans questions and observing human behavior in real/historical situations, or does it mean being able to put humans in arbitrary virtual environments (along with fake memories of how they got there) in order to observe their reactions? If it’s the former, I’m not sure how that lets us ignore “Distributional shift causing the learned utility function to become invalid”. If it’s the latter, I think a lot of people might be surprised by that assumption so it would be good to spell it out.
Would it be possible to post a bibliography for this sequence, similar to the one for embedded agency? It would be useful to know what body of research this sequence is based on.
A bayesian might not like this, because they’d prefer prove theorems like “The algorithm works well on average for a random environment drawn from the prior the agents use,” for which randomness is never useful.
It seems like a bayesian can conclude that randomness is useful, if their prior puts significant weight on “the environment happens to contain something that iterates over my decision algorithm and returns its worst-case input, or something that’s equivalent to or approximates this” (which they should, especially after updating on their own existence). I guess right now we don’t know how to handle this in a naturalistic way (e.g., let both intentional and accidental adversaries fall out of some simplicity prior) and so are forced to explicitly assume the existence of adversaries (as in game theory and this post).
The “E * 1e9” (note that 1e9 is a billion) part is supposed to model “Thus, if I happen to have a strong enough preference that E output True”. Does that help?
I think there are a couple of situations where trying to build FAI by specifying a utility function can make sense (which don’t include things like “get me a coffee”).
We can determine with some certainty that just maximizing some simple utility function can get us most of the potential value of the universe/multiverse. See this post of mine.
We can specify a utility function using “indirect normativity”. See this post by Paul Christiano (which doesn’t work but gives an idea of what I mean here).
I’m not sure if the papers that you’re puzzled about or criticizing have one of these in mind or something else. It might be helpful if you cited a few of them.
What if during training you can’t come close to maxing out regret for the agents that have to be trained with human involvement? That “missing” regret might come due at any time after deployment, and has to be paid with additional oversight/feedback/training data in order for those agents to continue to perform well, right? (In other words, there could be a distributional shift that causes the agents to stop performing well without additional training.) But at that time human feedback may be horribly slow compared to how fast AIs think, thus forcing IDA to either not be competitive with other AIs or to press on without getting enough human feedback to ensure safety.
Am I misunderstanding anything here? (Are you perhaps assuming that we can max out regret during training for the agents that have to be trained with human involvement, but not necessarily for the higher level agents?)
I would highly welcome that. BTW if you see me argue with Paul in the future (or in the past) and I seem to be not getting something, please feel free to jump in and explain it a different way. I often find it easier to understand one of Paul’s ideas from someone else’s explanation.
Yes, that seems easy, but actually helping seems much harder.
How do you determine what is “high-impact” before you have a utility function? Even “reversible” is relative to a utility function, right? It doesn’t mean that you literally can reverse all the consequences of an action, but rather that you can reverse the impact of that action on your utility?
It seems to me that “avoid irreversible high-impact actions” would only work if one had a small amount of uncertainty over one’s utility function, in which case you could just avoid actions that are considered “irreversible high-impact” by any the utility functions that you have significant probability mass on. But if you had a large amount of uncertainty, or just have very little idea what your utility function looks like, that doesn’t work because almost any action could be “irreversible high-impact”. For example if I were a negative utilitarian I perhaps ought to spend all my resources trying to stop technological progress leading to space colonization, so anything that I do besides that would be “irreversible high-impact” unless I could go back in time and change my resource allocation.
BTW, here is a section from a draft post that I’m working on. Do you think it would be easy to solve or avoid all of these problems? (This post isn’t specifically addressing Paul’s approach so some of them may be easy to solve under his approach but I don’t think all of them are.)
How to prevent “aligned” AIs from unintentionally corrupting human values? We know that ML systems tend to have problems with adversarial examples and distributional shifts in general. There seems to be no reason not to expect that human value functions have similar problems, which even “aligned” AIs could trigger unless they are somehow designed not to. For example, such AIs could give humans so much power so quickly or put them in such novel situations that their moral development can’t keep up, so their value systems no longer give sensible answers. (Sort of the AI assisted version of the classic “power corrupts” problem.) AIs could give us new options that are irresistible to some parts of our motivational systems, like more powerful versions of video game and social media addiction. Even in the course of trying to figure out how the world could be made better for us, they could in effect be searching for adversarial examples on our value functions. Finally, at our own request or in a sincere attempt to help us, they could generate philosophical or moral arguments that are wrong but extremely persuasive.
(Some of these issues, like the invention of new addictions and new technologies in general, would happen even without AI, but I think AIs would likely, by default, strongly exacerbate the problem by differentially accelerating such technologies faster than progress in understanding how to avoid or safely handle them.)
Why is that a further-away problem? Even if it is, we still need people to work on them now, if only to generate persuasive evidence in case they really are very hard problems so we can pursue some other strategy to avoid them like stopping or delaying the development of advanced AI as much as possible.