Vladimir_Nesov
Karma: 20,515
NewTop
Learning how to design goal-directed agents seems like an almost inevitable milestone on the path to figuring out how to safely elicit human preference in an actionable form. But the steps involved in eliciting and enacting human preference don’t necessarily make use of a concept of preference or goal-directedness. An agent with a goal aligned with the world can’t derive its security from the abstraction of goal-directedness, because the world determines that goal, and so the goal is vulnerable to things in the world, including human error. Only self-contained artificial goals are safe from the world and may lead to safety of goal-directed behavior. A goal built from human uploads that won’t be updated from the world in the future gives safety from other things in the world, but not from errors of the uploads.
When the issue is figuring out which influences of the world to follow, it’s not clear that goal-directedness remains salient. If there is a goal, then there is also a world-in-the-goal and listening to your own goal is not safe! Instead, you have to figure out which influences in your own goal to follow. You are also yourself part of the world and so there is an agent-in-the-goal that can decide aspects of preference. This framing where a goal concept is prominent is not obviously superior to other designs that don’t pursue goals, and instead focus on pointing at the appropriate influences from the world. For example, a system may seek to make reliable uploads, or figure out which decisions of uploads are errors, or organize uploads to make sense of situations outside normal human environments, or be corrigible in a secure way, so as to follow directions of a sane external operator and not of an attacker. Once we have enough of such details figured out (none of which is a goal-directed agent), it becomes possible to take actions in the world. At that point, we have a system of many carefully improved kluges that further many purposes in much the same way as human brains do, and it’s not clearly an improvement to restructure that system around a concept of goals, because that won’t move it closer to the influences of the world it’s designed to follow.
My guess is that agents that are not primarily goal-directed can be good at defending against goal-directed agents (especially with first mover advantage, preventing goal-directed agents from gaining power), and are potentially more tractable for alignment purposes, if humans coexist with AGIs during their development and operation (rather than only exist as computational processes inside the AGI’s goal, a situation where a goal concept becomes necessary).
I think the assumption that useful agents must be goal-directed has misled a lot of discussion of AI risk in the past. Goal-directed agents are certainly a problem, but not necessarily the solution. They are probably good for fixing astronomical waste, but maybe not AI risk.
Trying to have influence over aspects of value change that people don’t much care about … [is] reasonable … to do to make the future better
This could refer to value change in AI controllers, like Hugh in HCH, or alternatively to value change in people living in the AI-managed world. I believe the latter could be good, but the former seems very questionable (here “value” refers to true/normative/idealized preference). So it’s hard for the same people to share the two roles. How do you ensure that value change remains good in the original sense without a reference to preference in the original sense, that hasn’t experienced any value change, a reference that remains in control? And for this discussion, it seems like the values of AI controllers (or AI+controllers) is what’s relevant.
It’s agent tiling for AI+controller agents, any value change in the whole seems to be a mistake. It might be OK to change values of subagents, but the whole shouldn’t show any value drift, only instrumentally useful tradeoffs that sacrifice less important aspects of what’s done for more important aspects, but still from the point of view of unchanged original values (to the extent that they are defined at all).
I understand that there is no point examining one’s algorithm if you already execute it and see what it does.
Rather there is no point if you are not going to do anything with the results of the examination. It may be useful if you make the decision based on what you observe (about how you make the decision).
you say “nothing stops you”, but that is only possible if you could act contrary to your own algorithm, no?
You can, for a certain value of “can”. It won’t have happened, of course, but you may still decide to act contrary to how you act, two different outcomes of the same algorithm. The contradiction proves that you didn’t face the situation that triggers it in actuality, but the contradiction results precisely from deciding to act contrary to the observed way in which you act, in a situation that a priori could be actual, but is rendered counterlogical as a result of your decision. If instead you affirm the observed action, then there is no contradiction and so it’s possible that you have faced the situation in actuality. Thus the “chicken rule”, playing chicken with the universe, making the present situation impossible when you don’t like it.
So your reasoning is inaccurate
You don’t know that it’s inaccurate, you’ve just run the computation and it said $5. Maybe this didn’t actually happen, but you are considering this situation without knowing if it’s actual. If you ignore the computation, then why run it? If you run it, you need responses to all possible results, and all possible results except one are not actual, yet you should be ready to respond to them without knowing which is which. So I’m discussing what you might do for the result that says that you take the $5. And in the end, the use you make of the results is by choosing to take the $5 or the $10.
This map from predictions to decisions could be anything. It’s trivial to write an algorithm that includes such a map. Of course, if the map diagonalizes, then the predictor will fail (won’t give a prediction), but the map is your reasoning in these hypothetical situations, and the fact that the map may say anything corresponds to the fact that you may decide anything. The map doesn’t have to be identity, decision doesn’t have to reflect prediction, because you may write an algorithm where it’s not identity.
For example, in the 5&10 game an agent would examine its own algorithm, see that it leads to taking $10 and stop there.
Why do even that much if this reasoning could not be used? The question is about the reasoning that could contribute to the decision, that could describe the algorithm, and so has the option to not “stop there”. What if you see that your algorithm leads to taking the $10 and instead of stopping there, you take the $5?
Nothing stops you. This is the “chicken rule” and it solves some issues, but more importantly illustrates the possibility in how a decision algorithm can function. The fact that this is a thing is evidence that there may be something wrong with the “stop there” proposal. Specifically, you usually don’t know that your reasoning is actual, that it’s even logically possible and not part of an impossible counterfactual, but this is not a hopeless hypothetical where nothing matters. Nothing compels you to affirm what you know about your actions or conclusions, this is not a necessity in a decision making algorithm, but different things you do may have an impact on what happens, because the situation may be actual after all, depending on what happens or what you decide, or it may be predicted from within an actual situation and influence what happens there. This motivates learning to reason in and about possibly impossible situations.
What if you examine your algorithm and find that it takes the $5 instead? It could be the same algorithm that takes the $10, but you don’t know that, instead you arrive at the $5 conclusion using reasoning that could be impossible, but that you don’t know to be impossible, that you haven’t decided yet to make impossible. One way to solve the issue is to render the situation where that holds impossible, by contradicting the conclusion with your action, or in some other way. To know when to do that, you should be able to reason about and within such situations that could be impossible, or could be made impossible, including by the decisions made in them. This makes the way you reason in them relevant, even when in the end these situations don’t occur, because you don’t a priori know that they don’t occur.
(The 5-and-10 problem is not specifically about this issue, and explicit reasoning about impossible situations may be avoided, perhaps should be avoided, but my guess is that the crux in this comment thread is about things like usefulness of reasoning from within possibly impossible situations, where even your own knowledge arrived at by pure computation isn’t necessarily correct.)
so long as you can’t change their notions of status there is nothing you can do to communicate “you are fundamentally wrong about how this works” without them hearing it as “I don’t realize how far out of my depth I am right now”.
But from the other direction, it seems quite possible to hear what the wrong-status person says about how I’m wrong. So “nothing you can do” seems excessive. Perhaps politeness often suffices, for arguments that would be accepted when delivered by an appropriate-status person, as long as you are being heard at all.
These are decisions in different situations. Transitivity of preference is about a single situation. There should be three possible actions A, B and C that can be performed in a single situation, with B preferred to A and C preferred to B. Transitivity of preference says that C is then preferred to A in that same situation. Betting on a fight of B vs. A is not a situation where you could also bet on C, and would prefer to bet on C over betting on B.
It seems like a fairly straightforward claim, to say that a quality like integrity is all the more valuable for being rare.
I think the difference is between associating with producers vs. consumers. When something is more scarce, its price increases, which makes it less valuable to consumers and more valuable to producers of individual items. So for people who perceive themselves as producers of things like integrity and virginity and honest labor, scarcity would contribute to their value. And for things like medicine or the naive models of bitcoin and diamonds, decrease their value by increasing the price, since the audience of the article identify as consumers.
You can decide to question any such principles, which is how they get formulated in the first place, as designs for improved cognition devised by an evolved mind that doesn’t originally follow any particular crisp design, but can impose order on itself. The only situation where they remain stable is if the decision always comes out in their favor, which will happen if they are useful for agents pursuing your preference. When these agents become sufficiently different, they probably shouldn’t use any object level details of the design of cognition that holds for you. The design improves, so it’s not the same.
Examples of such principles are pursuit of well-calibrated empirical beliefs, of valid mathematical knowledge, of useful plans, and search for rational principles of cognition.
I don’t know how to describe the thing that remains through correct changes, which is probably what preference should be, so it’s never formal. There shouldn’t be a motivation to “be at peace” with it, since it’s exactly the thing you turn out to be at peace with, for reasons other than being at peace with it.
This is a duplicate of the post made yesterday.
I don’t know how shminux interprets the words, and if the question was related to this, but there is an issue in your use of “impossible”. Things that happen in this world are actual, and things that happen in alternative worlds are possible. (The set of alternative worlds needs to be defined for each question.) An impossible situation is one that can’t occur, that doesn’t happen in any of the alternative worlds.
Thus outputs of an algorithm different from the actual output are not actual, and furthermore not possible, as they don’t occur in the alternative worlds with the same algorithm receiving the same inputs. But there are alternative worlds (whose set is different than in the previous sentence, no longer constrained by the state of input) with the same algorithm that receives different inputs, and so despite not being actual, the different inputs are still possible, in other words not impossible.
Bug report: Posts by Day page shows incorrect URLs for most posts, for example it lists the post “The Rocket Alignment Problem” with a link to Fasting Mimicking Diet Looks Pretty Good. Looks like some kind of intermittent off-by-one error that only affects some of the entries. The entries on the same page loaded with the “Load More Days” link don’t seem to be affected. I’m not seeing this issue on the front page, and it only appeared around yesterday. Interestingly, if I click the “All Posts/Daily” link on the front page, which leads to the same URL, then there is no issue; it’s only reproduced if I open the URL directly.
There’s an archived copy here.
(That may be a useful clue for identifying the meaning of the question, as understood by the people pursuing it, but not necessarily a good reason to agree that it currently should be considered mysterious or that it’s a sensible question to pursue.)
[W]e still need a coherent moral framework to use to generate our AI’s utility function if we want it to be aligned, so morality is important, and we do need to develop an acceptable solution to it.
This is not clear. It’s possible for the current world to exist as it is, and similarly for any other lawful simulated world that’s not optimized in any particular direction. So an AI could set up such a world without interfering, and defend its lawful operation from outside interference. This is a purposeful thing, potentially as good at defending itself as any other AI, that sets up a world that’s not optimized by it, that doesn’t need morality for the world it maintains, in order to maintain it. Of course this doesn’t solve any problems inside that world, and it’s unclear how to make such a thing, but it illustrates the problems with the “morality is necessary for AGI” position. Corrigibility also fits this description, being something unlike an optimization goal for the world, but still a purpose.
The concept is fine, it’s the misuse of the words denoting it that can cause problems by diluting the concept, giving bad training data so that the concept gets learned in many distorted forms. It might be useful to taboo the words that denote the concept in order to make explaining it easier. Or just refer people to a blog post somewhere that explains the idea, it’s the same as any knowledge.
It’s also not necessary to show people how to be an effective altruist (or to believe or assert that it’s a good idea) in order to explain the concept.
I write notes in a single plain text file, using the dates they are made to cite them in newer notes. There are two types of notes, brainstorming throw-away ones that maintain the process of thinking about a problem or of learning something (such as carefully reading a paper), and more lucid ones, with some re-reading value, which are marked differently and have a one-sentence summary. The notes are intended to never be made public, so that I feel free to use them to resolve any silly confusions.
The useful question is about value of the data that can be collected about people, not so much its usefulness for achieving a particular task, because it may no longer make sense to perform that task once it becomes possible to do so (as in giant cheesecake fallacy). A system that can reconstruct people from data can do many other things that may be more valuable than reconstruction of people even from the point of view of these people that could’ve been reconstructed. It’s a question of what should be done with the resources and capabilities of that system.
The value of the data is in its contribution to the value of the best things that can be made, and these best things don’t necessarily improve through availability of that data, because they are not necessarily reconstructed people. I’m not sure there is any knowable-on-human-level difference in value between what can be done with the world given the knowledge about particular people who used to live in the past (either through indirect data or cryopreserved brains), compared to the value of what can be done without that knowledge. I guess it can’t hurt if it doesn’t consume resources that could otherwise find a meaningful purpose.
I worry there might be leaks in logical time that let the agent choose an action that takes into account that an impactful action will be denied. For example, a sub-agent could be built so that it’s a maximizer that’s not constrained by an impact measure. The sub-agent then notices that to maximize its goal, it must constrain its impact, or else the main agent won’t be allowed to create it. And so it will so constrain its impact and will be allowed to be created, as a low-impact and maximally useful action of the main agent. It’s sort of a daemon, but with respect to impact measure and not goals, which additionally does respect the impact measure and only circumvents it once in order to get created.
By the way, comment permalinks don’t work for comments in collapsed subthreads (example). The anchor should be visible from javascript, so this could be fixed by expanding the subthread and navigating to the anchor.