Vladimir_Nesov
Karma: 20,592 (LW), 6 (AF)
NewTop
What you describe does not want to be a thought experiment, because it doesn’t abstract away relevant confounders (moral value of human life). The setup in the post is better at being a thought experiment for the distinctions being discussed (moral value of golem’s life more clearly depends on a moral framework). In this context, it’s misleading to ask whether something should be done instead of whether it’s the action that’s hedonistic utilitarian / preference utilitarian / autonomy-preserving.
The latter, where “a lot of work” is the kind of thing humanity can manage in subjective centuries. In an indirect normativity design, doing much more work than that should still be feasible, since it’s only specified abstractly, to be predicted by an AI, enabling distillation. So we can still reach it, if there is an AI to compute the result. But if there is already such an AI, perhaps the work is pointless, because the AI can carry out the work’s purpose in a different way.
Humans are not immediately prepared to solve many decision problems, and one of the hardest problems is formulation of preference for a consequentialist agent. In expanding the scope of well-defined/reasonable decisions, formulating our goals well enough for use in a formal decision theory is perhaps the last milestone, far outside of what can be reached with a lot of work!
Indirect normativity (after distillation) can make the timeline for reaching this milestone mostly irrelevant, as long as there is sufficient capability to compute the outcome, and amplification is about capability. It’s unclear how the scope of reasonable decisions is related to capability within that scope, amplification seems ambiguous between the two, perhaps the scope of reasonable decisions is just another kind of stuff that can be improved. And it’s corrigibility’s aspect to keep AI within the scope of well-defined decisions.
But with these principles in place, it’s unclear if formulating goals for consequentialist agents remains a thing, when instead it’s possible to just continue to expand the scope of reasonable decisions and to distill/amplify them.
A well-order has a least element in all non-empty subsets, and 1 > 1⁄2 > 1⁄4 > … > 0 has a non-empty subset without a least element, so it’s not a well-order.
I worry that in the context of corrigibility it’s misleading to talk about alignment, and especially about utility functions. If alignment characterizes goals, it presumes a goal-directed agent, but a corrigible AI is probably not goal-directed, in the sense that its decisions are not chosen according to their expected value for a persistent goal. So a corrigible AI won’t be aligned (neither will it be misaligned). Conversely, an agent aligned in this sense can’t be visibly corrigible, as its decisions are determined by its goals, not orders and wishes of operators. (Corrigible AIs are interesting because they might be easier to build than aligned agents, and are useful as tools to defend against misaligned agents and to build aligned agents.)
In the process of gradually changing from a corrigible AI into an aligned agent, an AI becomes less corrigible in the sense that corrigibility ceases to help in describing its behavior, it stops manifesting. At the same time, goal-directedness starts to dominate the description of its behavior as the AI learns well enough what its goal should be. If during the process of learning its values it’s more corrigible than goal-directed, there shouldn’t be any surprises like sudden disassembly of its operators on molecular level.
I thought the point of idealized humans was to avoid problems of value corruption or manipulation
Among other things, yes.
which makes them better than real ones
This framing loses the distinction I’m making. More useful when taken together with their environment, but not necessarily better in themselves. These are essentially real humans that behave better because of environments where they operate and lack of direct influence from the outside world, which in some settings could also apply to the environment where they were raised. But they share the same vulnerabilities (to outside influence or unusual situations) as real humans, which can affect them if they are taken outside their safe environments. And in themselves, when abstracted from their environment, they may be worse than real humans, in the sense that they make less aligned or correct decisions, if the idealized humans are inaccurate predictions of hypothetical behavior of real humans.
If it’s too hard to make AI systems in this way and we need to have them learn goals from humans, we could at least have them learn from idealized humans rather than real ones.
My interpretation of how the term is used here and elsewhere is that idealized humans are usually in themselves, and when we ignore costs, worse than real ones. For example, they could be based on predictions of human behavior that are not quite accurate, or they may only remain sane for an hour of continuous operation from some initial state. They are only better because they can be used in situations where real humans can’t be used, such as in an infinite HCH, an indirect normativity style definition of AI goals, or a simulation of how a human develops when exposed to a certain environment (training). Their nature as inaccurate predictions may make them much more computationally tractable and actually available in situations where real humans aren’t, and so more useful when we can compensate for the errors. So a better term might be “abstract humans” or “models of humans”.
If these artificial environments with models of humans are good enough, they may also be able to bootstrap more accurate models of humans and put them into environments that produce better decisions, so that the initial errors in prediction won’t affect the eventual outcomes.
More to the point, these failure modes are ones that we can talk about from outside
So can the idealized humans inside a definition of indirect normativity, which motivates them to develop some theory and then quarantine parts of the process to examine their behavior from outside the quarantined parts. If that is allowed, any failure mode that can be fixed by noticing a bug in a running system becomes anti-inductive: if you can anticipate it, it won’t be present.
By the way, comment permalinks don’t work for comments in collapsed subthreads (example). The anchor should be visible from javascript, so this could be fixed by expanding the subthread and navigating to the anchor.
Learning how to design goal-directed agents seems like an almost inevitable milestone on the path to figuring out how to safely elicit human preference in an actionable form. But the steps involved in eliciting and enacting human preference don’t necessarily make use of a concept of preference or goal-directedness. An agent with a goal aligned with the world can’t derive its security from the abstraction of goal-directedness, because the world determines that goal, and so the goal is vulnerable to things in the world, including human error. Only self-contained artificial goals are safe from the world and may lead to safety of goal-directed behavior. A goal built from human uploads that won’t be updated from the world in the future gives safety from other things in the world, but not from errors of the uploads.
When the issue is figuring out which influences of the world to follow, it’s not clear that goal-directedness remains salient. If there is a goal, then there is also a world-in-the-goal and listening to your own goal is not safe! Instead, you have to figure out which influences in your own goal to follow. You are also yourself part of the world and so there is an agent-in-the-goal that can decide aspects of preference. This framing where a goal concept is prominent is not obviously superior to other designs that don’t pursue goals, and instead focus on pointing at the appropriate influences from the world. For example, a system may seek to make reliable uploads, or figure out which decisions of uploads are errors, or organize uploads to make sense of situations outside normal human environments, or be corrigible in a secure way, so as to follow directions of a sane external operator and not of an attacker. Once we have enough of such details figured out (none of which is a goal-directed agent), it becomes possible to take actions in the world. At that point, we have a system of many carefully improved kluges that further many purposes in much the same way as human brains do, and it’s not clearly an improvement to restructure that system around a concept of goals, because that won’t move it closer to the influences of the world it’s designed to follow.
My guess is that agents that are not primarily goal-directed can be good at defending against goal-directed agents (especially with first mover advantage, preventing goal-directed agents from gaining power), and are potentially more tractable for alignment purposes, if humans coexist with AGIs during their development and operation (rather than only exist as computational processes inside the AGI’s goal, a situation where a goal concept becomes necessary).
I think the assumption that useful agents must be goal-directed has misled a lot of discussion of AI risk in the past. Goal-directed agents are certainly a problem, but not necessarily the solution. They are probably good for fixing astronomical waste, but maybe not AI risk.
Trying to have influence over aspects of value change that people don’t much care about … [is] reasonable … to do to make the future better
This could refer to value change in AI controllers, like Hugh in HCH, or alternatively to value change in people living in the AI-managed world. I believe the latter could be good, but the former seems very questionable (here “value” refers to true/normative/idealized preference). So it’s hard for the same people to share the two roles. How do you ensure that value change remains good in the original sense without a reference to preference in the original sense, that hasn’t experienced any value change, a reference that remains in control? And for this discussion, it seems like the values of AI controllers (or AI+controllers) is what’s relevant.
It’s agent tiling for AI+controller agents, any value change in the whole seems to be a mistake. It might be OK to change values of subagents, but the whole shouldn’t show any value drift, only instrumentally useful tradeoffs that sacrifice less important aspects of what’s done for more important aspects, but still from the point of view of unchanged original values (to the extent that they are defined at all).
I understand that there is no point examining one’s algorithm if you already execute it and see what it does.
Rather there is no point if you are not going to do anything with the results of the examination. It may be useful if you make the decision based on what you observe (about how you make the decision).
you say “nothing stops you”, but that is only possible if you could act contrary to your own algorithm, no?
You can, for a certain value of “can”. It won’t have happened, of course, but you may still decide to act contrary to how you act, two different outcomes of the same algorithm. The contradiction proves that you didn’t face the situation that triggers it in actuality, but the contradiction results precisely from deciding to act contrary to the observed way in which you act, in a situation that a priori could be actual, but is rendered counterlogical as a result of your decision. If instead you affirm the observed action, then there is no contradiction and so it’s possible that you have faced the situation in actuality. Thus the “chicken rule”, playing chicken with the universe, making the present situation impossible when you don’t like it.
So your reasoning is inaccurate
You don’t know that it’s inaccurate, you’ve just run the computation and it said $5. Maybe this didn’t actually happen, but you are considering this situation without knowing if it’s actual. If you ignore the computation, then why run it? If you run it, you need responses to all possible results, and all possible results except one are not actual, yet you should be ready to respond to them without knowing which is which. So I’m discussing what you might do for the result that says that you take the $5. And in the end, the use you make of the results is by choosing to take the $5 or the $10.
This map from predictions to decisions could be anything. It’s trivial to write an algorithm that includes such a map. Of course, if the map diagonalizes, then the predictor will fail (won’t give a prediction), but the map is your reasoning in these hypothetical situations, and the fact that the map may say anything corresponds to the fact that you may decide anything. The map doesn’t have to be identity, decision doesn’t have to reflect prediction, because you may write an algorithm where it’s not identity.
For example, in the 5&10 game an agent would examine its own algorithm, see that it leads to taking $10 and stop there.
Why do even that much if this reasoning could not be used? The question is about the reasoning that could contribute to the decision, that could describe the algorithm, and so has the option to not “stop there”. What if you see that your algorithm leads to taking the $10 and instead of stopping there, you take the $5?
Nothing stops you. This is the “chicken rule” and it solves some issues, but more importantly illustrates the possibility in how a decision algorithm can function. The fact that this is a thing is evidence that there may be something wrong with the “stop there” proposal. Specifically, you usually don’t know that your reasoning is actual, that it’s even logically possible and not part of an impossible counterfactual, but this is not a hopeless hypothetical where nothing matters. Nothing compels you to affirm what you know about your actions or conclusions, this is not a necessity in a decision making algorithm, but different things you do may have an impact on what happens, because the situation may be actual after all, depending on what happens or what you decide, or it may be predicted from within an actual situation and influence what happens there. This motivates learning to reason in and about possibly impossible situations.
What if you examine your algorithm and find that it takes the $5 instead? It could be the same algorithm that takes the $10, but you don’t know that, instead you arrive at the $5 conclusion using reasoning that could be impossible, but that you don’t know to be impossible, that you haven’t decided yet to make impossible. One way to solve the issue is to render the situation where that holds impossible, by contradicting the conclusion with your action, or in some other way. To know when to do that, you should be able to reason about and within such situations that could be impossible, or could be made impossible, including by the decisions made in them. This makes the way you reason in them relevant, even when in the end these situations don’t occur, because you don’t a priori know that they don’t occur.
(The 5-and-10 problem is not specifically about this issue, and explicit reasoning about impossible situations may be avoided, perhaps should be avoided, but my guess is that the crux in this comment thread is about things like usefulness of reasoning from within possibly impossible situations, where even your own knowledge arrived at by pure computation isn’t necessarily correct.)
so long as you can’t change their notions of status there is nothing you can do to communicate “you are fundamentally wrong about how this works” without them hearing it as “I don’t realize how far out of my depth I am right now”.
But from the other direction, it seems quite possible to hear what the wrong-status person says about how I’m wrong. So “nothing you can do” seems excessive. Perhaps politeness often suffices, for arguments that would be accepted when delivered by an appropriate-status person, as long as you are being heard at all.
These are decisions in different situations. Transitivity of preference is about a single situation. There should be three possible actions A, B and C that can be performed in a single situation, with B preferred to A and C preferred to B. Transitivity of preference says that C is then preferred to A in that same situation. Betting on a fight of B vs. A is not a situation where you could also bet on C, and would prefer to bet on C over betting on B.
It seems like a fairly straightforward claim, to say that a quality like integrity is all the more valuable for being rare.
I think the difference is between associating with producers vs. consumers. When something is more scarce, its price increases, which makes it less valuable to consumers and more valuable to producers of individual items. So for people who perceive themselves as producers of things like integrity and virginity and honest labor, scarcity would contribute to their value. And for things like medicine or the naive models of bitcoin and diamonds, decrease their value by increasing the price, since the audience of the article identify as consumers.
You can decide to question any such principles, which is how they get formulated in the first place, as designs for improved cognition devised by an evolved mind that doesn’t originally follow any particular crisp design, but can impose order on itself. The only situation where they remain stable is if the decision always comes out in their favor, which will happen if they are useful for agents pursuing your preference. When these agents become sufficiently different, they probably shouldn’t use any object level details of the design of cognition that holds for you. The design improves, so it’s not the same.
Examples of such principles are pursuit of well-calibrated empirical beliefs, of valid mathematical knowledge, of useful plans, and search for rational principles of cognition.
I don’t know how to describe the thing that remains through correct changes, which is probably what preference should be, so it’s never formal. There shouldn’t be a motivation to “be at peace” with it, since it’s exactly the thing you turn out to be at peace with, for reasons other than being at peace with it.
This is a duplicate of the post made yesterday.
Worse. It doesn’t change who you are, you are the person being blackmailed. This you know. What you don’t know is whether you exist (or ever existed). Whether you ever existed is determined by your decision.
(The distinction from the quoted sentence may matter if you put less value on the worlds of people slightly different from yourself, so you may prefer to ensure your own existence, even in a blackmailed situation, over the existence of the alternative you who is not blackmailed but who is different and so less valuable. This of course involves unstable values, but motivates degree of existence phrasing of the effect of decisions, over the change in the content of the world phrasing, since the latter doesn’t let us weigh whole alternative worlds differently.)