Vladimir_Nesov
Karma: 20,486
NewTop
This is a duplicate of the post made yesterday.
I don’t know how shminux interprets the words, and if the question was related to this, but there is an issue in your use of “impossible”. Things that happen in this world are actual, and things that happen in alternative worlds are possible. (The set of alternative worlds needs to be defined for each question.) An impossible situation is one that can’t occur, that doesn’t happen in any of the alternative worlds.
Thus outputs of an algorithm different from the actual output are not actual, and furthermore not possible, as they don’t occur in the alternative worlds with the same algorithm receiving the same inputs. But there are alternative worlds (whose set is different than in the previous sentence, no longer constrained by the state of input) with the same algorithm that receives different inputs, and so despite not being actual, the different inputs are still possible, in other words not impossible.
Bug report: Posts by Day page shows incorrect URLs for most posts, for example it lists the post “The Rocket Alignment Problem” with a link to Fasting Mimicking Diet Looks Pretty Good. Looks like some kind of intermittent off-by-one error that only affects some of the entries. The entries on the same page loaded with the “Load More Days” link don’t seem to be affected. I’m not seeing this issue on the front page, and it only appeared around yesterday. Interestingly, if I click the “All Posts/Daily” link on the front page, which leads to the same URL, then there is no issue; it’s only reproduced if I open the URL directly.
There’s an archived copy here.
(That may be a useful clue for identifying the meaning of the question, as understood by the people pursuing it, but not necessarily a good reason to agree that it currently should be considered mysterious or that it’s a sensible question to pursue.)
[W]e still need a coherent moral framework to use to generate our AI’s utility function if we want it to be aligned, so morality is important, and we do need to develop an acceptable solution to it.
This is not clear. It’s possible for the current world to exist as it is, and similarly for any other lawful simulated world that’s not optimized in any particular direction. So an AI could set up such a world without interfering, and defend its lawful operation from outside interference. This is a purposeful thing, potentially as good at defending itself as any other AI, that sets up a world that’s not optimized by it, that doesn’t need morality for the world it maintains, in order to maintain it. Of course this doesn’t solve any problems inside that world, and it’s unclear how to make such a thing, but it illustrates the problems with the “morality is necessary for AGI” position. Corrigibility also fits this description, being something unlike an optimization goal for the world, but still a purpose.
The concept is fine, it’s the misuse of the words denoting it that can cause problems by diluting the concept, giving bad training data so that the concept gets learned in many distorted forms. It might be useful to taboo the words that denote the concept in order to make explaining it easier. Or just refer people to a blog post somewhere that explains the idea, it’s the same as any knowledge.
It’s also not necessary to show people how to be an effective altruist (or to believe or assert that it’s a good idea) in order to explain the concept.
I write notes in a single plain text file, using the dates they are made to cite them in newer notes. There are two types of notes, brainstorming throw-away ones that maintain the process of thinking about a problem or of learning something (such as carefully reading a paper), and more lucid ones, with some re-reading value, which are marked differently and have a one-sentence summary. The notes are intended to never be made public, so that I feel free to use them to resolve any silly confusions.
The useful question is about value of the data that can be collected about people, not so much its usefulness for achieving a particular task, because it may no longer make sense to perform that task once it becomes possible to do so (as in giant cheesecake fallacy). A system that can reconstruct people from data can do many other things that may be more valuable than reconstruction of people even from the point of view of these people that could’ve been reconstructed. It’s a question of what should be done with the resources and capabilities of that system.
The value of the data is in its contribution to the value of the best things that can be made, and these best things don’t necessarily improve through availability of that data, because they are not necessarily reconstructed people. I’m not sure there is any knowable-on-human-level difference in value between what can be done with the world given the knowledge about particular people who used to live in the past (either through indirect data or cryopreserved brains), compared to the value of what can be done without that knowledge. I guess it can’t hurt if it doesn’t consume resources that could otherwise find a meaningful purpose.
I worry there might be leaks in logical time that let the agent choose an action that takes into account that an impactful action will be denied. For example, a sub-agent could be built so that it’s a maximizer that’s not constrained by an impact measure. The sub-agent then notices that to maximize its goal, it must constrain its impact, or else the main agent won’t be allowed to create it. And so it will so constrain its impact and will be allowed to be created, as a low-impact and maximally useful action of the main agent. It’s sort of a daemon, but with respect to impact measure and not goals, which additionally does respect the impact measure and only circumvents it once in order to get created.
It could as easily be “do this one slightly helpful thing”, an addition on top of doing nothing. It doesn’t seem like there is an essential distinction between such different framings of the same outcome that intent verification can capture.
I was talking about what I understand the purpose/design of intent verification to be, not specifically the formalizations you described. (I don’t think it’s particularly useful to work out the details without a general plan or expectation of important technical surprises.)
It’s Rice’s theorem, though really more about conceptual ambiguity. We can talk about particular notions of agents or goals, but it’s never fully general, unless we by construction ensure that unexpected things can’t occur. And even then it’s not what we would have wanted the notions of agents or goals to be, because it’s not clear what that is.
Intent verification doesn’t seem to capture things that smuggle in a tiny bit of helpfulness when these things are actually required to deliver that helpfulness, especially after other routes to improving the outcome have been exhausted (this is what the paragraph about hashes in the first comment was about). So the neutral magic could be helpful a tiny, bounded amount. This is one of the ways the balance between doing nothing and releasing a sub-agent could be broken.
Unleashing this agent would change resource availability and increase or decrease the power of an arbitrary maximizer from that vantage point.
It’ll replenish the resources it takes, help any maximizer it impedes so as to exactly cancel out the impediment etc.
Suppose that an arbitrary maximizer could not co-opt this new agent - its ability to achieve goals is decreased compared to if it hadn’t activated the agent.
To the extent its existence could pose a problem for another agent (according to the measure, which can’t really talk about goals of agents-in-general), it’ll surrender its resources without a fight or help with the other agent’s goals. It will destroy any surplus in its resources before it can be captured. A change in variance resulting from unpredictability of such outcomes can be similarly targeted.
In general, these seem like the details that neutral magic can work out as their possibility occurs to it, so the difficuly is shifted from the main agent to the trusted ability of its sub-agent to do the rest. Worst case scenario, it can self-destruct early, but there’s probably enough leeway from that outcome.
The sub-agent in this scenario won’t be helping the main agent with achieving any goals. It only hides. Its nature precludes any usefulness. It’s no more useful than its absence. But for the main agent, it’s as easy to set up as its absence. And there might be reasons for this balance to be broken in favor of creating the sub-agent.
If we want to argue this, we should first come up with a terrible x-scenario which is low objective impact.
Here is a scenario that I think low impact methods can’t address. Let’s say the AI has the opportunity to easily (without affecting its ability to do its task) create a “free” copy or a sub-agent on the Internet. That sub-agent can be tasked with pursuing the goal of restricting whatever measure of impact the main agent operates under. The sub-agents very carefully eats the universe, while ensuring that according to the impact measure there is no effect of its activity. It has no other purpose. Any resources it needs, it can gather where they would’ve been wasted without its intervention (like reaching distant stars before other agents and capturing some of their output that wouldn’t otherwise be captured).
This functions like “neutral magic” in the environment where everything becomes weird, agents lurking in the shadows, but anything that was thought of in the impact measure is no different than without this magic. Of course, this is very bad for Goodhart reasons, since no impact measure is perfect. (This is like the example with an explosion in the desert, but everywhere, harder to notice exactly where it would become noticeable.)
A general technique to avoid this might be a way of always keeping the agent busy, computing hashes or something, even when it’s done with its task (maybe as a lexicographically lower goal), so that it doesn’t do something like that because it can. This also looks like an impact measure.
(This is related to how impact measure is close to being a goal (transformation of goals), so the failure modes of misalignment apply to it as well. I think there is a useful distinction between goals and corrigibility, which might be reproduced for something like low impact, as a third kind of thing that an agent might pursue, which is neither a goal nor a corrigibility.)
There are pre-formal facts about what words should mean, or what meanings to place in the context where these words may be used. You test a possible definition against the word’s role in the story, and see if it’s apt. This makes use of facts outside any given definition, just as with the real world.
And here, it’s not even clear what the original definitions of agents should be capable of, if you step outside particular decision theories and look at the data they could have available to them. Open source game theory doesn’t require anything fundamentally new that a straightforward idealization of an agent won’t automatically represent. It’s just that the classical decision theories will discard that data in their abstraction of agents. In Newcomb’s problem, it’s essentially discarding part of the problem statement, which is a strange thing to expect of a good definition of an agent that needs to work on the problem.
Not every belief is about truth. You may hold X to be useful because it’s useful, and Y to be true because it’s true. The mistake is to automatically hold X to be true as well, or Y to be useful. Restating that as “X being useful is true” erases the distinction or invites unnecessary rigor. In the same vein, there are judgements of falsehood or necessity or need, and these are about falsehood or necessity or need and not about truth.
I’m not sure what you changed your mind about. Some of the examples you give are unconvincing, as they do have simple meta-algorithms that both discover the more complicated better solutions and analyse their behavior. My guess is that the point is that for example looking into nuance of things like decision theory is an endless pursuit, with more and more complicated solutions accounting for more and more unusual aspects of situations (that can no longer be judged as clearly superior), and no simple meta-algorithm that could’ve found these more complicated solutions, because it wouldn’t know what to look for. But that’s content of values, the thing you look for in human behavior, and we need at least a poor solution to the problem of making use of that. Perhaps you mean that even this poor solution is too complicated for humans to discover?
You can decide to question any such principles, which is how they get formulated in the first place, as designs for improved cognition devised by an evolved mind that doesn’t originally follow any particular crisp design, but can impose order on itself. The only situation where they remain stable is if the decision always comes out in their favor, which will happen if they are useful for agents pursuing your preference. When these agents become sufficiently different, they probably shouldn’t use any object level details of the design of cognition that holds for you. The design improves, so it’s not the same.
Examples of such principles are pursuit of well-calibrated empirical beliefs, of valid mathematical knowledge, of useful plans, and search for rational principles of cognition.
I don’t know how to describe the thing that remains through correct changes, which is probably what preference should be, so it’s never formal. There shouldn’t be a motivation to “be at peace” with it, since it’s exactly the thing you turn out to be at peace with, for reasons other than being at peace with it.