The things avoided seem like they increase, not decrease risk.
Yes, but it might be that the means needed to avoid them—maybe heavy-handed AI interventions? - could be even more dangerous.
Stuart_Armstrong(Stuart Armstrong)
Karma: 16,150
The second one (though I think there is some overlap with the first).
It is indeed our moral intuitions that are underdefined, not the states of the universe.
Morally underdefined situations can be deadly
You need to add some assumptions to make it work. For example, I believe the following works:
“In second order arithmetic, we can prove that NP1 implies NF, where NP1 is the statement ‘there exists no first order proof of the conjecture’ and NF is the statement ‘the conjecture isn’t false’.”
Because our preferences are inconsistent, and if an AI says “your true preferences are ”, we’re likely to react by saying “no! No machine will tell me what my preferences are. My true preferences are , which are different in subtle ways”.
I see this issue as being a fundamental problem:
Thanks for developing the argument. This is very useful.
The key point seems to be whether we can develop an AI that can successfully behave as a low impact AI—not as a “on balance, things are ok”, but a genuinely low impact AI that ensure that we don’t move towards a world where our preference might be ambiguous or underdefined.
But consider the following scenario: the AGI knows that, as a consequence of its actions, one AGI design will be deployed rather than another. Both of these designs will push the world into uncharted territory. How should it deal with that situation?
The successor problem is important, but it assumes we have the values already.
I’m imagining algorithms designing successors with imperfect values (that they know to be imperfect). It’s a somewhat different problem (though solving the classical successor problem is also important).
My thought is that when deciding to take a morally neutral act with tradeoffs, the AI needs to be able to balance the positive and negative to get a reasonable acceptable tradeoff, and hence needs to know both positive and negative human values to achieve that.
I agree there are superintelligent unconstrained AIs that can accomplish tasks (making a cup of tea) without destroying the world. But I feel it would have to have so much of human preferences already (to compute what is and what isn’t an acceptable tradeoff in making you your cup of tea) that it may as well be fully aligned anyway—very little remains to define full alignment.
General alignment plus human values, or alignment via human values?
Thanks! Lots of useful thoughts here.
Beyond the human training distribution: would the AI CEO create almost-illegal teddies?
Classical symbol grounding and causal graphs
Cool, thanks; already in contact with them.
Those are very relevant to this project, thanks. I want to see how far we can push these approaches; maybe some people you know would like to take part?
For the moment, I’m going to be trying to resolve practical questions of model splintering, and then I’ll see if this formalism turns out to be useful for them.
Vertigo, lust, pain reactions, some fear responses, and so on, don’t involve a model. Some versions of “learning that it’s cold outside” don’t involve a model, just looking out and shivering; the model aspect comes in when you start reasoning about what to do about it. People often drive to work without consciously modelling anything on the way.
Think model-based learning versus Q-learning. Anything that’s more Q-learning is not model based.
I agree with you; this is an old post that I don’t really agree with any more.