I suppose one argument could be: Humans are a kind of general intelligence. Humans tend to make this mistake. So making this mistake doesn’t show that we are far away from human-level intelligence.
If anything, this seems to be evidence for a third position, rather for either scale is all you need or language models being really unimpressive.
In the section: “The role of naturalized induction in decision theory” a lot of variables seem to be missing.
(Evolution) → (human values) is not the only case of inner alignment failure which we know about. I have argued that human values themselves are inner alignment failures on the human reward system. This has happened billions of times in slightly different learning setups.
I expect that it has also happened to an extent with animals as well. I wonder if anyone has ever looked into this.
converge to
Converge to 1? (Context is “9. Non-Dogmatic...”).
Anyway, thanks so much for writing this! I found this to be a very useful resource.
Have you considered the possibility of this initiative creating x-risk?
“The plan suggests a relatively narrow AI would be sufficient to contribute to alignment research, while being too narrow to be dangerous. This seems implausible: Much easier problems than alignment research have been called AGI-complete, and general reasoning ability is widely thought to be a requirement for doing research”—Could you clarify what some of these easier problems are and why they are AGI-complete?
I don’t need the experiments to persuade me, so I’m not in the target audience. I’m trying to think about what people who are more critical might want to see.
One key question I had (apologies if you already covered this as I only skimmed this post), is how you’re modeling earplugs? To be most persuasive, you’d want the earplugs to have been build using dynamics it could already make sense of in the world instead of just being “magic”, however that seems hard.
Perhaps one option would be to have a transmitter that can be destroyed during test, but not during training, because they would differ in the material they are made out of? For example, if the AI learns during training that wooden objects can be destroyed, but metal objects can’t be.
If we want to make it easier for the AI to learn, perhaps we create a few scenarios in training where it sees a transmitter spontaneously explode so it sees that if the transmitter is destroyed then the signal will end, but it can’t destroy the transmitter itself. Then we could try to get an AI to learn on hard mode where it never sees the transmitter interfered with during training.
Anyway, I’d be keen to hear what you think about this scheme.
Fantastic post.
I’ve recently been leaning more in the direction of there being multiple coherent approaches to the Counterfactual Mugging problem, but I was still feeling quite confused by it. I really think that the multi-agent perspective adds a lot of clarity.
Cousin_it and I discovered the Counterfactual Prisoner’s Dilemma, which it similar to the Counterfactual Mugging, but presents a case where the kind of agent that pay attention to their counterfactual selves does better in each arm. I think it presents a limitation to thinking of yourself as completely disconnected from agents in other branches, but I suppose we can still model the situation as independent agents with some kind of subjunctive link.
You may find it interesting that I’ve also argued in favour of viewing counterfactuals as circular here and that I even ran a competition on this last year.
I’m already finding it useful in terms of writing posts, especially when I have a paragraph, but it doesn’t quite flow right. I feel it makes the writing process so much easier.
Another risk is that if we broaden the focus too much, then that distracts us from preventing existential risks.
It seems strange to treat ontological crises as a subset of embedded world-models, as it seems as though a Cartesian agent could face the same issues?
I asked my friend for a resource that explained the 5-and-10 problem well and he provided this link. Unfortunately, I still don’t have a good link for “Playing Chicken with the universe”.
I wish that we were clearer that in a lot of circumstances we don’t actually need a utility maximiser for our argument, but rather an AI that is optimising sufficiently hard. Unfortunately, I suspect competitive dynamics and the unilateralist’s curse might push us further down this path than we’d like.
Fair enough! Would be keen to hear your thoughts here.
Yeah, I’m worried that copyright restrictions might slow down responsible projects in Western countries vs. less responsible projects in jurisdictions that we are likely to have less influence over.
Have you read much about the concerns people have about AI safety on Less Wrong because if you agree with those concerns that would likely change your answer here!
Feel free to try if you want, though I’d bet you’d just be making things harder for yourself.
I thought this was a really important point, although I might be biased because I was finding it confusing how some discussions were talking about the gradient landscape as though it could be modified and not clarifying the source of this (for example, whether they were discussing reinforcement learning).