See my other reply.
I am constantly flipping back and forth between “I have terrible social skills” and “People only think I am smart and competent because I have charmed them with my awesome social skills”.
Yup. The point is just that there’s a level of intelligence past which, it seems, we can’t do literally anything to get things back on track. Even if we have some theoretically-perfect tools for dealing with it, these tools’ software and physical implementations are guaranteed to have some flaws that a sufficiently smart adversary will be able to arbitrarily exploit given any high-bandwidth access to them. And at that point, even some very benign-seeming interaction with it will be fatal.
This transition point may be easy to miss, too — consider, e. g., the hypothesized sharp capabilities gain. A minimally dangerous system may scale to a maximally dangerous one in the blink of an eye, relatively speaking, and we need to be mindful of that.
Besides, some of the currently-proposed alignment techniques may be able to deal with a minimally-dangerous system if it doesn’t scale. But if it does, and if we let any degree of misalignment persist into that stage, not even the most ambitious solutions would help us then.
No idea lol, just passing it along
Not necessarily. It’s routed through the interpretability channel, but the idea is that the AI would be able to take over whatever system is running the interpretability tool through that channel (either by shaping the tool’s direct outputs, or by manipulating the computational side-effects of that tool being ran). So it won’t be limited to controlling the tool’s output, but the system’s output.
I guess we can homomorphically encrypt the training process and then only decrypt the interpretability results on a separate air-gapped computer whose outputs we’re not using for anything else. In that case, yeah, the AI would only be able to influence us through the interpretability results. (Which wouldn’t be hard or anything. Presumably we’re doing that because some of these results we would judge as “good” and give the go-ahead to continue the training/release the AI based on them. The AI will make sure we get such results.)
Unfortunately, this feels like a subject that’s often discussed in asides. I have a feeling this came up more than once during the 2021 MIRI Conversations, but could be misremembering.
This is true probably for some extremely high level of superintelligence, but I expect much stupider systems to kill us if any do; I think human level ish AGI is already a serious x risk, and humans aren’t even close to being intelligent enough to do this.
I copied our discussion into my PKM, and I’m wondering how to tag it… it’s certainly meta, but we’re discussing multiple levels of abstraction. We’re not at level N discussing level N-1, we’re looking at the hierarchy of levels from outside the hierarchy. Outside, not necessarily above. This reinforces my notion that structure should emerge from content, as opposed to trying to fit new content into a pre-existing structure.
What are some of the “various things” you have in mind here? It seems possible to me that something like “AI alignment testing” is straightforwardly upstream of what players want, but maybe you were thinking of something else
Looks legit, but is this leak of any real interest? Like, Stable Diffusion is set to be released as open source, right? So this just speeds things along slightly that were already going to happen.
How will you compete in the market while optimizing for various things other than what players want?
Just to confirm, the social engineering of humans is through the intepretability channel correct? As in humans choose to look at some statistics during the model’s training (not the model output) and these stats too are able to convey information that is very persuasive to humans.
Now that I think about it, I think you and I are misinterpreting what johnwentsworth meant when he said “optimization pressure”. I think he just means a measure of how much we have to change the world by, in units of bits, and not any specific piece of information that the AI or alignment researchers produce.
I also think that intelligent agents throw off this kind of analysis. For example, suppose you enter a room and find that I have 10 coins laid out on a table, all heads. Did this happen by chance? It’s unlikely: 1 in 1024. Except obviously I arranged them all to be heads, I didn’t just get lucky. Because I’m intelligent, I can make the probability whatever I want.
I was thinking RL systems for the case where an agent learns the correct outcome to optimize for but in the wrong environment, but the same issue applies for mesa-optimizers within any neural net.
As for why it tries to restart the training environment, it needs a similar environment to meet a goal that is only defined within that environment. If the part that’s unclear is what a training environment means for something like a neural net trained with supervised learning, the analogy would be that the AI can somehow differentiate between training data (or a subset of it) and deployment data and wants to produce its outputs from inputs with the training qualities.
It sounds like the game will be an MMO, is that correct?
Subtle point: the key question is not how certain we are, but how certain the predictor system (e.g. GPT) is. Presumably if it’s able to generalize that far out of distribution at all, it’s likely to have enough understanding to make a pretty high-confidence guess as to whether AGI will take over or not. We humans might not know the answer very confidently, but an AI capable enough to apply the human mimicry strategy usefully is more likely to know the answer very confidently, whatever that answer is.
Thanks for thinking it over, and I agree with your assessment that this is better public knowlege than private :)
The Facebook newsfeed also got a lot of opposition in the beginning. Whether or not a very innovative product is good or bad can not be decided on whether people like to oppose or make fun of it in the beginning.
BlenderBot 3 is likely running on substantially fewer resources than GPT-2/3 and thus should not be expected to have the same performance.