Analytic philosopher with a background in physics, now working in AI.
Peter Kuhn
Karma: 42
Agree. If we had evidence that human reports (and judgements) about their internal states are more or less random this would constitute evidence that our sense of being conscious is probably utterly fictitious. But we just don’t have such evidence.
This post has gotten a lot of upvotes so I fear I am not getting something.
It still seems to me that the experiment does put considerable stress on the assumption that LLM talk about their experiences should make us infer anything about actual experiences. You correctly point out that the result may not be very surprising, given the architecture of LLMs. But that seems to be an independent point.
Thanks. I checked determinism on Mistral using a simple script (see github link), but the random seed is a better suggestion and I might do that on the weekend and post an update ;)
I think the interesting discussion is not about how certain exactly our predictions of doom are or can be.
Let me put the central point another way: However pessimistic you are about the success of alignment you should become more pessimistic once you realize that alignment requires the prediction of an AIs actions. Any notion that we could circumvent this by engineering values into the system is illusory.
The primary thing I am arguing, as Max as already said, is that the AI alignment paradigm obscures the most fundamental problem of AI safety: That of prediction. It does this by conflating various interpretations of what values or utility functions are.
One of the most fundamental insights entailed by a move from an alignment to a predictive paradigm is that is becomes far from clear whether the relevant problems are solvable.
Nothing in this shows that AGI is “guaranteed to be destructive to human preferences” of course. Rather, it shows that various paradigms that one may choose to try to make AI safe, like RLHF, should actually not make us more confident in our AGI systems at all because they address the wrong questions: We can never know in advance whether they will work and this is true for all paradigms that try to sidestep the prediction problem by appealing to values (bracketing hard-wired values of course).
I have problems getting the first point. If bugs are hard to find then shouldn’t this precisely entail that dangerous AI is hard to differentiate from benign AI?! Any literature you can suggest on the subject?
Regarding the second point. I don’t find Eliezer’s idea entirely convincing. But I don’t think the fire thesis hinges on his view. Rather, it is built on the much weaker and simpler view that if we don’t know the utility function of some AGI system then this system is dangerous—I find it very hard to see any convincing reasons for thinking this is false. Eliezer thinks doom is default. I just assume that ignorance makes it rational to air on the side of caution.
One could actually check this alternative hypothesis, too. The internal states being proposed work like hidden variables in quantum mechanics. One could simply check if the LLM’s answers over multiple runs violate the Bell inequalities. Or in less fancy terms: check if the distribution over counterfactual branches can be explained by any distribution over numbers that is stable across branches.