Yes, because I thought the why was obvious. I still do!
If doom has tiny probability, it’s better to focus on other issues. While I can’t give you a function mapping the doom mechanism to correct actions, different mechanisms of failure often require different techniques to address them—and even if they don’t, we want to check that the technique actually addresses them.
I think the question of whether doom is of moderate or tiny probability is action relevant, and also how & why doom is most likely to happen is very action relevant
You could also downweight plans that are too far from any precedent.
Ok, I guess I just read Eliezer as saying something uninteresting with a touch of negative sentiment towards neural nets.
Would you say Yudkowsky’s views are a mischaracterisation of neural network proponents, or that he’s mistaken about the power of loose analogies?
In contrast, I think we can explain humans’ tendency to like ice cream using the standard language of reinforcement learning.
I think you could defend a stronger claim (albeit you’d have to expend some effort): misgeneralisation of this kind is a predictable consequence of the evolution “training paradigm”, and would in fact be predicted by machine learning practitioners. I think the fact that the failure is soft (humans don’t eat ice cream until they die) might be harder to predict than the fact that the failure occurs.
I do not think that optimizing a network on a given objective function produces goals orientated towards maximizing that objective function. In fact, I think that this almost never happens. For example, I don’t think GPTs have any sort of inner desire to predict text really well.
I think this is looking at the question in the wrong way. From a behaviourist viewpoint:
it considers all of the possible 1-token completions of a piece of text
then selects the most likely one (or randomises according to its distribution or something similar)
on this account, it “wants to predict text accurately”. But Yudkowsky’s claim is (roughly):
it considers all of the possible long run interaction outcomes
it selects the completion that leads to the lowest predictive loss for the machine’s outputs across the entire interaction
and perhaps in this alternative sense it “wants to predict text accurately”.
I’d say the first behaviour has high priors and strong evidence, and the second is (apparently?) supported by the fact that both behaviours are compatible with the vague statement “wants to predict text accurately”, which I don’t think is very compelling.
My response in Why aren’t other people as pessimistic as Yudkowsky? includes a discussion of adversarial vulnerability and why I don’t think points to any irreconcilable flaws in current alignment techniques.
I think this might be the wrong link. Either that, or I’m confused about how the sentence relates to the podcast video.
Not that I know of. People talk about raccoon dogs as a candidate for market spillover, not bats
I think that if RLHF reduced to a proper loss on factual questions, these probabilities would coincide (given enough varied training data). I agree it’s not entirely obvious that having these probabilities come apart is problematic, because you might recover more calibrated probabilities by asking for them. Still, knowing the logits are directly incentivised to be well calibrated seems like a nice property to have.
An agent says yes if it thinks yes is the best thing to say. This comes apart from “yes is the correct answer” only if there are additional considerations determining “best” apart from factuality. If you’re restricted to “yes/no”, then for most normal questions I think an ideal RLHF objective should not introduce considerations beyond factuality in assessing the quality of the answer—and I suspect this is also true in practical RLHF objectives. If I’m giving verbal confidences, then there are non-factual considerations at play—namely, I want my answer to communicate my epistemic state. For pretrained models, the question is not whether it is factual but whether someone would say it (though somehow it seems to come close). But for yes/no questions under RLHF, if the probabilities come apart it is due to not properly eliciting the probability (or some failure of the RLHF objective to incentivise factual answers).
I still think it’s curious that RLHF doesn’t seem to reduce to a proper loss on factual questions, and I’d guess that it’d probably be better if it did (at least, with contexts that strictly ask for a “yes/no” answer without qualification)
Yeah, I saw that—I’m wondering if previous models benefitted more from RLHF
Did gpt 3.5 get high scores on human exams before fine tuning? My rough impression is “gpt4 relies less on fine tuning for its capabilities”
A few reflections on this piece:
It helped me understand Nate’s view. I’d read previous examples of his suggesting “cutting edge scientific work necessitates CIS pursuit” (e.g. the laser in the fog), but it wasn’t clear to me how important he considered these examples
Theories about how ML generalises play a substantial role in everyone’s thinking here. AFAIK we don’t have precise theories that are much help in this regard, and in practice people usually use imprecise theories. Just because they’re imprecise doesn’t mean they can’t be explained. Perhaps some effort into explicating theories of ML generalisation could help everyone understand one another.
As AI gets more powerful, it makes sense that AI systems will make higher and higher level decisions about the objectives that they pursue. Holden & Nate focusing on the benchmark of “needle moving scientific research” seems to suggest agreement on the following:
In order to sustain the trend of more powerful AI making higher and higher level decisions, we will need substantial innovation in our technologies for AI control
The rate of innovation possible under business as usual human science seems unlikely to keep up with this need
Thus we require AI acceleration of AI control science
Regarding this last point: it’s not clear to me whether slow progress in AI control systems will lead to slow progress in AI making higher and higher level decisions or not. That is, it’s not obvious to me that AI control systems failing to keep up necessarily leads to catastrophe. I acknowledge that very powerful AI systems may seem to work well with poor control technologies, but I’m uncertain about whether moderately powerful AI systems work well enough with poor control technologies for the very powerful systems to be produced (and also what the relevant levels of power are, compared to today’s systems).
One more thing: I’m suspicious of equivocation between “some convergent instrumental sub goals” and “worrisome convergent instrumental sub goals”. There are probably many collections of CISs that aren’t similar to the worrisome ones in the machine’s picture of the world.
And another more thing:
To be clear, I think both Nate and I are talking about a pretty “thin” version of POUDA-avoidance here, more like “Don’t do egregiously awful things” than like “Pursue the glorious transhumanist future.” Possibly Nate considers it harder to get the former without the latter than I do.
I’m still unsure how much pivotal act considerations weigh in Nate/MIRI’s views. My view is roughly:
Cutting edge scientific work without disastrous instrumental behaviour seems pretty attainable
Unilaterally preventing anyone else from building AI seems much more likely to entail disastrous instrumental behaviour
and I can easily imagine finding it difficult to be confident you’re avoiding any catastrophes if you’re aiming for the second.
What coherence theorem do you have in mind that has these implications?
For that matter, what implications are you referring to?
Here’s a hypothesis about the inverse correlation arising from your observation: When we evaluate a thing’s coherence, we sample behaviours in environments we expect to find the thing in. More intelligent things operate in a wider variety of environments, and the environmental diversity leads to behavioural diversity that we attribute to a lack of coherence.
Sure, and that’s why I said “coherence” instead of coherence
This is a cool result—I think it’s really not obvious why intelligence and “coherence” seem inversely correlated, but it’s interesting that you replicated it across three different classes of things (ML models, animals, organisations).
The shoggoth is not an average over masks. If you want to see the shoggoth, stop looking at the text on the screen and look at the input token sequence and then the logits that the model spits out. That’s what I mean by the behavior of the shoggoth.
We can definitely implement a probability distribution over text as a mixture of text generating agents. I doubt that an LLM is well understood as such in all respects, but thinking of a language model as a mixture of generators is not necessarily a type error.
The logits and the text on the screen cooperate to implement the LLM’s cognition. Its outputs are generated by an iterated process of modelling completions, sampling them, then feeding the sampled completions back back to the model.
So, if I’m understanding you correctly:
if it’s possible to build a single AI system that executes a catastrophic takeover (via self-bootstrap or whatever), it’s also probably possible to build a single aligned sovereign, and so in this situation winning once is sufficient
if it is not possible to build a single aligned sovereign, then it’s probably also not possible to build a single system that executes a catastrophic takeover and so the proposition that the model only has to win once is not true in any straightforward way
in this case, we might be able to think of “composite AI systems” that can catastrophically take over or end the acute risk period, and for similar reasons as in the first scenario, winning once with a composite system is sufficient, but such systems are not built from single acts
and you think the second scenario is more likely than the first.
A quick guess is that at about 1 in 10 000 chance of AI doom working on it is about as good as ETG to GiveWell top charities