If you have the time look up “Terence Tao” on Gwern’s website.
In case anyone else is going looking, here is the relevant account of Tao as a child and here is a screenshot of the most relevant part:
mattmacdermott
Karma: 1,222
Is instrumental convergence a thing for virtue-driven agents?
Why does ChatGPT voice-to-text keep translating me into Welsh?
I use ChatGPT voice-to-text all the time. About 1% of the time, the message I record in English gets seemingly-roughly-correctly translated into Welsh, and ChatGPT replies in Welsh. Sometimes my messages go back to English on the next message, and sometimes they stay in Welsh for a while. Has anyone else experienced this?
Example: https://chatgpt.com/share/67e1f11e-4624-800a-b9cd-70dee98c6d4e
I think the commenter is asking something a bit different—about the distribution of tasks rather than the success rate. My variant of this question: is your set of tasks supposed to be an unbiased sample of the tasks a knowledge worker faces, so that if I see a 50% success rate on 1 hour long tasks I can read it as a 50% success rate on average across all of the tasks any knowledge worker faces?
Or is it more impressive than that because the tasks are selected to be moderately interesting, or less impressive because they’re selected to be measurable, etc
I vaguely remember a LessWrong comment from you a couple of years ago saying that you included Agent Foundations in the AGISF course as a compromise despite not thinking it’s a useful research direction.
Could you say something about why you’ve changed your mind, or what the nuance is if you haven’t?
I’d be curious to know if conditioning on high agreement alone had less of this effect than conditioning on high karma alone (because something many people agree on is unlikely to be a claim of novel evidence, and more likely to be a take.
Flagging for posterity that we had a long discussion about this via another medium and I was not convinced.
More or less, yes. But I don’t think it suggests there might be other prompts around that unlock similar improvements—chain-of-thought works because it allows the model to spend more serial compute on a problem, rather than because of something really important about the words.
Agree that pauses are a clearer line. But even if a pause and tool-limit are both temporary, we should expect the full pause to have to last longer.
One difference is that keeping AI a tool might be a temporary strategy until you can use the tool AI to solve whatever safety problems apply to non-tool AI. In that case the co-ordination problem isn’t as difficult because you might just need to get the smallish pool of leading actors to co-ordinate for a while, rather than everyone to coordinate indefinitely.
I now suspect that there is a pretty real and non-vacuous sense in which deep learning is approximated Solomonoff induction.
Even granting that, do you think the same applies to the cognition of an AI created using deep learning—is it approximating Solomonoff induction when presented with a new problem at inference time?
I think it’s not, for reasons like the ones in aysja’s comment.
Agreed, this only matters in the regime where some but not all of your ideas will work. But even in alignment-is-easy worlds, I doubt literally everything will work, so testing would still be helpful.
I wrote it out as a post here.
Validating against a misalignment detector is very different to training against one
I think it’s downstream of the spread of hypotheses discussed in this post, such that we can make faster progress on it once we’ve made progress eliminating hypotheses from this list.
Fair enough, yeah—this seems like a very reasonable angle of attack.
It seems to me that the consequentialist vs virtue-driven axis is mostly orthogonal to the hypotheses here.
As written, aren’t Hypothesis 1: Written goal specification, Hypothesis 2: Developer-intended goals, and Hypothesis 3: Unintended version of written goals and/or human intentions all compatible with either kind of AI?
Hypothesis 4: Reward/reinforcement does assume a consequentialist, and so does Hypothesis 5: Proxies and/or instrumentally convergent goals as written, although it seems like ‘proxy virtues’ could maybe be a thing too?
(Unrelatedly, it’s not that natural to me to group proxy goals with instrumentally convergent goals, but maybe I’m missing something).
Maybe I shouldn’t have used “Goals” as the term of art for this post, but rather “Traits?” or “Principles?” Or “Virtues.”
I probably wouldn’t prefer any of those to goals. I might use “Motivations”, but I also think it’s ok to use goals in this broader way and “consequentialist goals” when you want to make the distinction.
One thing that might be missing from this analysis is explicitly thinking about whether the AI is likely to be driven by consequentialist goals.
In this post you use ‘goals’ in quite a broad way, so as to include stuff like virtues (e.g. “always be honest”). But we might want to carefully distinguish scenarios in which the AI is primarily motivated by consequentialist goals from ones where it’s motivated primarily by things like virtues, habits, or rules.
This would be the most important axis to hypothesise about if it was the case that instrumental convergence applies to consequentialist goals but not to things like virtues. Like, I think it’s plausible that
(i) if you get an AI with a slightly wrong consequentialist goal (e.g. “maximise everyone’s schmellbeing”) then you get paperclipped because of instrumental convergence,
(ii) if you get an AI that tries to embody slightly the wrong virtue (e.g. “always be schmonest”) then it’s badly dysfunctional but doesn’t directly entail a disaster.
And if that’s correct, then we should care about the question “Will the AI’s goals be consequentialist ones?” more than most questions about them.
You know you’re feeling the AGI when a compelling answer to “What’s the best argument for very short AI timelines?” lengthens your timelines
Interesting. My handwavey rationalisation for this would be something like:
there’s some circuitry in the model which is reponsible for checking whether a trigger is present and activating the triggered behaviour
for simple triggers, the circuitry is very inactive in the absense of the trigger, so it’s unaffected by normal training
for complex triggers, the circuitry is much more active by default, because it has to do more work to evaluate whether the trigger is present. so it’s more affected by normal training
I think this gets at the heart of the question (but doesn’t consider the other possible answer). Does a powerful virtue-driven agent optimise hard now for its ability to embody that virtue in the future? Or does it just kinda chill and embody the virtue now, sacrificing some of its ability to embody it extra-hard in the future?
I guess both are conceivable, so perhaps I do need to give an argument why we might expect some kind of virtue-driven AI in the first place, and see which kind that argument suggests.