EA and AI safety
AI safety meta research at Nonlinear
Theoretical Alignment research at SERI MATS
Yeah, I totally agree. My motivation for writing the first section was that people use the word ‘deception’ to refer to both things, and then make what seem like incorrect inferences. For example, current ML systems do the ‘Goodhart deception’ thing, but then I’ve heard people use this to imply that it might be doing ‘consequentialist deception’.
These two things seem close to unrelated, except for the fact that ‘Goodhart deception’ shows us that AI systems are capable of ‘tricking’ humans.
This phrase is mainly used and known by the euthanasia advocacy movement. This might also be bad, but not as bad as the association with Jim Jones.
Will this happen? Bet here! https://manifold.markets/PeterBarnett/will-someone-recieve-funding-to-inv
My initial thought is that I don’t see why this powerful optimizer would attempt to optimize things in the world, rather than just do some search thing internally.
I agree with your framing of “how did this thing get made, since we’re not allowed to just postulate it into existence?”. I can imagine a language model which manages to output words which causes strokes in whoever reads its outputs, but I think you’d need a pretty strong case for why this would be made in practice by the training process.
Say you have some powerful optimizer language model which answers questions. If you ask a question which is off its training distribution, I would expect it to either answer the question really well (e.g. it genealises properly), or it kinda breaks and answers the question badly. I don’t expect it to break in such a way where it suddenly decides to optimize for things in the real world. This would seem like a very strange jump to make, from ‘answer questions well’ to ‘attempt to change the state of the world according to some goal’.
But I think if we trained the LM on ‘have a good on-going conversation with a human’, such that the model was trained with reward over time, and its behaviour would effect its inputs (because it’s a conversation), then I think it might do dangerous optimization, because it was already performing optimization to affect the state of the world. And so a distributional shift could cause this goal optimization to be ‘pointed in the wrong direction’, or uncover places where the human and AI goals become unaligned (even though they were aligned on the training distribution).
This list of email scripts from 80,000 Hours also seems useful here https://80000hours.org/articles/email-scripts/
This is what I send people when they tell me they haven’t read Slate Star Codex and don’t know where to start.
Here are a bunch of lists of top ssc posts:
These lists are vaguely ranked in the order of how confident I am that they are good
https://www.slatestarcodexabridged.com/ (if interested in psychology almost all the stuff here is good https://www.slatestarcodexabridged.com/Psychiatry-Psychology-And-Neurology)
I think that there are probably podcast episodes of all of these posts listed below. The headings are not ranked by which heading I think is important, but within each heading they are roughly ranked. These are missing any new great ACX posts. I have also left off all posts about drugs, but these are also extremely interesting if you like just reading about drugs, and were what I first read of SSC.
If you are struggling to pick where to start I recommend either Epistemic Learned Helplessness or Nobody is Perfect, Everything is Commensurable.
The ‘Thinking good’ posts I think have definitely improved how I reason, form beliefs, and think about things in general. The ‘World is broke’ posts have had a large effect on how I see the world working, and what worlds we should be aiming for. The ‘Fiction’ posts are just really really good fiction short stories. The ‘Feeling ok about yourself’ have been extremely helpful for developing some self-compassion, and also for being compassionate and non-judgemental about others; I think these posts specifically have made me a better person.
Posts I love:
World is broke and how to deal with it
https://slatestarcodex.com/2015/08/17/the-goddess-of-everything-else-2/ (read after moloch stuff I think)
Feeling ok about yourself
https://slatestarcodex.com/2018/04/01/the-hour-i-first-believed/ (this is actual crazy, read last)
Hmm, this might come down to how independent the parameters are. I think in large networks there will generally be enough independence between the parameters for local minima to be rare (although possible).
As a toy example of moving from one algorithm to another, if the network is large enough we can just have the output being a linear combination of the two algorithms and up-regulate one, and down regulate the other:
And a is changed from 1 to 0.
The network needs to be large enough so the algorithms don’t share parameters, so changing one doesn’t affect the performance of the other. I do think this is just a toy example, and it seems pretty unlikely to spontaneously have two multiplication algorithms develop simultaneously.
I think there is likely to be a path from a model with shallow circuits to a model with deeper circuits which doesn’t need any ‘activation energy’ (it’s not stuck in a local minimum). For a model with many parameters, there are unlikely to be many places where all the deriviatives of the loss wrt all the parameters are zero. There will almost always be at least one direction to move in which decreases the shallow circuit while increasing the general one, and hence doesn’t really hurt the loss.
I’m from Dunedin and went to highschool there (in the 2010s), so I guess I can speak to this a bit.
Co-ed schools were generally lower decile (=lower socio-economic backgrounds) than the single sex schools (here is data taken from wikipedia on this). The selection based on ‘ease of walking to school’ is still a (small) factor, but I expect this would have been a larger factor in the 70s when there was worse public transport. In some parts of NZ school zoning is a huge deal, with people buying houses specifically to get into a good zone (especially in Auckland) but this isn’t that much the case in Dunedin.
Based on (~2010s) stereotypes about the schools, rates of drug use seemed pretty similar between co-ed and all-boys schools, and less in all-girls. And rates of violence were higher in all-boys schools, and less in co-ed and all-girls. But this is just my impression, and likely decades too late.
I think that this is a possible route to take, I don’t think we currently have a good enough understanding of how to control\align mesa-optimizers to be able to do this.
I worry a bit that even if we correctly align a mesa-optimizer (make its mesa-objective aligned with the base-objective), its actual optimization process might be faulty/misaligned and so it would mistakenly spawn a misaligned optimizer. I think this faulty optimization process is most likely to happen sometime in the middle of training, where the mesa-optimizer is able to make another optimizer but not smart enough to control it.
But if we develop a better theory of optimization and agency, I could see it being safe to allow mesa-optmizers to create new optimizers. I currently think we are nowhere near that stage, especially considering that we don’t seem to have great definitons of either optimization or agency.
So do you think that the only way to get to AGI is via a learned optimizer? I think that the definitions of AGI (and probably optimizer) here are maybe a bit fuzzy.
I think it’s pretty likely that it is possible to develop AI systems which are more competent than humans in a variety of important domains, which don’t perform some kind of optimization process as part of their computation.
Thanks for your reply! I agree that I might be a little overly dismissive of the loss landscape frame. I agree mostly with your point about convergent gradient hackers existing minima of the base-objective, I briefly commented on this (although not in the section on coupling)
Here we can think of the strong coupling regime as being a local minimum in the loss landscape, such that any step to remove the gradient hacker leads locally to worse performance. In the weak coupling regime, the loss landscape will still increase in the directions which directly hurt performance on the mesa-objective, but gradient descent is still able to move downhill in a direction that erases the gradient hacker.
(I also agree with your rough sketch and had some similar notes which didn’t make it into the post)
I think I am still a little unsure how to mix the two frames of ‘the gradient hacker has control’ and ‘it’s all deterministic gradient descent+quirks’, which I guess is just the free will debate. It also seems like you could have a gradient hacker arise on a slope (which isn’t non-convergent) and then uses its gradient hacking ways to manipulate how it is further modified, but this might just be a further case of a convergent gradient hacker.
The loss landscape frame doesn’t seem to really tell us how a gradient hacker would act or modify itself. From the ‘gradient hacker’ perspective there doesn’t seem to be much difference between manipulating the gradients via the outputs/cognition and manipulating the gradient via quirks/stop-gradients. It still seems like the non-convergent gradient hacker would also require coupling to get started in the first place; the model needs to be pretty smart to be able to know how to use these quirks/bugs and I think this needs to come from some underlying general ability.
One counter to this could be that there is a non-convergent gradient hacker which can ‘look at’ the information that the underlying capabilities have (eg the non-convergent gradient hacker can look for information on how to gradient hack). But it still seems here that there will be some parts of the gradient hacker which will have some optimization pressure on them from gradient descent, and so without coupling they would get eroded.