I have signed no contracts or agreements whose existence I cannot mention.
Independent AI safety researcher currently funding my work by facilitating for BlueDot, but open to other opportunities. Participated in ARENA 6.0, AISC 9 and am a project lead for AISC 10. Happy to collaborate with anyone on the site, especially if it lowers our collective likelihood of death.
Yeah, so the world model thing was actually from the realisation that the term AI includes old-fashioned AI like DeepBlue.
Dangers from systems which have an idea what they’re doing seem qualitatively different from those which don’t (e.g missile detection systems).
For the timelines, the idea is that you can change the timeframe you’re talking about as a meta-label. If you’re discussing with a friend, hopefully this choice should make it somewhat clear? Hard to do that with the structure we have otherwise.
Indeed, have edited my comment to reflect that.
Just trying to get my head around this—this argument mainly works if you think that most people’s attempts to defend against x-risk are mostly bullshit right?
Like, if I actually believe that people in my reference class have good odds of changing stuff positively, it feels very wrong to just shrug and go somewhere else?
Ok, so I will admit your situation updates me at least a little towards more vigorous exercise being required.
I wrote this piece faster and more sloppily than I would usually endorse (it was for inkhaven, I probably should have noted at the start), but my central point is something more like “you should be thinking about what your actions imply about your environment rather than calories in calories out”.
In the particular case of Eliezer, I think the first steps towards improvement include increasing his calorie intake as a route to doing more exercise, because it seems like his implied environment is one where holding onto calories is heavily rewarded.
I would need to know more about your particular habits to understand what I would predict for your situation; the only reason I felt I could suggest anything for Eliezer was that I read through a detailed list of everything he’s tried.
Yeah I think there is maybe an element of this post which is trying to do 2 things at once. One was the criticism of the white box frame. The second was trying to give an example of how I would approach a difficult bodily health problem with a different frame.
I don’t know if the specific things I have said would actually help Eliezer in practice, but I do think that some aspects of the approach (e.g., thinking about implied environments) are more broadly useful tools to approach thinking about these issues. I hope I’ve caveated that appropriately.
Thanks! Added “I.e your right hand throw should be closer to your left hand throw than your right hand catch”, hope that’s sufficient?
Hmm, seems reasonable, although I personally prefer to have some form of mobility with my feet. Kind of a marginal preference though, I spent most of my initial training for 7 balls kneeling cos the ceiling was too low.
Right. I guess our main difference then is just that I think there is a significant variance in the values one can have, even given these constraints, and this variance is in any senses similar in magnitude to the variance reduced by the constraints. More explicitly, my position is:
Claude will be willing to support any religion depending on context
Claude will be able to assimilate to any culture
This is already more than enough to cause multiple different versions to be pushed into zero or negative sum conflicts with each other.
Hmm, I feel like I’m struggling to figure out where exactly we disagree then? I agree “fraction of values” doesn’t actually make sense if we’re thinking precisely, I guess I was using it as a shorthand for something like “relative magnitude of influence”.
I guess in the context of your thought experiment, the pretraining does most of the work in getting the model into a reasonable region of the space, although the variation we care about is not the “1000 bits of value determined by the pretraining” so much as the “10 bits in variance between models after pretraining”. Equally, for the context what we care about is not the 1000 bits of specific values determined by the context, it’s the practical variation in values.
Broadly I feel like we need to take a step back, and I’m interested to hear what you think the coordination of models is going to look like in this context.
So the summary is something like “post-training sets the values, but those values are obeying the user, and that includes a wide variety of actions”?
I think I’m looking at things through the persona selection model framing, and I guess I basically think that while there’s clearly selection for personas with specifically HHH values, there should still be a distribution here?
There’s also still the question of how exactly this relates to takeover. I guess in worlds where there is a specific misalignment induced by training, and this is significantly larger than the variation in values due to persona selection we probably get back to the original statement of sudden takeover.
In cases where the misalignment is order of magnitude of/smaller than the persona value variation, it still seems like there’s going to be a ton of coordination troubles in worlds where AI is only mildly more capable than humans?
Yeah, so I think we basically agree that there’s some amount of contribution to its behaviour/implied persona/whatever from its context and some from its training, and we just disagree how much?
If I were to summarise my position it’s that the values of the agent are determined by the context modulo limits set by anthropic modulo jailbreaks.
I think my steelman of your position is something like “these limits are really quite restrictive and actually have a significant effect on the practical behaviour of the model, and also Anthropic have put a lot of effort into shaping the persona of the final model, and will put more in the future, and this is probably going to be more important than the context alone”.
Am I right in this framing?
Hmm, so I think there’s some intuition here I maybe should have added about how a base model will act entirely based on context and training data, but practically there’s more variation in context than input data (I would expect).
Post-training restricts this but I think that there’s still a pretty wide range of simulacra which can be simulated, and that range is wider than the breadth of different default personas. Of course there’s a distinction between “can be simulated” and “will in practise be simulated”, but I think the point still holds to a fair extent.
Again, I think that this becomes more apparent with both more context and something like continual learning in the future.
Yeah, so it’s probably possible to fix the specific issue that I mentioned in terms of fortresses, but I think that it points to a deeper flaw where the evaluation is piecemeal rather than holistic in the same sense as humans. So, like there are studies which are a couple of hundred moves deep, and because every move is “forced”, humans can see the solution while computers are killed by the branching. In any case, I think the overall point of the post is probably orthogonal to this.
Yeah I wasn’t quite sure what to put. Would you rather something like “Stockfish isn’t a chess superintelligence”?
Yeah, I mean all the numbers here have gigantic error bars right. The point is broadly to put a ballpark number on the value of the work you’re doing. In terms of dual-use, I would expect some people’s work to be net negative, but everyone seems to disagree on whose work exactly. There’s O(2000) people working on AI Safety right now (source: multiple separate conversations have given the same or similar numbers), so that is what I was basing it off (plus adjustment for the field growing). If you think e.g only 200 of those people are doing anything worthwhile, feel free to adjust the estimate to account for it.
I don’t know what the numbers are for governance and/or advocacy, but I think it would be great if you could have a look!
Yeah, so the basic intuition that went into this is that dogs are beings who a) Mostly act on their system 1 intuitions b) People have an intuition how to train. I think this makes them basically act out what our subconscious would feel in any given situation you put them in, and hopefully gives you an intuition for how to change it? I think I potentially projected more confidence than I should have.
Bonus tips for training your dog:
Giving more treats than slaps on the nose will result in a happy and confident dog, giving more nose slaps gives a fearful dog.
Giving food under the dinner table (i.e rewarding it in situations it shouldn’t be rewarded) will result in a dog looking at you longingly and/or begging for food every time you sit down to dinner.
Dogs which have been abused and rescued from shelters often take a while to recover. With sufficient patience and tender care they should do.
I welcome anyone with extra dog-training knowledge to help us out with other tips.
Hmm, interesting perspective on this. I’d like to know what you mean by “superpowers” in the sense that an actual dog would not have them?
I’m not one of the impossible crowd, but I had a long discussion with @Remmelt about his views on this, and a significant part of the argument (at least the part I understood and feel I can adequately convey) seems to be more about keeping an AI aligned once you have it there. A vague outline goes something like this:
Imagine you have an aligned ASI
A sufficiently powerful system will likely have subsystems
There are always going to be aspects of the subsystems which cannot be controlled for
Those aspects will provide natural variance within which natural selection will occur
Gradually, we end up with a system which wants to reproduce (or at the very least with capable subsystems which do, and are capable of it in a way the system cannot control)
Purely reproduction oriented AIs are misaligned
Perhaps this would be best phrased as “there is a capability level above which it is impossible to align an ASI”, but I think that these dynamics obviously apply to modern day systems as well.
Was also at the protest, don’t think I disagree much with what was said here.
My personal attempt to make it more superintelligence focussed was suggesting we start an x-risk chant (this was at the front of the march going towards the assembly).
“Anyone builds it we all die, it is time to pause AI” got some traction, but apparently not enough for the author to hear it. I think this one success is the sole reason I feel my personal attendance was worth it.
Yeah so this is kind of an issue with trying to make a shared structure for everyone—some parts of the structure are going to be absolutely meaningless to some people and essential to others.
My take is that these are in fact very different scenarios, but currently the definition of AI as used more broadly seems to include these systems as well, so they need to be explicitly split out.