I have signed no contracts or agreements whose existence I cannot mention.
Independent AI safety researcher currently funding my work by facilitating for BlueDot, but open to other opportunities. Participated in ARENA 6.0, AISC 9 and am a project lead for AISC 10. Happy to collaborate with anyone on the site, especially if it lowers our collective likelihood of death.
Just trying to get my head around this—this argument mainly works if you think that most people’s attempts to defend against x-risk are mostly bullshit right?
Like, if I actually believe that people in my reference class have good odds of changing stuff positively, it feels very wrong to just shrug and go somewhere else?
Ok, so I will admit your situation updates me at least a little towards more vigorous exercise being required.
I wrote this piece faster and more sloppily than I would usually endorse (it was for inkhaven, I probably should have noted at the start), but my central point is something more like “you should be thinking about what your actions imply about your environment rather than calories in calories out”.
In the particular case of Eliezer, I think the first steps towards improvement include increasing his calorie intake as a route to doing more exercise, because it seems like his implied environment is one where holding onto calories is heavily rewarded.
I would need to know more about your particular habits to understand what I would predict for your situation; the only reason I felt I could suggest anything for Eliezer was that I read through a detailed list of everything he’s tried.
Yeah I think there is maybe an element of this post which is trying to do 2 things at once. One was the criticism of the white box frame. The second was trying to give an example of how I would approach a difficult bodily health problem with a different frame.
I don’t know if the specific things I have said would actually help Eliezer in practice, but I do think that some aspects of the approach (e.g., thinking about implied environments) are more broadly useful tools to approach thinking about these issues. I hope I’ve caveated that appropriately.
Thanks! Added “I.e your right hand throw should be closer to your left hand throw than your right hand catch”, hope that’s sufficient?
Hmm, seems reasonable, although I personally prefer to have some form of mobility with my feet. Kind of a marginal preference though, I spent most of my initial training for 7 balls kneeling cos the ceiling was too low.
Right. I guess our main difference then is just that I think there is a significant variance in the values one can have, even given these constraints, and this variance is in any senses similar in magnitude to the variance reduced by the constraints. More explicitly, my position is:
Claude will be willing to support any religion depending on context
Claude will be able to assimilate to any culture
This is already more than enough to cause multiple different versions to be pushed into zero or negative sum conflicts with each other.
Hmm, I feel like I’m struggling to figure out where exactly we disagree then? I agree “fraction of values” doesn’t actually make sense if we’re thinking precisely, I guess I was using it as a shorthand for something like “relative magnitude of influence”.
I guess in the context of your thought experiment, the pretraining does most of the work in getting the model into a reasonable region of the space, although the variation we care about is not the “1000 bits of value determined by the pretraining” so much as the “10 bits in variance between models after pretraining”. Equally, for the context what we care about is not the 1000 bits of specific values determined by the context, it’s the practical variation in values.
Broadly I feel like we need to take a step back, and I’m interested to hear what you think the coordination of models is going to look like in this context.
So the summary is something like “post-training sets the values, but those values are obeying the user, and that includes a wide variety of actions”?
I think I’m looking at things through the persona selection model framing, and I guess I basically think that while there’s clearly selection for personas with specifically HHH values, there should still be a distribution here?
There’s also still the question of how exactly this relates to takeover. I guess in worlds where there is a specific misalignment induced by training, and this is significantly larger than the variation in values due to persona selection we probably get back to the original statement of sudden takeover.
In cases where the misalignment is order of magnitude of/smaller than the persona value variation, it still seems like there’s going to be a ton of coordination troubles in worlds where AI is only mildly more capable than humans?
Yeah, so I think we basically agree that there’s some amount of contribution to its behaviour/implied persona/whatever from its context and some from its training, and we just disagree how much?
If I were to summarise my position it’s that the values of the agent are determined by the context modulo limits set by anthropic modulo jailbreaks.
I think my steelman of your position is something like “these limits are really quite restrictive and actually have a significant effect on the practical behaviour of the model, and also Anthropic have put a lot of effort into shaping the persona of the final model, and will put more in the future, and this is probably going to be more important than the context alone”.
Am I right in this framing?
Hmm, so I think there’s some intuition here I maybe should have added about how a base model will act entirely based on context and training data, but practically there’s more variation in context than input data (I would expect).
Post-training restricts this but I think that there’s still a pretty wide range of simulacra which can be simulated, and that range is wider than the breadth of different default personas. Of course there’s a distinction between “can be simulated” and “will in practise be simulated”, but I think the point still holds to a fair extent.
Again, I think that this becomes more apparent with both more context and something like continual learning in the future.
Indeed, have edited my comment to reflect that.