Sean Herrington

Agents, Si­mu­la­tors and Interpretability

• Sean HerringtonMay 31, 2025, 8:09 AM

  1 point

  0

  in reply to: williawa’s comment on: Case Stud­ies in Si­mu­la­tors and Agents

  I did do a very quick loose experiment to check that the idea that later weights are the ones that tend to change at the end of training, while early weights tend to change early on. This was more of a sanity check than anything else though.

• Sean HerringtonMay 26, 2025, 1:41 PM

  0 points

  0

  in reply to: williawa’s comment on: Case Stud­ies in Si­mu­la­tors and Agents

  So the next post in the sequence addresses some elements of what happens when you do RL after supervised learning.

  The conclusion (although this is similarly tentative to yours) is that RL will change the final weights in the network more, essentially giving us a system with an agent using the output of a simulator.

  I don’t think this is completely incompatible with your suggestion—this process requires simulatory circuits becoming agenty circuits (in so far as these exist, it seems entirely unclear to me whether these concepts make sense at the circuit level rather than being an emergent phenomenon). I think it would be great if we could do some more research on how exactly this process would take place.

• Sean HerringtonMay 25, 2025, 7:16 AM

  0 points

  0

  in reply to: Isopropylpod’s comment on: Case Stud­ies in Si­mu­la­tors and Agents

  As mentioned in our last post, we see simulators and agents as having distinct threat models and acting differently, and in future posts we plan to share other ideas for how to distinguish them.

  As an intuition, a simulator will typically produce a variety of simulacra with independent goals, whereas an agent will typically have a single coherent goal. This difference should be measurable.

  That said, you are correct that it can often be challenging to tell the difference.

Case Stud­ies in Si­mu­la­tors and Agents

• Sean HerringtonMay 16, 2025, 5:06 AM

  5 points

  2

  in reply to: SurfingOrca’s comment on: Doomers Should try Much Harder to get Famous

  At least in democracies, convincing the people of something is an effective way to get politicians to pay attention to it—their job depends on getting these people to vote for them.

  Notably in the UK, David Cameron gave the people a vote on whether to leave the EU because this was an idea that was gaining popularity. He did this despite not himself believing in the idea.

  Naturally, plenty of legislation also gets passed without most people noticing, and in this respect we are better off convincing lawmakers. But I think that if we are able to convince a significant portion of the public, we will by extension convince a substantial number of lawmakers through their interaction with the public.

  I have not read through the whole of the blogpost that you linked, but I disagreed with the “two important facts” used as a premise (1. People’s opinions are mostly genetic and 2. Most people’s opinions are completely random unless they’re smart.), and did not therefore trust any conclusions that might come from them.

  Equally I get the impression that given the scale of the challenge, even if we were to cede that convincing the public is less important than convincing politicians, we will most likely need to do both to have a reasonable shot at passing anything that looks like good legislation.

Align­ing Agents, Tools, and Simulators

Agents, Tools, and Simulators