beren(Beren Millidge)

• beren 6 Feb 2023 21:36 UTC

  1 point

  0 ∶ 0

  in reply to: rvnnt’s comment on: Em­pa­thy as a nat­u­ral con­se­quence of learnt re­ward models

  I suspect that {the cognitive process that produced the above sentence} is completely devoid of security mindset. If so, might be worth trying to develop security mindset? And/​or recognize that one is liable to (i.a.) be wildly over-optimistic about various alignment approaches. (I notice that that sounded unkind; sorry, not meaning to be unkind.)
  

  Yep this is definitely not proposed as some kind of secure solution to alignment (if only the world were so nice!). The primary point is that if this mechanism exists it might provide some kind of base signal which we can then further optimize to get the agent to assign some kind of utility to others. The majority of the work will of course be getting that to actually work in a robust way.

  You pointed out that empathy is not a silver bullet. I have a vague (but poignant) intuition that says that the problem is a lot worse than that: Not only is empathy not a silver bullet, it’s a really really imprecise heuristic/​proxy/​shard for {what we actually care about}, and is practically guaranteed to break down when subjected to strong optimization pressure.

  Yes. Realistically, I think almost any proxy like this will break down under strong enough optimization pressure, and the name of the game is just to figure out how to prevent this much optimization pressure being applied without imposing too high a capabilities tax.

• beren 6 Feb 2023 21:32 UTC

  1 point

  0 ∶ 0

  in reply to: Steven Byrnes’s comment on: Em­pa­thy as a nat­u­ral con­se­quence of learnt re­ward models

  I think this is a mechanism that actually happens a lot. People generally do lose a lot of empathy with experience and age. People definitely get de-sensitized to both strongly negative and strongly positive experiences after viewing them a lot. I actually think that this is more likely than the RL story—especially with positive-valence empathy which under the RL story people would be driven to seek out.

  But then every time that empathy thing happens, I obviously don’t then immediately eat chocolate. So the reward model would get an error signal—there was a reward prediction, but the reward didn’t happen. And thus the brain would eventually learn a more sophisticated “correct” reward model that didn’t fire empathetically. Right?

  My main model for why this doesn’t happen in some circumstances (but definitely not all) is that the brain uses these signals and has a mechanism for actually providing positive or negative reward when they fire depending on other learnt or innate algorithms. For instance, you could pass the RPE through to some other region to detect whether the empathy triggered for a friend or enemy and then return either positive or negative reward, so implementing either shared happiness or schadenfreude. Generally I think of this mechanism as a low level substrate on which you can build up a more complex repertoire of social emotions by doing reward shaping on these signals.

  Also—I really like your post on empathy that cfoster linked above! I have read a lot of your work but somehow missed that one lol. Cool we are thinking at least somewhat along similar lines

• beren 6 Feb 2023 21:13 UTC

  1 point

  0 ∶ 0

  in reply to: cfoster0’s comment on: Em­pa­thy as a nat­u­ral con­se­quence of learnt re­ward models

  It seems like the AIs we build will be very different from us, at least in terms of basic drives. I can definitely empathize when there’s some common currency to the experience (for ex. they’re feeling pain, and I’ve also experienced pain), but probably less so when there’s a greater gap. Since AIs won’t share any of our physiology or evolutionary history, I worry that that common currency will be missing, which would seemingly incentivize the AI having separate circuits for modeling humans and for modeling itself

  Yes, this depends a lot on the self model of the AGI. It’s definitely not a silver bullet. The AGI will almost certainly have a very good model of humans, their culture, and how their minds work from various self-supervised losses. Whether the AGI conceptualises itself as close to this or not depends on the representations of AGI in the dataset as well as potentially our training regime.

  Nitpick about terminology: I think the stuff you’re talking about is primarily attributable to having a learned value function rather than to having a learned reward model in the narrow sense of a predictor of immediate reward. I tend to use value function to refer to the thing that, alongside the reward function, produces visceral (gut-like) reactions to thoughts based on forecasts that were learned via something like TD learning

  I agree it is not necessarily the reward model that generates direct feelings. I think it is hard to connect any part of an RL system directly to gut level ‘feels’ because we don’t really know what these are. The value function is just the estimate of the long run reward and is trained on a supervised bellman equation. It is very possible that the machinery that creates this won’t exist at all in the AGI, or maybe it is just some intrinsic property of RL agents I don’t know.

• beren 6 Feb 2023 20:28 UTC

  3 points

  0 ∶ 0

  in reply to: Charlie Steiner’s comment on: Em­pa­thy as a nat­u­ral con­se­quence of learnt re­ward models

  My prediction is that there really is an evolved nudge towards empathy in the human motivational system, and that human psychology—like usually being empathetic but sometimes modulating it and often justifying self-serving actions—is sculpted by such evolved nudges, and wouldn’t be recapitulates in AI lacking those nudges.

  I agree—this is partly what I am trying to say in the contextual modulation section. The important thing is that the base capability for empathy might exist as a substrate to then get sculpted by gradient descent /​ evolution to implement a wide range of adaptive pro or anti-social emotions/​behaviours. Which of these behaviours, if any, get used by the AI will depend on the reward function /​ training data it sees.

Em­pa­thy as a nat­u­ral con­se­quence of learnt re­ward models

AGI will have learnt util­ity functions

• beren 25 Jan 2023 13:29 UTC

  2 points

  0 ∶ 0

  in reply to: leogao’s comment on: Gra­di­ent hack­ing is ex­tremely difficult

  Goal misgeneralization (the global min might also be malign OOD). The thing you’re talking about here I would basically describe as the first thing.

  Is there a way this is different from standard goodhearting concerns? I totally agree that this is a problem but it seems importantly conceptually distinct to me from gradient hackers/​mesaoptimization

• beren 25 Jan 2023 13:26 UTC

  7 points

  0 ∶ 0

  on: “Endgame safety” for AGI

  Strongly upvoted this post. I agree very strongly with every point here. The biggest consideration for me is that alignment seems like the kind of problem which is primarily bottlenecked on serial conceptual insights rather than parallel compute. If we already had alignment methods that we know would work if we just scaled them up, the same way we have with capabilities, then racing to endgame might make sense given the opportunity costs of delaying aligned AGI. Given that a.) we don’t have such techniques and b.) even if we did it would be hard to be so certain that they are actually correct, racing to endgame appears very unwise.

  There is a minor tension with capabilities in that I think that for alignment to progress it does need some level of empirical capabilities results both in revealing information about likely AGI design and threat models and also so we can actually test alignment techniques. I think that e.g. if ML capabilities had frozen at the level of 2007 for 50 years, then at some point we would stop being able to make alignment progress without capabilities advancements but I think that in the current situation we are very very far from this Pareto frontier.

• beren 25 Jan 2023 13:15 UTC

  2 points

  0 ∶ 0

  in reply to: leogao’s comment on: Gra­di­ent hack­ing is ex­tremely difficult

  The local minima point is interesting. My initial question is how this squares with both theoretical and empirical findings that networks generally don’t seem to get stuck in local minima and the lots of hints that the general loss landscape in which they operate is fairly benign?

• beren 25 Jan 2023 12:18 UTC

  2 points

  0 ∶ 0

  in reply to: Mauricio’s comment on: Gra­di­ent hack­ing is ex­tremely difficult

  I think this is only possible if the coupling between the gradient hacker’s implementation of its malign behaviour and the good performance is extremely strong and essentially the correlation has to be 1. It is not like gradient descent has only one knob to turn for ‘more gradient hacker’ or ‘less gradient hacker’. Instead, it has access to all of the internal weights of the gradient hacker and will change them to both a.) strengthen the positive aspects of the gradient hacker wrt the outer loss and b.) weaken the negative aspects. I.e. so if the gradient hacker is good at planning which is useful for the model but is malign in some other way, then gradient descent will strengthen the planning related parameters and weaken the malign ones simultaneously. The only way this fails is if there is literally no way to decouple these two aspects of the model, which I think would be very hard to maintain in practice. This is basically property 1 that gradient descent optimises all parameters in the network and leaves no slack.

• beren 24 Jan 2023 20:01 UTC

  LW: 7 AF: 5

  0 ∶ 0

  AF

  on: Some of my dis­agree­ments with List of Lethalities

  I broadly agree with a lot of shard theory claims. However, the important thing to realise is that ‘human values’ do not really come from inner misalignment wrt our innate reward circuitry but rather are the result of a very long process of social construction influenced both by our innate drives but also by the game-theoretic social considerations needed to create and maintain large social groups, and that these value constructs have been distilled into webs of linguistic associations learnt through unsupervised text-prediction-like objectives which is how we practically interact with our values. Most human value learning occurs through this linguistic learning grounded by our innate drives but extended to much higher abstractions by language.i.e. for humans we learn our values as some combination of bottom-up (how well do our internal reward evaluators in basal ganglia/​hypothalamus) accord with the top-down socially constructed values) as well as top-down association of abstract value concepts with other more grounded linguistic concepts.

  With AGI, the key will be to work primarily top-down since our linguistic constructs of values tend to reflect much better our ideal values than our actually realised behaviours. Using the AGI’s ‘linguistic cortex’ which already has encoded verbal knowledge about human morality and values to evaluate potential courses of action and as a reward signal which can then get crystallised into learnt policies. The key difficulty is understanding how, in humans, the base reward functions interact with behaviour to make us ‘truly want’ specific outcomes (if humans even do) as opposed to reward or their correlated social assessments. It is possible, even likely, that this is just the default outcome of model-free RL experienced from the inside and in this case our AGIs would look highly anthropomorphic.

  Also in general I disagree about aligning agents to evaluations of plans being unnecessary. What you are describing here is just direct optimization. But direct optimization -- i.e .effectively planning over a world model—is necessary in situations where a.) you can’t behaviourally clone existing behaviour and b.) you can’t self-play too much with a model-free RL algorithms and so must rely on the world-model. In such a scenario you do not have ground truth reward signals and the only way to amake progresss is to optimise against some implicit learnt reward function.

  I also am not sure that an agent that explicitly optimises this is hard to align and the major threat is goodhearting. We can perfectly align Go-playing AIs with this scheme because we have a ground truth exact reward function. Goodhearting is essentially isomorphic to a case of overfitting and can in theory be solved with various kinds of regularisation, especially if the AI maintains a well-calibrated sense of reward function uncertainty then in theory we can derive quantification bounds on its divergence from the true reward function.

• beren 24 Jan 2023 19:53 UTC

  3 points

  0 ∶ 0

  on: My take on Ja­cob Can­nell’s take on AGI safety

  The convergence theorems basically say that optimizing for P[t] converges to optimizing for T[t+d] for some sufficient timespan d.

  The idea of a convergence theorem showing that optimizing any objective leads to empowerment has been brought up a bunch of times in these discussions, as in this quote. Is there some well-known proof/​paper where this is shown? AFAICT the original empowerment do not show any proof like this (may have missed it). Is this based off of Alex Turner’s work (https://​​arxiv.org/​​pdf/​​1912.01683.pdf) which results in a different measure than information theoretic empowerment (but intuitively related), or something else?

  Excellent post btw.

• beren 24 Jan 2023 15:52 UTC

  8 points

  0 ∶ 0

  in reply to: Chris_Leong’s comment on: Some of my dis­agree­ments with List of Lethalities

  Yes they have. There’s quite a large literature on animal emotion and cognition and my general synthesis is that animals (at least mammals) have at least the same basic emotions as humans and often quite subtle ones such as empathy and a sense of fairness. It seems pretty likely to me whatever the set of base reward functions encoded in the mammalian basal ganglia and hypothalamus is, it can quite robustly generate expressed behavioural ‘values’ that fall within some broadly humanly recognisable set.

Gra­di­ent hack­ing is ex­tremely difficult

• beren 13 Jan 2023 22:05 UTC

  3 points

  0 ∶ 0

  in reply to: DragonGod’s comment on: Scal­ing laws vs in­di­vi­d­ual differences

  This is definitely the case. My prior is relatively strong that intelligence is compact, at least for complex and general tasks and behaviours. Evidence for this comes from ML—the fact that the modern ML paradigm of huge network + lots of data + general optimiser being able to solve a large number of tasks is a fair bit of evidence for this. Other evidence is existence of g and cortical uniformity in general, as well as our flexibility at learning skills like chess, mathematics etc which we clearly do not have any evolutionarily innate specialisation for.

  Of course some skills such as motor reflexes and a lot of behaviours are hardwired but generally we see that as intelligence and. generality grows these decrease in proportion.

• beren 13 Jan 2023 22:00 UTC

  1 point

  0 ∶ 0

  in reply to: Jonathan_Graehl’s comment on: Scal­ing laws vs in­di­vi­d­ual differences

  I’m basing my thinking here primarily off of Herculano Houzel’s work. If you have reasons you think this is wrong or counterarguments, I would be very interested in them as this is a moderately important part of my general model of AI.

• beren 13 Jan 2023 16:02 UTC

  1 point

  0 ∶ 0

  in reply to: gwern’s comment on: Scal­ing laws vs in­di­vi­d­ual differences

  the brain imaging studies also show predicting intelligence taps into a lot more aspects of static neuroanatomy or dynamic patterns than simply brain volume

  Do you have links for these studies? Would leave to have a read about the static and dynamic correlates of g are from brain imaging!

• beren 13 Jan 2023 15:52 UTC

  2 points

  0 ∶ 0

  in reply to: Kyle Walsh’s comment on: Scal­ing laws vs in­di­vi­d­ual differences

  This is a good idea! I hadn’t thought that much about specific synaptic efficiency metrics. If we think about this in a bit more detail, these would effectively corresponds to some kind of changes in hyper parameters for an ML model. I.e. more rapid synaptic changes = potential for higher learning rate effectively. The more rapid synaptic formation (and potentially pruning?) is harder to model in ML but I guess would be an increase in effective parameter count.

• beren 10 Jan 2023 14:35 UTC

  1 point

  0 ∶ 0

  in reply to: Vladimir_Nesov’s comment on: Scal­ing laws vs in­di­vi­d­ual differences

  To some extent yes speed can compensate for intelligence but this isn’t really related to the question of FOOM.

  In theory, if we have an AGI which is human level but 1000x faster, it might be able to perform at the level of 1000 humans rather than a human from the year 3000. If we have a giant population of AGIs such that we can replicate the entire edifice of human science but running at 1000x faster, then sure. In practice though by Amdahl’s law such a speed increase would just move the bottleneck to something else (probably running experiments/​gathering data in the real world) so the speedup would be much less.

  The general point I agree with though that we don’t need foom or RSI for x-risk.

• beren 10 Jan 2023 14:24 UTC

  7 points

  0 ∶ 0

  in reply to: Steven Byrnes’s comment on: Scal­ing laws vs in­di­vi­d­ual differences

  I largely disagree about the intrinsic motivation/​reward function points. There is a lot of evidence that there is at least some amount of general intelligence which is independent of interest in particular fields/​topics. Of course, if you have a high level of intelligence + interest then your dataset will be heavily oriented towards that topic and you will gain a lot of skill in it, but the underlying aptitude/​intelligence can be factored out of this.

  How exactly specific interests are encoded is a different and also super fascinating question! It definitely isn’t a pure ‘bit prediction’ intrinsic curiosity since different people seem to care a lot about different kinds of bits. It is at least somewhat affected by external culture /​ datasets but not entirely (people can often be interested in things against cultural pressure or often before they really know what their interest is). It doesn’t seem super influenced by external reward in a lot of cases. To some extent it ties in with intrinsic aptitude (people tend to be interested in things they are good at) but of course this is at least somewhat circular since people tend to get better at things they are interested in, ceteris paribus.

  The hyperparameters is a good point. I was thinking about this largely as architectural changes but I think that I was wrong about this they are much more continuous and also potentially much more flexible genetically. This seems to be a better and more likely explanation for continuous IQ distributions than architecture directly. It would definitely be interesting to know how robust the brain is to these kinds of hyper parameter distributions (i.e over what range do people vary and is it systematic). In ML my understanding is that at large scale models are generally pretty robust to small hyper parameter variations (allowing people to get away with cargo culting hyperparams from other related papers instead of always sweeping themselves) although of course really bad hyperparams destroy performance. The brain may also be less stable due to some combination of recurrent dynamics/​active data selection leading to positive or negative loops, as well as just more weird architectural hyper parameters leading to more interactions and ways for things to go wrong.