Oliver Sourbut

Karma: 343

Call me Oliver or Oly—I don’t mind which.

I’m particularly interested in sustainable collaboration and the long-term future of value. I’d love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I’m currently (2022) just embarking on a PhD in AI in Oxford, and also spend time in (or in easy reach of) London. Until recently I was working as a senior data scientist and software engineer, and I’ve been doing occasional AI alignment research with SERI.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read—let me know your suggestions! In no particular order, here are some I’ve enjoyed recently

Ord—The Precipice
Pearl—The Book of Why
Bostrom—Superintelligence
McCall Smith—The No. 1 Ladies’ Detective Agency (and series)
Melville—Moby-Dick
Abelson & Sussman—Structure and Interpretation of Computer Programs
Stross—Accelerando
Graeme—The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

Hanabi (can’t recommend enough; try it out!)
Pandemic (ironic at time of writing...)
Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who’ve got to know me only recently are sometimes surprised to learn that I’m a pretty handy trumpeter and hornist.

Oliver Sourbut 28 Nov 2022 11:36 UTC
1 point
0 ∶ 0
in reply to: Akash’s comment on: What I Learned Running Refine
FWIW my experience of MATS 0.1 (i.e. the first run/pilot 2021-22) was that it was more open-ended and diversity-focused than subsequent MATS, which has been more apprenticeship-focused. That was helpful for me at the time, but I don’t know if it was ever the intention per se, and I agree that the focus of MATS now is different. I haven’t thought long enough to decide if this is good or bad.

Oliver Sourbut 28 Nov 2022 10:49 UTC
3 points
0 ∶ 0
in reply to: Razied’s comment on: Why square errors?
This approach also makes lots of regularisation techniques transparent. Typically regularisation corresponds to applying some prior (over the weights/parameters of the model you’re fitting). e.g. L2 norm aka ridge aka weight decay regularisation corresponds exactly to taking a Gaussian prior on the weights and finding the Maximum A Priori estimate (rather than the Maximum Likelihood).

Oliver Sourbut 22 Oct 2022 20:43 UTC
1 point
0 ∶ 0
in reply to: interstice’s comment on: Counterarguments to the basic AI x-risk case
(and ‘self-replicating’ for some reasonable operationalisation)

Oliver Sourbut 3 Oct 2022 9:55 UTC
1 point
in reply to: Oliver Sourbut’s comment on: Some real examples of gradient hacking
In short, I think ADS available as a mechanism to the extent that the responses of a system can affect subsequent inputs to the system (technically this is always, but in practice the degree of effect varies enormously). This need not be a system subject to further training updates, though if it is, depending how those updates are generated, ADS behaviour may or may not be reinforced.

Gradient hacking was originally coined to mean deliberate, situationally aware influence over training updates. (ADS is one mechanism by which this could be achieved.)

The term ‘gradient hacking’ seems to also be used commonly to refer to any kind of system influence over training updates, whether situationally aware/deliberate or no. I think it’s helpful to distinguish these so I often say ‘deliberate gradient hacking’ to make sure.

Oliver Sourbut 3 Oct 2022 9:40 UTC
6 points
in reply to: aogara’s comment on: Some real examples of gradient hacking
Yeah, I read the ADS paper(s) after writing this post. I think it’s a useful framing, more ‘selection theorem’ ey and with less emphasis on deliberateness/purposefulness.

Additionally, I think there is another conceptual distinction worth attending to
- auto-induced distributional shift is about affecting environment to change inputs
  - the system itself might remain unchanging and undergo no further learning, and still qualify
- gradient hacking is about changing environment/inputs/observations to change updates (gradients)
  - the system is presumed subject to updates, which it is taking (some amount of deliberate) influence over
In this post I wrote

I’m looking for deliberate behaviours which are intended to affect the fitness landscape of the outer process.

which I think rules out (hopefully!) contemporary recommender systems on the above two distinctions (as you gestured to regarding mesa-optimization).

In practice, for a system subject to online outer training, ADS changes the inputs which changes the training distribution, in fact causing some change in the updates to the system (perhaps even a large change!). But ADS per se doesn’t imply these effects are deliberate, though again you might be able to say something selection-theorem-ey about this process if iterated. Indeed, a competent and deliberate gradient hacker might use means of ADS quite effectively.

None of this is to say that ADS is not a concern, I just think it’s conceptually somewhat distinct!

Oliver Sourbut 30 Sep 2022 15:46 UTC
1 point
0 ∶ 0
in reply to: DanielFilan’s comment on: Reward is not the optimization target
smarter exploration strategies will depend on the agent’s value function

I think this is plausible but overconfident.

FWIW I think with moderate confidence that smarter exploration strategies are fundamental to advanced agency—I think of things like play, ‘deliberate exploration’, experiment design, goal-backchaining and so-on. Mainly because epsilon exploration is scuppered for sparse rewards and real-world dynamics are super-duper highly-branching.

I also think we’ve barely scratched the surface of understanding exploration, though there are some interesting directions like EMPA^[1], VariBAD^[2], HER^[3], and older stuff like pseudocount-based and prediction-error-based ‘curiosity’.

If humans (and/or supervised speedups of humans or similar) can provide dense signals, this claim is weaker, but I think the key problem for AGI learning is OOD dense signals, and I don’t think humans are capable of safe/accurate OOD dense reward/value signals.
1. ↩︎
  Tsividis et al—Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
2. ↩︎
  Zintgraf et al—VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
3. ↩︎
  Andrychowicz et al—Hindsight Experience Replay

Oliver Sourbut 30 Sep 2022 15:24 UTC
1 point
0 ∶ 0
in reply to: TurnTrout’s comment on: Reward is not the optimization target
This is a really helpful thread, for me, thank you both.

in humans… noticing your values drifted in a bad way is probably a negative reinforcement event

Are you hypothesising a shardy explanation for this (like, former, now dwindled shards get activated for some reason, think ‘what have I done?’; they emit a strong negative reinforcement—maybe they predict low value and some sort of long-horizon temporal-difference credit assignment kicks in...? And squashes/weakens/adjusts the new driften shards...? (The horizon is potentially very long?)) Or just that this is a thing in humans in particular somehow?

Oliver Sourbut 7 Sep 2022 18:27 UTC
3 points
0 ∶ 0
on: Against Relying on Evolution to Forecast AI Outcomes (Part 1)
This is a neat distillation of Steve’s piece, and also a helpful and persuasive extension. I appreciated the arguments 2, 3, and 4 in particular (‘2. We have more total evidence from human outcomes’, ‘3. Human learning trajectories represent a broader sampling of the space of possible learning processes’, ‘4. Evidence from humans are more accessible than evidence from evolution’).

I wanted to raise two counterpoints, without a strong opinion on how much weight they deserve.
1. You’re contrasting human learning → human values with evolution → human values. But without here strongly justifying, I think it makes sense to look more broadly at evolution → animal values (and potentially even broader than that). After all, the field of RL had its beginnings as a descriptive theory for animal and human behaviour more broadly.
  - a counter counter is that humans are the most obvious (maybe only) general-ish and reflective intelligence we know about
2. There might be an explanatory gap here for people caring quite so much about things like lineage, legacy, having children and descendants
  - maybe this is explained by within-lifetime reinforcement due to social norms, which are themselves subject to selection? (but this is just another evolution, right?) I don’t know if this is enough.
  - so I’m not sure it’s entirely true to say ‘5. Evolution could not have succeeded anyways’
    unless you think the original optimization daemon point is completely inevitable

Oliver Sourbut 5 Sep 2022 16:45 UTC
4 points
1 ∶ 0
in reply to: GeneSmith’s comment on: The shard theory of human values
That’s a really interesting reference to lamprey decision-making, thanks!

The descriptions and even the terminological choices are very similar to that in the hierarchical reinforcement learning literature, for example Sutton, Precup and Singh 1999. They use ‘option’ to refer to a kind of sub-policy which sequentially composes atomic/primitive actions, and can be activated or deactivated by higher-level controllers, and learned or improved through reinforcement—I assume Quintin and/or Alex invoke this or similar by use of the same term.

Oliver Sourbut 5 Sep 2022 16:29 UTC
6 points
0 ∶ 0
on: The shard theory of human values
A person might deliberately avoid passing through the sweets aisle in a supermarket in order to avoid temptation.

I like this example, and the related discussion around reflective endorsement and contextual activation/weighting of various ‘decision-making influences’ is great.

This relates closely in my reading to the concepts of Actualism and Possibilism from moral philosophy^[1].

In short, actualism emphasises what the ongoing agent would actually (be expected to) do as the decision-relevant factors—in the sweets example, perhaps that’s succumbing to a sweet tooth given the context of proximity to the sweets, and buying lots of packets.

Possibilism instead emphasises the ongoing counterfactual of what options are considered available as the decision-relevant factors—even if you can predict that you’ll succumb, the right thing to do is to take the shorter route via the sweets and then just have the willpower dammit to overcome the sweet tooth!

Some authors contrast naive possibilist with ‘resolute’ or ‘sophisticated’ decisions for this kind of sequential problem^[2].

In my mind these relate closely to the concepts of policy-relative action-advantage functions from reinforcement learning, except applied to more compound ‘options’ than whatever plain atomic action space is assumed to exist. But I’ve not seen this comparison made anywhere.

(Presumably you use the term ‘option’ as I do, to refer to something similar to the contextually-activated semi-policies of Sutton and Precup and others?)

By the way, I find it helpful to think about decision-makers only existing in their present form for the duration of a ‘single decision’ and I think this substantially gets at the heart of embeddedness.
1. ↩︎
  See the short Wikipedia take here or the longer Stanford take here.
2. ↩︎
  See Stanford here with their discussion of Ulysses under Sequential decisions

Oliver Sourbut 22 Aug 2022 11:27 UTC
3 points
0 ∶ 0
in reply to: Mark Xu’s comment on: How To Go From Interpretability To Alignment: Just Retarget The Search
I agree. My comment here on Rohin and John’s thread is a poor attempt at saying something similar, but also observing that having the machinery to do the ‘find the good heuristics’ thing is itself a (somewhat necessary?) property of ‘recursive-ish search’ (at least of the flavour applicable to high-dimensional ‘difficult’ problem-spaces). In humans and animals I think this thing is something like ‘motivated exploration’ aka ‘science’ aka ‘experimentation’, plus magic abstraction-formation and -recomposition.

I think it’s worth trying to understand better how these pieces fit together, and to what extent these burdens can (or will) be overcome by compute and training scale.

Oliver Sourbut 22 Aug 2022 11:12 UTC
6 points
1 ∶ 0
in reply to: Rohin Shah’s comment on: How To Go From Interpretability To Alignment: Just Retarget The Search
most of the action is in which plans were generated in the first place and “retarget the search” doesn’t necessarily solve your problem

I definitely buy this and I think the thread under this between you and John is a useful elaboration.

The thing that generates the proposals has to do most of the heavy lifting in any interestingly-large problem. e.g. I would argue most of the heavy lifting^[1] of AlphaGo and that crowd is done by the fact that the atomic actions are all already ‘interestingly consequential’ (i.e. the proposal generation doesn’t have to consider millisecond muscle twitches but rather whole ‘moves’, a short string of which is genuinely consequential in-context).

Nevertheless I reasonably strongly think that something of the ‘retargetable search’ flavour is a useful thing to expect, look for, and attempt to control.

For one, once you have proposals which are any kind of good at all, running a couple of OOMs of plan selection can buy you a few standard deviations of plan quality, provided you can evaluate plans ex ante better than randomly, which is just generically applicable and useful. But this isn’t the main thing, because with just that picture we’re still back to most of the action being the generator/heuristics.

The main things are that
1. (as John pointed out) recursive-ish generic planning is enormously useful and general, and implies at least some degree of retargetability.
2. (this is shaky and insidey) how do you arrive at the good heuristics/generators? It’s something like
  - ‘magic abstraction from relevantly-similar experience’
  - ‘magic recomposition of abstractions’
  - how do you get relevantly-similar experience?
    it’s ‘easy’ for ‘easy’ problems (e.g. low dimensional, defined action-space already ‘interestingly consequential’, someone already collected a dataset of examples, …)
    maybe one or more of these will apply to all the necessary pieces for PASTA-like AGI, but I doubt it
    
    what about for ‘hard’ problems (e.g. high dimensional, action-space not pre-fitted to the problem, few or no existing examples)?
    you need to ‘deliberately explore’ aka ‘do science’ aka ‘experiment’
    recursive-ish generic planning also looks to me like a good tool (‘the right/only tool’?) for pulling this off!
    cf $P_{2} B$ : Plan to $P_{2} B$ Better and my restatement of instrumental convergence
1. ↩︎
  (this is an entirely unfair defamation of Silver et al which I feel the need to qualify is at least partly rhetorical and not in fact my entire take on the matter)

Oliver Sourbut 18 Aug 2022 12:12 UTC
3 points
0 ∶ 0
on: What’s General-Purpose Search, And Why Might We Expect To See It In Trained ML Systems?
This is a fantastic point well articulated, reminiscent of some conversations we had a few months ago at Lightcone.
I’d say that a “general-purpose search” process is something which:
- Takes in a problem or goal specification (from a fairly broad range of possible problems/goals)
- … and returns a plan which solves the problem or scores well on the goal
I think we probably agree on what things there actually are, but I think this particular definition of ‘general purpose search’ is slightly too general to be a most useful pointer/carving.

This because it seems to include things like matrix inversion for least-squares solutions (unless ‘from a fairly broad range of possible problems/goals’ is taken to preclude this meaningfully?) which I deem importantly different. I’d class matrix inversion least-squares as a (powerful) heuristic^[1] (a ‘proposal’ in my deliberation terminology), but not as (proper) search itself.

I think it remains useful to distinguish algorithms which evaluate/promote or otherwise weigh proposals^[2]. This is what I’ve started calling ‘proper deliberation’ and it’s generally what I mean when I talk about search.

In the case of applying matrix inversion to ordinary least squares, for me, the ‘general deliberation’ consists of something like
1. noticing the relevant features of the problem (this is ‘abstraction/pattern-matching magic’)
2. cognitively retrieving the OLS abstraction and matrix-inversion as a cached heuristic (this is ‘propose’)
3. thinking ‘yes, this will work’ (this is ‘promote’)
4. applying matrix inversion to solve
A clever/practised enough deliberator does steps 1, 2 and 3 ‘right’ and doesn’t need to iterate for this particular problem (my point here is that if your heuristics are good enough you can deliberate with only one proposal and say ‘yep, good enough, let’s go’). But counterfactually step 2 might make various alternative proposals, or step 3 might think ‘actually there are too many dimensions in this case for inversion to be tractable’ or something, and thus there’s an evaluation and an internal update.
1. ↩︎
  Peter Barnett and Ian McKenzie coined ‘God-level heuristic’ for really solid mathematically-justified heuristics like this, which I quite like
2. ↩︎
  I don’t require this to be a ‘full consequentialist model-based valuation’, but that would be one example. See my deliberation simple examples for less sophisticated versions which are quite pervasive and nevertheless embody the ‘propose;promote’ breakdown

Oliver Sourbut 17 Aug 2022 12:43 UTC
6 points
0 ∶ 0
on: Team Shard Status Report
I love how your intro has the flavour of

We are Hydra. We are legion.

p.s. Hail Team Shard

p.p.s I’ve read a bunch of so-called Shard Theory stuff and I’m still not sure how it differs from the concepts of optimization daemons/mesa optimization besides less exclusively emphasising the ‘post-general’ regime (for want of a better term).

Oliver Sourbut 7 Aug 2022 10:11 UTC
1 point
0 ∶ 0
in reply to: Vaniver’s comment on: Reward is not the optimization target

Maybe the argument is “but if it’s never tried the action of optimizing harder for reward, then the RL algorithm won’t be able to reinforce that internal action”?

That’s my reading, yeah, and I agree it’s strained. But yes, the ‘internal action’ of even ‘thinking about how to’ optimise for reward may be not trivial to discover.

Separately, the action-weight downstream of that ‘thinking’ has to yield better actions than whatever the action results of the ‘rest of’ cognition are, to be reinforced (it stands to reason that they might, but plausibly heuristics amounting to ‘shaped’ value and reward proxies are easier to get right, hence inner misalignment).

I agree that once you find ways to directly seek reward you’re liable to get hooked to some extent.

I think this sort of thing is worth trying to get nuance on, but I certainly don’t personally derive much hope from it directly (I think this sort of reasoning may lead to useable insights though).

Oliver Sourbut 5 Aug 2022 17:33 UTC
3 points
0 ∶ 0
in reply to: Quintin Pope’s comment on: Human values & biases are inaccessible to the genome
This response is really helpful, thank you! I take various of the points as uncontroversial^[1], so I’ll respond mainly to those where I think you seem surprisingly confident (vs my own current epistemic position).

I and Alex both agree that the genome can influence learned behavior and concepts by exploiting its access to sensory ground truth… the imprinting circuitry is… imprecise

Of course!

It seems like there are two salient hypotheses that can come out of the imprinting phenomenon, though (they seem to sort of depend on what direction you draw the arrows between different bits of brain?):
1. Hard-coded proxies fire for the thing in question. (Maybe also this encourages more attention and makes it more likely for the runtime learner to develop corresponding abstractions.) Corresponding abstractions are highly correlated with the proxies, and this strong signal helps with symbol grounding. (And the now-grounded ‘symbols’ feed into whatever other circuitry.) Maybe decision-making is—at least partially—defined relative to these ‘symbols’.
2. Hard-coded proxies fire for the thing in question. (Maybe also this encourages more attention and makes it more likely for the runtime learner to develop corresponding abstractions.) These proxies directly wire to reward circuits. There is runtime reinforcement learning. The runtime reinforcement learner generates corresponding abstractions because these are useful ‘features’ for reinforced behaviour. Decision-making is the product of reinforced behaviour.
Both of these seem like useful things to happen from the POV of natural selection, so I don’t see how to rule out either (and tentatively expect both to be true). I think you and Alex are exploring the hypothesis 2?

FWIW I tentatively wonder if to the extent that human and animal decision-making fits something like an actor-critic or a propose-promote deliberation framing, the actor/propose might be more 2-ish and the critic/promote might be more 1-ish.

there’s some explanation that specifically predicts sunk cost / framing / goal conflation as the convergent consequences of the human learning process.

We could probably dig further into each of these, but for now I’ll say: I don’t think these have in common a material/mechanical cause much lower than ‘the brain’ and I don’t think they have in common a moving cause much lower than ‘evolution did it’. Framing, like anchoring, seems like a straightforward consequence of ‘sensible’ computational shortcuts to make world modelling tractable (on any computer, not just a human brain).

I think most high level goals / values are learned… don’t think most are directly installed by evolution

I basically can’t evaluate whether I agree with this because I don’t know what ‘high level’ and ‘most’ means. This isn’t intended as a rebuttal; this topic is in general hard to discuss with precision. I also find it disconcertingly hard to talk/think about high and low level goals in humans without bumping into ‘consciousness’ one way or another and I really wish that was less of a mystery. I basically agree that the vast majority of what seem to pass for goals at almost any level are basically instrumental and generated at runtime. But, is this supposed to be a surprise? I don’t think it is.

learning systems don’t develop a single ontology… values “learn” to generalize across different ontolgies well before you learn that people are made of cells

Seems uncontroversial to me. I think we’re on the same page when I said

ontological shifts are just supplementary world abstractions being installed which happen to overlap with preexisting abstractions

I don’t see any reason for supplementary abstractions to interfere with values, terminal or otherwise, resting on existing ontologies. (They can interfere enormously with new instrumental things, for epistemic reasons, of course.)

I note that sometimes people do have what looks passingly similar to ontological crises. I don’t know what to make of this except by noting that people’s ‘most salient active goals’ are often instrumental goals expressed in one or other folk ontology and subject to the very conflation we’ve agreed exists, so I suppose if newly-installed abstractions are sufficiently incompatible in the world model it can dislodge a lot of aggregate weight from the active goalset. A ‘healthy’ recovery from this sort of thing usually looks like someone identifying the in-fact-more-fundamental goals (which might putatively be the ones (or closer to the ones) installed by evolution, I don’t know).

Thanks again for this clarifying response, and I’m looking forward to more stuff from you and Alex and/or others in this area.
1. ↩︎
  By the way, I get a sense of ‘controversy signalling’ from some of this ‘shard theory’ stuff. I don’t have a good way to describe this, but it seems to make it harder for me to engage because I’m not sure what’s supposed to be new and for some reason I can’t really tell what I agree with. cf Richard’s comment. Please take this as a friendly note because I understand you’ve had a hard time getting some people to engage constructively (Alex told me something to the effect of ‘most people slide off this’). I’m afraid I don’t have positive textual/presentational advice here beyond this footnote.

Oliver Sourbut 5 Aug 2022 14:57 UTC
LW: 1 AF: 1
0 ∶ 0
AF
in reply to: Vaniver’s comment on: Reward is not the optimization target
I think Quintin^[1] is maybe alluding to the fact that in the limit of infinite counterfactual exploration then sure, the gradient in sample-based policy gradient estimation will push in that direction. But we don’t ever have infinite exploration (and we certainly don’t have counterfactual exploration; though we come very close in simulations with resets) so in pure non-lookahead (e.g. model free) sample-based policy gradient estimation, an action which has never been tried can not be reinforced (except as a side effect of generalisation by function approximation).

This seems right to me and it’s a nuance I’ve raised in a few conversations in the past. On the other hand kind of half the point of RL optimisation algorithms is to do ‘enough’ exploration! And furthermore (as I mentioned under Steven’s comment) I’m not confident that such simplistic RL is the one that will scale to AGI first. cf various impressive results from DeepMind over the years which use lots of shenanigans besides plain old sample-based policy gradient estimation (including model-based lookahead as in the Alpha and Mu gang). But maybe!
1. ↩︎
  This is a guess and I haven’t spoken to Quintin about this—Quintin, feel free to clarify/contradict

Oliver Sourbut 26 Jul 2022 9:13 UTC
1 point
1 ∶ 0
in reply to: Steven Byrnes’s comment on: Reward is not the optimization target
FWIW I upvoted but disagree with the end part (hurray for more nuance in voting!)

I think “reward is the antecedent-computation-reinforcer” will probably be true in RL algorithms that scale to AGI

At least from my epistemic position there looks like an explanation/communication gap here: I don’t think we can be as confident of this. To me this claim seems to preclude ‘creative’ forward-looking exploratory behaviour and model-based planning, which have more of a probingness and less of a merely-antecedent-computation-reinforcingness. But I see other comments from you here which talk about foresighted exploration (and foresighted non-exploration!) and I know you’ve written about these things at length. How are you squaring/nuancing these things? (Silence or a link to an already-written post will not be deemed rude.)
What links here?
- Oliver Sourbut's comment on Reward is not the optimization target by TurnTrout (5 Aug 2022 14:57 UTC; 1 point)

Oliver Sourbut 26 Jul 2022 8:55 UTC
4 points
1 ∶ 0
on: Reward is not the optimization target
In my ontology, there’s only an inner alignment problem: How do we grow good cognition inside of the trained agent?

This seems like a great takeaway and the part I agree with most here, although probably stated less strongly. Did you see Richard Ngo’s Shaping Safer Goals (2020) or my Motivations, Natural Selection, and Curriculum Engineering (2021) responding to it^[1]? Both relate to this sort of picture.

So the RL agent’s algorithm won’t make it e.g. explore wireheading either, and so the convergence theorems don’t apply even a little—even in spirit… I started off analyzing model-free actor-based approaches, but have also considered a few model-based setups

For various reasons I expect model-based RL to be a more viable path to AGI, in main because I think creative exploration is a missing ingredient addressing reward sparsity and the computational complexity barrier to tree-ish planning. Maybe a sufficiently carefully constructed curriculum can get over these, but that’s likely to be a really substantial additional hurdle, perhaps dominating the engineering effort, and perhaps simply intractable.

I also expect model-based + creative exploration^[2] to be much more readily able to make exploratory leaps, perhaps including wireheading-like activities. cf humans who aren’t all that creative but still find ever more inventive ways to wirehead—as a society quite a lot of selection and intelligent design has gone into setting up incentive structures to push people away from wireheading-like activities. Also, in humans, because our hardware is pretty messy and difficult to wirehead, such activities also typically harm or destroy capability, which selects against. But in general I don’t expect wireheading to necessarily harm capability.

So we definitely can’t rule out agents which strongly (and not just weakly) value antecedent-computation-reinforcement. But it’s also not the overdetermined default outcome. More on that in future essays.

Looking forward to it!

p.s. I’m surprised you think that RL researchers on the whole in fact believe that RL produces reward-maximisers but your (few) pieces of evidence do indeed seem to suggest that! I suppose on the whole the apparent ‘surprisingness’ of the concept of inner misalignment should also point the same way. I’d still err toward assuming a mixture of sloppy language and actual-mistakenness.
1. ↩︎
  Warning: both are quite verbose in my opinion and I expect both would be shorter if more time had been taken!
2. ↩︎
  By the way ‘creative exploration’ is mostly magic to me but I have reason to think it relates to temporal abstraction and recomposition in planning
What links here?
- Oliver Sourbut's comment on Reward is not the optimization target by TurnTrout (26 Jul 2022 9:13 UTC; 1 point)

Oliver Sourbut 24 Jul 2022 17:28 UTC
1 point
on: Conditions for mathematical equivalence of Stochastic Gradient Descent and Natural Selection
I’m informed^[1] that the concept I employed in the ‘recovering the equivalence’ section of the ‘fixed parts’ and ‘in flight mutations’ is similar to the ‘coalescence’ of ‘Coalescent theory’ which is an apparently relatively niche Biology tool, whose applications appear interesting (if unrelated) from a cursory look.
1. ↩︎
  by Holly Elmore, thanks!