Communications lead at MIRI. Unless otherwise indicated, my posts and comments here reflect my own views, and not necessarily my employer’s.
Rob Bensinger
Karma: 9,814
A couple minutes after I wrote this question I found out Scott Alexander said July 29:
I don’t have the energy to write a 5000 word blog post explaining my reasoning, but I think ≤10% chance HCQ has clinically significant effects against COVID, chances of really impressive effects even lower.
What’s your current epistemic state re hydroxychloroquine?
Abram added a lot of additional material to this today: https://www.lesswrong.com/posts/9vYg8MyLL4cMMaPQJ/updates-and-additions-to-embedded-agency.
Updates and additions to “Embedded Agency”
From a January 2017 Facebook conversation:
Rob B: I gather Ra is to a first approximation just ‘the sense that things are impersonally respectable / objective / authoritative / credible / prestigious / etc. based only on superficial indirect indicators of excellence.’
Ruby B: I too feel like I do not understand Ra. [...] Moloch, in my mind, was very clearly defined. For any given thing, I could tell you confidently whether it was Moloch or not. I can’t do that with Ra. Also, Moloch is a single clear concept while Ra seems to be a vague cluster if it’s anything. [...]
Rob B: Is there anything confusing or off about the idea that Ra is ‘respectability and prestige maintained via surface-level correlates of useful/valuable things that are not themselves useful/valuable (in the context at hand)’? Either for making sense of Sarah’s post or for applying the concept to real-world phenomena?
Ruby B: Yes, there is something off about that summary since the original post seems to contain a lot more than “seeking prestige via optimizing for correlates of value than actual value”. [...] If your summary is at the heart of it, there are some links missing to the “hates introspection”, “defends itself with vagueness, confusion, incoherence.” [...]
Rob B: There are two ideas here:
(1) “a drive to seek prestige by optimizing for correlates of value that aren’t themselves valuable”
(2) “a (particular) drive toward inconsistency / confusion / vagueness / ambiguity”
The connection between these two ideas is this paragraph in Sarah’s essay:
”‘Respectability’ turns out to be incoherent quite often — i.e. if you have any consistent model of the world you often have to take extreme or novel positions as a logical conclusion from your assumptions. To Ra, disrespectability is damnation, and thus consistent thought is suspect.”
1 is the core idea that Sarah wants to point to when she says “Ra”. 2 is a particular phenomenon that Sarah claims Ra tends to cause (though obviously lots of other things can cause fuzzy/inconsistent thinking too, and a drive toward such). Specifically, Sarah is defining Ra as 1, and then making the empirical claim that this is a commonplace drive, that pursuing any practical or intellectual project sufficiently consistently will at least occasionally require one to either sacrifice epistemics or sacrifice prestige, and that the drive is powerful enough that a lot of people do end up sacrificing epistemics when that conflict arises.Ruby B: Okay, yeah, I can start to see that. Thanks for making it clearer to me, Rob!
Rob B: I think Sarah’s essay is useful and coherent, but weirdly structured: she writes a bunch of poetry and mentions a bunch of accidental (and metaphorical, synesthetic, etc.) properties of Ra before she starts to delve into Ra’s essential properties. I think part of why I didn’t find it confusing was that I skimmed the early sections and got to the later parts of the essay that were more speaking-to-the-heart-of-the-issue, then read it back in reverse order. :P So I got to relatively clear things like the Horus (/ manifest usefulness / value / prestige-for-good-reasons) vs. Ra (empty respectability / shallow indicators of value / prestige-based-on-superficial-correlates-of-excellence) contrast first:
”Horus likes organization, clarity, intelligence, money, excellence, and power — and these things are genuinely valuable. If you want to accomplish big goals, it is perfectly rational to seek them, because they’re force multipliers. Pursuit of force multipliers — that is, pursuit of power — is not inherently Ra. There is nothing Ra-like, for instance, about noticing that software is a fully general force multiplier and trying to invest in or make better software. Ra comes in when you start admiring force multipliers for no specific goal, just because they’re shiny.”
And:
”When someone is willing to work for prestige, but not merely for money or intrinsic interest, they’re being influenced by Ra. The love of prestige is not only about seeking ‘status’ (as it cashes out to things like high quality of life, admiration, sex), but about trying to be an insider within a prestigious institution.”
(One of the key claims Sarah makes about respectability and prestige maintained via surface-level correlates of useful/valuable things that are not themselves useful/valuable (/ Ra) is that this kind of respectability accrues much more readily to institutions, organizations, and abstractions than to individuals. Thus a lot of the post is about how idealized abstractions and austere institutions trigger this lost-purposes-of-prestige mindset more readily, which I gather is because it’s harder to idealize something concrete and tangible and weak, like an individual person. Or maybe it has to do with the fact that it’s harder to concretely visualize the proper function and work of something that’s more abstract and large-scale, so it’s easier to lose sight of the rationale for what you’re seeing?)
”Seen through Ra-goggles, giving money to some particular man to spend on the causes he thinks best is weird and disturbing; putting money into a foundation, to exist in perpetuity, is respectable and appropriate. The impression that it is run collectively, by ‘the institution’ rather than any individual persons, makes it seem more Ra-like, and therefore more appealing.”
All of that stuff makes sense. The earlier stuff from the first 2 sections of the post doesn’t illuminate much, I think, unless you already have a more specific sense of what Sarah means by “Ra” from the later sections.Ruby B: Your restructuring and rephrasing is vastly more comprehensible. That said, poetry and poetic imagery is nice and I don’t begrudge Sarah her attempt.
And given your explanation, perhaps your summary description could be made slightly more comprehensive (though less comprehensible) like so:“Ra is a drive to seek prestige by optimizing for correlates of value that aren’t themselves valuable because you have forgotten the point of the correlates was to attain actual value.” [...]
Rob B: Maybe “Ra is a drive to seek prestige by optimizing for correlates of value, in contexts where the correlates are not themselves valuable but this fact is made non-obvious by the correlate’s abstract/impersonal/far-mode-evoking nature”?
From Facebook:
Mark Norris Lance: [...] There is a long history of differential evaluation of actions taken by grassroots groups and similar actions taken by elites or those in power. This is evident when we discuss violence. If a low-power group places someone under their control it is kidnapping. If they assess their crimes or punish them for it, it is mob justice or vigilanteism. [...]
John Maxwell: Does the low power group in question have a democratic process for appointing judges who then issue arrest warrants?That’s a key issue for me… “Mob rule” is bad because the process mobs use to make their judgements are bad. Doubly so if the mob attacks anyone who points that out.
A common crime that modern mobs accuse people of is defending bad people. But if people can be convicted of defending bad people, that corrupts the entire justice process, because the only way we can figure out if someone really is bad is by hearing what can be said in their defense.
Yet almost everyone agrees the world will likely be importantly different by the time advanced AGI arrives.
Why do you think this? My default assumption is generally that the world won’t be super different from how it looks today in strategically relevant ways. (Maybe it will be, but I don’t see a strong reason to assume that, though I strongly endorse thinking about big possible changes!)
A part I liked and thought was well-explained:
I think there’s a strong argument for deception being simpler than corrigibility. Corrigibility has some fundamental difficulties in terms of… If you’re imagining gradient descent process, which is looking at a proxy aligned model and is trying to modify it so that it makes use of this rich input data, it has to do some really weird things to make corrigibility work.
It has to first make a very robust pointer. With corrigibility, if it’s pointing at all incorrectly to the wrong thing in the input data, wrong thing in the world model, the corrigible optimizer won’t correct that pointer. It’ll just be like, “Well, I have this pointer. I’m just trying to optimize for what this thing is pointing for,” and if that pointer is pointing at a proxy instead, you’ll just optimize that proxy. And so you have this very difficult problem of building robust pointers. With deception, you don’t have this problem. A deceptive model, if it realizes the loss function is different than what it thought, it’ll just change to doing the new loss function. It’s actually much more robust to new information because it’s trying to do this instrumentally. And so in a new situation, if it realizes that the loss function is different, it’s just going to automatically change because it’ll realize that’s the better thing to do instrumentally.
The last time I saw it mentioned that COVID-19 can cause pulmonary fibrosis, it was in the context of autopsies. Do we have any more evidence about whether fibrosis is occurring in survivors, and if so about how common it is?
Devoodooifying Psychology says “the best studies now suggest that the placebo effect is probably very weak and limited to controlling pain”.
Lucas Perry: I guess I imagine the coordination here is that information on relative training competitiveness and performance competitiveness in systems is evaluated within AI companies and then possibly fed to high power decision makers who exist in strategy and governance for coming up with the correct strategy, given the landscape of companies and AI systems which exist?
Evan Hubinger: Yeah, that’s right.
I asked Evan about this and he said he misheard Lucas as asking roughly ‘Are training competitiveness and performance competitiveness important for researchers at groups like GovAI to think about?’; Evan wasn’t intending to call for “high power decision makers who exist in strategy and governance” to set strategy for AI companies.
Transcription errors:
I think it’s useful to sort of have in the back of your mind this analogy to evolution, but I would also be careful not to take it too far. I imagine that everything is going to generalize to the case of machine learning because it is a different process.
Should be “I think it’s useful to sort of have in the back of your mind this analogy to evolution, but I would also be careful not to take it too far and imagine that everything is going to generalize to the case of machine learning, because it is a different process.”
HDH process
Should be “HCH process”.
If there’s deployment episodes than training episodes, and it just cares about how many times it goes through the blue door or the green arrow, or whatever, the green arrow is a proxy objective, and so if it gives up the fact that it’s optimizing for the green arrow, it’s going to get modified to not do that anymore.
Should be “If there’s more deployment episodes than training episodes, …”.
[Epistemic status: Piecemeal wild speculation; not the kind of reasoning you should gamble the future on.]
Some things that make me think suffering (or ‘pain-style suffering’ specifically) might be surprisingly neurologically conditional and/or complex, and therefore more likely to be rare in non-human animals (and in subsystems of human brains, in AGI subsystems that aren’t highly optimized to function as high-fidelity models of humans, etc.):
1. Degen and Finlay’s social account of suffering above.
2. Which things we suffer from seems to depend heavily on mental narratives and mindset. See, e.g., Julia Galef’s Reflections on Pain, from the Burn Unit.
Pain management is one of the main things hypnosis appears to be useful for. Ability to cognitively regulate suffering is also one of the main claims of meditators, and seems related to existential psychotherapy’s claim that narratives are more important for well-being than material circumstances.
Even if suffering isn’t highly social (pace Degen and Finlay), its dependence on higher cognition suggests that it is much more complex and conditional than it might appear on initial introspection, which on its own reduces the probability of its showing up elsewhere: complex things are relatively unlikely a priori, are especially hard to evolve, and demand especially strong selection pressure if they’re to evolve and if they’re to be maintained.
(Note that suffering introspectively feels relatively basic, simple, and out of our control, even though it’s not. Note also that what things introspectively feel like is itself under selection pressure. If suffering felt complicated, derived, and dependent on our choices, then the whole suite of social thoughts and emotions related to deception and manipulation would be much more salient, both to sufferers and to people trying to evaluate others’ displays of suffering. This would muddle and complicate attempts by sufferers to consistently socially signal that their distress is important and real.)
3. When humans experience large sudden neurological changes and are able to remember and report on them, their later reports generally suggest positive states more often than negative ones. This seems true of near-death experiences and drug states, though the case of drugs is obviously filtered: the more pleasant and/or reinforcing drugs will generally be the ones that get used more.
Sometimes people report remembering that a state change was scary or disorienting. But they rarely report feeling agonizing pain, and they often either endorse having had the experience (with the benefit of hindsight), or report having enjoyed it at the time, or both.
This suggests that humans’ capacity for suffering (especially more ‘pain-like’ suffering, as opposed to fear or anxiety) may be fragile and complex. Many different ways of disrupting brain function seem to prevent suffering, suggesting suffering is the more difficult and conjunctive state for a brain to get itself into; you need more of the brain’s machinery to be in working order in order to pull it off.
4. Similarly, I frequently hear about dreams that are scary or disorienting, but I don’t think I’ve ever heard of someone recalling having experienced severe pain from a dream, even when they remember dreaming that they were being physically damaged.
This may be for reasons of selection: if dreams were more unpleasant, people would be less inclined to go to sleep and their health would suffer. But it’s interesting that scary dreams are nonetheless common. This again seems to point toward ‘states that are further from the typical human state are much more likely to be capable of things like fear or distress, than to be capable of suffering-laden physical agony.’
Thanks! :) I’m currently not planning to polish it; part of the appeal of cross-posting from Facebook for me is that I can keep it timeboxed by treating it as an artifact of something I already said. I guess someone else could cannibalize it into a prettier stand-alone post.
A second question is why horror films and games seem to be increasingly converging on the creepy/uncanny/mysterious cluster of things, rather than on the overtly physically threatening cluster—assuming this is a real trend. Some hypotheses about the second question:
A: Horror games and movies are increasingly optimizing for dread instead of terror these days, maybe because it’s novel—pure terror feels overdone and out-of-fashion. Or because dread just lends itself to a more fun multi-hour viewing/playing experience, because it’s more of a ‘slow burn’. Or something else.
B: Horror games aren’t optimizing for dread to the exclusion of terror; rather, they’ve discovered that dread is a better way to maximize terror.
Why would B be true?
One just-so story you could tell is that humans have multiple responses to possible dangers, ranging from ‘do some Machiavellian scheming to undermine a political rival’ to ‘avoid eating that weird-smelling food’ to ‘be cautious near that precipice’ to ‘attack’ to ‘flee’. Different emotions correspond to different priors on ‘what reaction is likeliest to be warranted here?‘, and different movie genres optimize for different sets of emotions. And optimizing for a particular emotion usually involves steering clear of things that prime a person to experience a different emotion—people want a ‘purer’ experience.
So one possibility is: big muscular agents, lion-like agents, etc. are likelier to be dangerous (in reality) than a decrepit corpse or a creepy child or a mysterious frail woman; but the correct response to hulking masculine agents is much more mixed between ‘fight / confront’ and ‘run away / avoid’, whereas the correct response to situations that evoke disgust, anxiety, uncertainty, and dread is a lot more skewed toward ‘run away / avoid’. And an excess of jumpscare-ish, heart-pounding terror does tend to incline people more toward running away than toward fighting back, so it might be that both terror and dread are better optimized in tandem, while ‘fight back’ partly competes with terror.
On this view, ‘ratchet up the intensity of danger’ matters less for fear intensity than ‘eliminate likely responses to the danger other than being extra-alert or fleeing’.
… Maybe because movie/game-makers these days just find it really easy to max out our danger-intensity detectors regardless? Pretty much everything in horror movies is pretty deadly relative to the kind of thing you’d regularly encounter in the ancestral environment, and group sizes in horror contexts tend to be smaller than ancestral group sizes.People who want to enjoy the emotions corresponding purely to the ‘fight’ response might be likelier to watch things like action movies. And indeed, action movies don’t make much use of jumpscares or terror (though they do like tension and adrenaline-pumping intensity).
Or perhaps there’s something more general going on, like:
Hypothesis C: Dread increases ‘general arousal / sensitivity to environmental stimuli’, and then terror can piggy-back off of that and get bigger scares.
Perhaps emotions like ‘disgust’ and ‘uncertainty’ also have this property, hence why horror movies often combine dread, disgust, and uncertainty with conventional terror. In contrast, hypothesis B seems to suggest that we should expect disgust and terror to mostly show up in disjoint sets of movies/games, because the correct response to ‘disease-ish things’ and the correct response to ‘physical attackers’ is very different.
[Epistemic status: Thinking out loud, just for fun, without having done any scholarship on the topic at all.]
It seems like a lot of horror games/movies are converging on things like ‘old people’, ‘diseased-looking people’, ‘psychologically ill people’, ‘women’, ‘children’, ‘dolls’, etc. as particularly scary.
Why would that be, from an evolutionary perspective? If horror is about fear, and fear is about protecting the fearful from threats, why would weird / uncanny / out-of-evolutionary-distribution threats have a bigger impact than e.g. ‘lots of human warriors coming to attack you’ or ‘a big predator-looking thing stalking you’, which are closer to the biggest things you’d want to worry about in our environment of evolutionary adaptedness? Why are shambling, decrepit things more of a horror staple than big bulky things with claws?
(I mean, both are popular, so maybe this isn’t a real phenomenon. I at least subjectively feel as though those uncanny things are scarier than super-lions or super-snakes.)
Maybe we should distinguish between two clusters of fear-ish emotions:
Terror. This is closer to the fight-or-flight response of ‘act quick because you’re in imminent danger’. It’s a panicky ‘go go go go go!!’ type of feeling, like when a jumpscare happens or when you’re running from a monster in a game.
Dread. This is more like feeling freaked out or creeped out, and it can occur alongside terror, or it can occur separately. It seems to be triggered less by ‘imminent danger’ than by ambiguous warning signs of danger.
So, a first question is why uncanny, mysterious, ‘unnatural’ phenomena often cause the most dread, even though they thereby become less similar to phenomena that actually posed the largest dangers to us ancestrally. (E.g., big hulking people with giant spears or snakes/dragons or werewolves correlate more with ‘things dangerous to our ancestors’ than decrepit zombies. Sure, creepiness maybe requires that the threat be ‘ambiguous’, but then why isn’t an ambiguous shadow of a maybe-snake or maybe-hulking-monster creepier than an obviously-frail/sickly monster?)
Plausibly part of the answer is that more mysterious, inexplicable phenomena are harder to control, and dread is the brain’s way of saying something like ‘this situation looks hard to control in a way that makes me want to avoid situations like this’.
Terror-inspiring things like jumpscares have relatively simple triggers corresponding to a relatively simple response—usually fleeing. Dread-inspiring things like ‘the local wildlife has gotten eerily quiet’ have subtler and more context-sensitive triggers corresponding to responses like ‘don’t necessarily rush into any hasty action, but do pay extra close attention to your environment, and if you can do something to get away from the stimuli that are giving you these unpleasant uneasy feelings, maybe prioritize doing that’.
From https://twitter.com/JonHaidt/status/1166318786959609856:
Why are online political discussions perceived to contain elevated levels of hostility compared to offline discussions? In this manuscript, we leverage cross-national representative surveys and online behavioral experiments to [test] the mismatch hypothesis regarding this hostility gap. The mismatch hypothesis entails that novel features of online communication technology induce biased behavior and perceptions such that ordinary people are, e.g., less able to regulate negative emotions in online contexts. We test several versions of the mismatch hypothesis and find little to no evidence in all cases. Instead, online political hostility is committed by individuals who are predisposed to be hostile in all contexts. The perception that online discussions are more hostile seemingly emerges because other people are more likely to witness the actions of these individuals in the large, public network structure of online platforms compared to more private offline settings.
Rob Wiblin, August 2019:
It seems to me like guilt and shame function surprisingly poorly as motivators for good work in the modern world. Not only do they often not result in people getting things done at the time, they can create a positive feedback loop that makes people depressed and unproductive for months.
But then why have we evolved to feel them so strongly?
One possibility is that guilt and shame do work well, but their function is to stop us from doing bad things. In a world where there’s only a few things you can do, it’s clear how to do them all, and the priority is to stay away from a few especially bad options, that’s helpful.
But to do good skilled work, it’s not enough to know what you shouldn’t do — e.g. procrastinate. The main problem is figuring out what out of the million things you might do is the right one, and staying focussed on it. And for that curiosity or excitement or pride are much more effective. You need to be pulled in the right direction, not merely pushed away from doing nothing, or severely violating a social norm.
Another second variation on the same theme would be that modern work is different from the tasks our hunter gatherer ancestors did in all sorts of ways that can make it less motivating. In the past just feeling guilt about e.g. being lazy, was enough to get us to go gather some berries, but now for most of us, it isn’t. So guilt fails, and then we feel even more guilty, and then we’re even less energetic, so it fails again, etc.
A third possibility is that shame and guilt are primarily about motivating you to fit into a group and go along with its peculiar norms. But in a modern workplace that’s not the main thing most of us are lacking. Rather we need to be inspired by something we’re working on and give it enough focussed attention long enough to produce an interesting product.
Any other theories? Or maybe you think guilt and shame do work well?
Cf. Brienne Yudkowsky on shame and the discussion on https://www.facebook.com/robbensinger/posts/10160749026995447.
From an April 2019 Facebook discussion:
Rob Bensinger: avacyn:
I think one strong argument in favor of eating meat is that beef cattle (esp. grass-fed) might have net positive lives. If this is true, then the utilitarian line is to 1) eat more beef to increase demand, 2) continue advocating for welfare reforms that will make cows’ lives even more positive.
Beef cattle are different than e.g. factory farmed chicken in that they live a long time (around 3 years on average vs 6-7 weeks for broilers), and spend much of their lives grazing on stockers where they might have natural-ish lives.
Another argument in favor of eating beef is that it tends to lead to deforestation, which decreases total wild animal habitat, which one might think are worse than beef farms.
… I love how EA does veganism / animal welfare things. It’s really good.
(From the comment section on https://forum.effectivealtruism.org/posts/TyLxMrByKuCmzZx6b/reasons-to-eat-meat)
[… Note that in posting this I’m not intending] to advocate for a specific intervention; it’s more that it makes me happy to see thorough and outside-the-box reasoning from folks who are trying to help others, whether or not they have the same background views as me.
Jonathan Salter: Even if this line of reasoning might technically be correct in a narrow, first order effects type way, my intuition tells me that that sort of behaviour would lessen EAs credibility when pushing animal welfare messages, and that spreading general anti-speciesist norms and values to be more important in the long run. Just my two cents though.
Rob Bensinger: My model of what EA should be shooting for is that it should establish its reputation as
’that group of people that engages in wonkish analyses and debates of moral issues at great length, and then actually acts on the conclusions they reach
″that group of people that does lots of cost-benefit analyses and is willing to consider really counter-intuitive concerns rather than rejecting unusual ideas out of hand
″that group of people that seems to be super concerned about its actual impact and nailing down all the details, rather than being content with good PR or moral signaling’
I think that’s the niche EA would occupy if it were going to have the biggest positive impact in the future. And given how diverse EA is and how many disagreements there already are, the ship may have already sailed on us being able to coordinate and converge on moral interventions without any public discussion of things like wild animal suffering.This is similar to a respect in which my views have changed about whether EAs and rationalists should become vegan en masse. In the past, I’ve given arguments like [in Inhuman Altruism and Revenge of the Meat People]:
A lot more EAs and rationalists should go vegan, because it really does seem like future generations will view 21st-century factory farming similar to how we view 19th-century slavery today. It would be great to be “ahead of the curve” for once, and to clearly show that we’re not just ‘unusually good on some moral questions’ but actually morally exemplary in all the important ways that we can achieve.
I think this is really standing in for two different arguments:
First, a reputational argument saying ‘veganism is an unusually clear signal that we’re willing to take big, costly steps to do the right thing, and that we’re not just armchair theorists or insular contrarians; so we should put more value on paying that signal in order to convince other people that we’re really serious about this save-the-world, help-others, actually-act-based-on-the-abstract-arguments thing’.
Second, an ideal-advisor-style argument saying ‘meat-eating is probably worse than it seems, because the analytic arguments strongly support veganism but social pressure and social intuitions don’t back up those arguments, so we probably won’t emotionally feel their full moral force’.
One objection I got to the first argument is that it seems like the marginal effort and attention of a lot of EAs could save a lot more lives if it were going to things that can have global effects, rather than small-scale personal effects. The reputational argument weighs against this, but there’s a reputational argument going in the other way (I believe due to Katja Grace [update: Katja Grace wrote an in-depth response at the time, but this particular argument seems to be due to Paul Christiano and Oliver Habryka]):‘What makes EA’s brand special and distinctive, and puts us in an unusual position to have an outsized impact on the world, is that we’re the group that gets really finicky and wonkish about EV and puts its energy into the things that seem highest-EV for the world. Prioritizing personal dietary choices over other, better uses of our time, and especially doing so for reputational or signaling reasons, seem like it actively goes against that unique aspect of EA, which makes it very questionable as a PR venture in the first place.‘
This still left me feeling, on a gut level, like the ‘history will view us as participants in an atrocity’ argument is a strong one—not as a reputational argument, just as an actual argument (by historical analogy) that there’s something morally wrong with participating in factory farming at all, even if we’re (in other aspects of our life) trying to actively oppose factory farming or even-larger atrocities.
Since then, a few things have made me feel like the latter argument’s force isn’t so strong. First, I’ve updated some on the object level about the probability that different species are conscious and that different species in particular circumstances have net-negative lives (though I still think there’s a high-enough-to-be-worth-massively-worrying-about probability that farmed chicken, beef, etc. are all causing immense amounts of suffering).
Second, I’ve realized that when I’ve done my ‘what would future generations think?’ ideal-advisor test in the past, I’ve actually been doing something weird. I’m taking 21st-century intuitions about which things matter most and are most emotionally salient, and projecting them forward to imagine a society where salience works the same way on the meta level, but the object-level social pressures/dynamics are a bit different. But it seems like that heuristic might have been the wrong one for past generations to use, if they wanted to make proper use of this ideal-advisor heuristic.
Jeremy Bentham’s exemplary forward-thinking moral views, for example, seem like a thing you’d achieve by going ‘imagine future generations that are just super reasonable and analytical about all these things, and view things as atrocities in proportion to what the strongest arguments say’, rather than by drawing analogies to things that present-day people find especially atrocious about the past.(People who have read Bentham: did Bentham ever use intuition pumps like either of these? If so, did either line of thinking seem like it actually played a role in how he reached his conclusions, as opposed to being arguments for persuading others?)
Imagine instead a future society that’s most horrified, above all else, by failures of reasoning process like ‘foreseeably allocating attention and effort to something other than the thing that looks highest-EV to you’. Imagine a visceral gut-level reaction to systematic decision-making errors (that foreseeably have very negative EV) even more severe than modernity’s most negative gut-level reactions to world events. Those failures of reasoning process, after all, are much more directly in your control (and much more influencable by moral praise and condemnation) than action outcomes. That seems like a hypothetical that pushes in a pretty different direction in a lot of these cases. (And one that converges more with the obvious direct ‘just do the best thing’ argument, which doesn’t need any defense.)
Twig!