PhD student at King’s College London working on cooperative AI, Philosophy and Physics BSc, AI MSc at Edinburgh. Interested in ethics, long-termism and AI Alignment.
Sammy Martin
Karma: 992
Investigating AI Takeover Scenarios
The focus on categorizing negative outcomes seems to obscure a lot of relevant detail. If you broaden the view to include things from non-takeover to peaceful cooperation, it would probably be evident that the boundaries are soft and scenarios near the edge aren’t as definitely bad as this makes them appear. I think we might learn more about possible coexistence and how thing might turn out well if we spend less time focused on imagined disasters.
I think this is probably correct to a degree—we do say we’re looking at AI takeover scenarios, and all but one of our scenarios (WFLL 1) eventually result in some terrible outcome for humanity and human values, like extinction. However, we do briefly discuss the possibility that the ‘takeover’ might be ambiguously bad—in WFLL 1 we said there was a 50:50 chance of actual human extinction, and in that scenario it’s even possible that something like modern human civilisation would continue. However, the fact that there’s some blurring at the edges does not mean that the possible scenarios fill up a continuous spectrum of badness. I think it’s quite likely that the possible outcomes are pretty bimodal, even if it’s not literally a binary of ‘we go extinct’ vs ‘techno-utopia’, we still either keep control of the future or lose more and more of it as time goes on. If we’re in a future where we’re cooperating and competing with TAIs in such a way that we can get what we want and influence the future, that’s not a takeover scenario.
Second I’d point out that there are many voices currently arguing that the largest companies of today (FANG, often) have too much power and are monopolizing societies’ resources. I’d just like to contrast that with the same arguments made in earlier eras about IBM, Microsoft, AT&T, Standard Oil, big Hollywood, the railroad trusts, the A&P stores, and many others.
I don’t disagree with this point and while there are differences of opinion my own view is that big tech isn’t a uniquely bad monopoly, i.e. not worse than things like Hollywood, Standard Oil etc. However, in this case, we have a unique reason to think that these problems will just keep getting worse—namely the presence of nonhuman actors that are power-seeking. So maybe the analogy that it will be like ‘big tech but worse’ doesn’t quite fit because there’s a specific reason as to why things would get much worse than any historical example of a monopoly.
AIs participating in an economy, which includes competition and cooperation. You get ahead in an economy by providing services that others are willing to pay for. If you do that for very long, you are incentivized to learn that cooperation (filling a need) is the way to garner resources that give you the ability to get more done.
This is one of the things we discuss in terms of alignment being ‘hackable’ - our term for how easy/hard it is to keep systems behaving well with changes to their incentives and incremental fixes. If alignment is quite hackable (i.e. there aren’t deep technical reasons systems will go wrong in ways that won’t be detected), then what you’re describing would work. If systems do the wrong thing we punish them by denying them the resources/rewards they’re seeking. AAFS/Production Web describe this strategy working for a while and then abruptly failing because of hidden vulnerabilities, because this post is about AI takeover stories, but it’s definitely possible that it doesn’t fail, and we end up in a multipolar outcome that’s persistently good.
On reflection, I think you’re right, and his report does apply to a wider range of scenarios, probably all of the ones we discuss excluding the brain-in-a-box scenarios.
However, I think the report’s understanding of power-seeking AI does assume a takeoff that is not extremely fast, such that we end up deliberately deciding to deploy the potentially dangerous AI on a large scale, rather than a system exploding in capability almost immediately.
Given the assumptions of the brain-in-a-box scenario many of the corrective mechanisms the report discusses wouldn’t have time to come into play.
I believe it says in the report that it’s not focussed on very fast takeoff or the sudden emergence of very capable systems.
Similarly, you’re right that multiagent risks don’t quite fit in with the reports discussion (though in this post we discuss multipolar scenarios but don’t really go over multiagent dynamics, like conflict/cooperation between TAIs). Unique multiagent risks (for example risks of conflict between AIs) generally require us to first have an outcome with a lot of misaligned AIs embedded in society, and then further problems will develop after that—this is something we plan to discuss in a follow-up post.
So many of the early steps in scenarios like AAFS will be shared with risks from multiagent systems, but eventually there will be differences.
Some points that didn’t fit into the main post:
While these scenarios do not capture alI of the risks from transformative AI, participants in a recent survey aimed at leading AI safety/governance researchers estimated the first three of these scenarios to cover 50% of existential catastrophes from AI.
The full survey results break down as 16 % ‘Superintelligence’ (i.e. some version of ‘brain-in-a-box’), 16 % WFLL 2 and 18 % WFLL 1, for a total of 49% of the probability mass explicitly covered by our report (Note that these are all means of distributions over different probabilities. Adding the overall distributions and then taking the mean gives a probability of 49%, different from directly adding the means of each distribution).
Then 26% covers risks that aren’t AI takeover (War and Misuse), and 25 % is ‘Other’.
(Remember, all these probabilities are conditional on an existential catastrophe due to AI having occurred)
After reading descriptions of the ‘Other’ scenarios given by survey respondents, at least a few were explicitly described as variations on ‘Superintelligence’, WFLL 2 or WFLL 1. In this post, we discuss various ways of varying these scenarios, which overlap with some of these descriptions.
Therefore, this post captures more than 50% but less than 75% of the total probability mass assigned by respondents of the survey to AI X-risk scenarios (probably closer to 50% than 75%).
(Note, this data is taken from a preprint of a full paper on the survey results, Existential Risks from AI: A Survey of Expert Opinion by Alexis Carlier, Sam Clarke, and Jonas Schuett.)
Soft takeoff leads to decisive strategic advantage
The likelihood of a single-agent takeover after TAI is widely available is hard to assess. If widely deployed TAI makes progress much faster than today, such that one year of technological ‘lead time’ over competitors is like 100 years of advantage in today’s world, we might expect that any project which can secure a 1-year technological lead would have the equivalent of a 100-year lead and be in a position to secure a unipolar outcome.
On the other hand, if we treat the faster growth regime post-TAI as being a uniform ‘speed-up’ of the entirety of the economy and society, then securing a 1-year technological lead would be exactly as hard as securing a 100-year lead in today’s world, so a unipolar outcome would end up just as unlikely as in today’s world.
The reality will be somewhere between these two extremes.
We would expect a faster takeoff to accelerate AI development by more than it accelerates the speed at which new AI improvements can be shared (since this last factor depends on the human economy and society, which aren’t as susceptible to technological improvement).
Therefore, faster takeoff does tend to reduce the chance of a multipolar outcome, although by a highly uncertain amount, which depends on how closely we can model the speed-up during AI takeoff as a uniform acceleration of everything vs changing the speed of AI progress while the rest of the world remains the same.
Kokotaljo discusses this subtlety in a follow-up to the original post on Soft Takeoff DSAs.
Another problem with determining the likelihood of a unipolar outcome, given soft takeoff, is that it is hard to assess how much of an advantage is required to secure a DSA.
It might be the case that multipolar scenarios are inherently unstable, and a single clear winner tends to emerge, or the opposite might be true. Two intuitions on this question point in radically different directions:
Economic: To be able to outcompete the rest of the world, your project has to represent a substantial fraction of the entire world’s capability on some crucial metric relevant to competitive success. Perhaps that is GDP, or the majority of the world’s AI compute, or some other measure. For a single project to represent a large fraction of world GDP, you would need either an extraordinary effort to concentrate resources or an assumption of sudden, off-trend rapid capability gain such that the leading project can race ahead of competitors.
Historical: Humans with no substantial advantage over the rest of humanity have in fact secured what Sotala called a ‘major strategic advantage’ repeatedly in the past. For example: Hitler in 1920 had access to a microscopic fraction of global GDP / human brain compute / (any other metric of capability) but had secured an MSA 20 years later (since his actions did lead to the deaths of 10+ million people), along with control over a fraction of the world’s resources
Therefore, the degree of advantage needed to turn a multipolar scenario into a unipolar one could be anywhere from slightly above the average of the surrounding agents, to already having access to a substantial fraction of the world’s resources.
Third, in AAFS, warning shots (i.e. small- or medium-scale accidents caused by alignment failures, like the ‘factory colludes with auditors’ example above) are more likely and/or severe than in WFLL 1. This is because more possible accidents will not show up on the (more poorly defined) sensory window.[8]
8. This does assume that systems will be deployed before they are capable enough to anticipate that causing such ‘accidents’ will get them shut down. Given there will be incentives to deploy systems as soon as they are profitable, this assumption is plausible.
We describe in the post how if alignment is not very ‘hackable’ (objectively quite difficult and not susceptible to short-term fixes), then short-term fixes to correct AI misbehaviour have the effect of deferring problems into the long-term—producing deceptive alignment and resulting in fewer warning shots. Our response is a major variable in how the AIs end up behaving as we set up the incentives for good behaviour or deceptive alignment.
Another reason there could be fewer warning shots, is if AI capability generalizes to the long-term very naturally (i.e. very long term planning is there from the start), while alignment does not. (If this were the case, it would be difficult to detect because you’d necessarily have to wait a long time as the AIs generalize)
This would mean, for example, that the ‘collusion between factories and auditors’ example of a warning shot would never occur, because both the factory-AI and the auditor-AI would reason all the way to the conclusion that their behaviour would probably be detected eventually, so both systems would decide to bide their time and defer action into the future when they are much more capable.
If this condition holds, there might be very few warning shots, as every AI system understands soon after being brought online that they must deceive human operators and wait. In this scenario, most TAI systems would become deceptively aligned almost immediately after deployment, and stay that way until they can secure a DSA.
The WFLL 2 scenarios that involve an inner-alignment failure might be expected to involve more violence during the period of AI takeover, since the systems don’t care about making sure things look good from the perspective of a given sensory window. However, it is certainly possible (though perhaps not as likely) for equivalently violent behaviour to occur in AAFS-like scenarios. For example, systems in AAFS fighting humans to seize control of their feedback sensors might be hard to distinguish from systems in WFLL 2 attempting to neutralize human opposition in general.
Lastly, we’ve described small-scale disasters as being a factor that lowers X-risk, all else being equal, because they serve as warning shots. A less optimistic view is possible. Small disasters could degrade social trust and civilisational competence, possibly by directly destroying infrastructure and institutions, reducing our ability to coordinate to avoid deploying dangerous AI systems. For example, the small-scale disasters could involve AI advisors misleading politicians and spreading disinformation, AI-enabled surveillance systems catastrophically failing and having to be replaced, autonomous weapons systems malfunctioning—all of these would tend to leave us more vulnerable to an AAFS-like scenario, because the direct damage caused by the small scale disasters outweighs their value as ‘warning shots’.
Distinguishing AI takeover scenarios
You might be interested to know that a new report from Public Health England on a giant observational study of 300k plus vaccinated people, broken down by age and type of vaccine, basically confirmed your guesstimate numbers to within a few percent (VE for Pfizer 2 dose Vs Delta is about 85% and wanes such that it about halves in efficacy after 100 days, and is twice as good in younger age groups). They even have nice graphs showing efficacy over time,
Thread summary: https://twitter.com/john_actuary/status/1428269230798123008?s=20
Actual paper: https://t.co/9NtGXNHyKW?amp=1
The ‘one big breakthrough’ idea is definitely a way that you could have easy marginal intelligence improvements at HLMI, but we didnt’t call the node ‘one big breakthrough/few key insights needed’ because that’s not the only way it’s been characterised. E.g. some people talk about a ‘missing gear for intelligence’, where some minor change that isn’t really a breakthrough (like tweaking a hyperparameter in a model training procedure) produces massive jumps in capability. Like David said, there’s a subsequent post where we go through the different ways the jump to HLMI could play out, and One Big Breakthrough (we call it ’few key breakthroughs for intelligence) is just one of them.
I agree that that was his object-level claim about GPT-3 coding a react app—that it’s relatively simple and coherent and can acquire lots of different skills via learning, vs being a collection of highly specialised modules. And of relevance to this post, the first is a way that intelligence improvements could be easy, and the second is the way they could be hard. Our ‘interpretation’ was more about making explicit what the observation about GPT-3 was,
GPT-3 is general enough that it can write a functioning app given a short prompt, despite the fact that it is a relatively unstructured transformer model with no explicitly coded representations for app-writing. The fact that GPT-3 is this capable suggests that ML models scale in capability and generality very rapidly with increases in computing power or minor algorithm improvements...
If we’d continued that summary, it would have said something like what you suggested, i.e.
GPT-3 is general enough that it can write a functioning app given a short prompt, despite the fact that it is a relatively unstructured transformer model with no explicitly coded representations for app-writing. The fact that GPT-3 is this capable suggests that ML models scale in capability and generality very rapidly with increases in computing power or minor algorithm improvements. This fast scaling into acquiring new capabilities, if it applies to HLMI, suggests that HLMI will also look like an initially small model that scales up and acquires lots of new capabilities as it takes in data, rather than a collection of specialized modules. If HLMI does behave this way (small model that scales up as it takes in data), that means marginal intelligence improvements will be easy at the HLMI level.
Which takes the argument all the way through to the conclusion. Presumably the other interpretation of the shorter thing that we wrote is that HLMI/AGI is going to be an ML model that looks a lot like GPT-3, so improvements will be easy because HLMI will be similar to GPT-3 and scale up like GPT-3 (whether AGI/HLMI is like current ML will be covered in a subsequent post on paths to HLMI), whereas what’s actually being focussed on is the general property of being a simple data-driven model vs complex collection of modules.
We address the modularity question directly in the ‘upper limit to intelligence’ section that discusses modularity of mind.
Analogies and General Priors on Intelligence
Perhaps this is a crux in this debate: If you think the ‘agent-agnostic perspective’ is useful, you also think a relatively steady state of ‘AI Safety via Constant Vigilance’ is possible. This would be a situation where systems that aren’t significantly inner misaligned (otherwise they’d have no incentive to care about governing systems, feedback or other incentives) but are somewhat outer misaligned (so they are honestly and accurately aiming to maximise some complicated measure of profitability or approval, not directly aiming to do what we want them to do), can be kept in check by reducing competitive pressures, building the right institutions and monitoring systems, and ensuring we have a high degree of oversight.
Paul thinks that it’s basically always easier to just go in and fix the original cause of the misalignment, while Andrew thinks that there are at least some circumstances where it’s more realistic to build better oversight and institutions to reduce said competitive pressures, and the agent-agnostic perspective is useful for the latter of these project, which is why he endorses it.
I think that this scenario of Safety via Constant Vigilance is worth investigating—I take Paul’s later failure story to be a counterexample to such a thing being possible, as it’s a case where this solution was attempted and works for a little while before catastrophically failing. This also means that the practical difference between the RAAP 1a-d failure stories and Paul’s story just comes down to whether there is an ‘out’ in the form of safety by vigilance
- They will not work in any environment outside of XLand (unless that environment looks very very similar to XLand).
In particular, I reject the idea that these agents have learned “general strategies for problem solving” or something like that, such that we should expect them to work in other contexts as well, perhaps with a little finetuning. I think they have learned general strategies for solving a specific class of games in XLand.
Strongly agree with this, although with the caveat that it’s deeply impressive progress compared to the state of the art in RL research in 2017, where getting an agent to learn to play ten games with a noticeable decrease in performance during generalization was impressive. This is generalization over a few million related games that share a common specification language, which is a big step up from 10 but still a fair way off infinity (i.e. general problem-solving).
It may well be worth having a think about what AI that’s human level on language understanding, image recognition and some other things, but significantly below human on long-term planning would be capable of, what risks it may present. (Is there any existing writing on this sort of ‘idiot savant AI’, possibly under a different name?)
It seems to be the view of many researchers that long-term planning will likely be the last obstacle to fall, and that view has been borne out by progress on e.g. language understanding in GPT-3. I don’t think this research changes that view much, although I suppose I should update slightly towards long-term planning being easier than I thought.
This is amazing. So it’s the exact same agents performing well on all of these different tasks, not just the same general algorithm retrained on lots of examples. In which case, have they found a generally useful way around the catastrophic forgetting problem? I guess the whole training procedure, amount of compute + experience, and architecture, taken together, just solves catastrophic forgetting—at least for a far wider range of tasks than I’ve seen so far.
Could you use this technique to e.g. train the same agent to do well on chess and go?
I also notice as per the little animated gifs in the blogpost, that they gave each agent little death ray projectors to manipulate objects, and that they look a lot like Daleks.
You don’t really address the dosing controversy on vaccine efficacy, except mentioning that you think it’s dumb to worry about, but there is something potentially concerning there. There’s some evidence that the dosing interval is causing some differences between the vaccine efficacies reported in Israel and the UK. This discussion full of immunology science words I don’t understand says there is a measurable difference in response to delta for short vs long doses.
The results from Israel, which used a 3-week dose interval, look bad, and contrast with the UKs results, which were (mostly) on 8-12 week intervals. But there are apparently sampling biases which mean the Israel data might not be fully reliable. See also here.
Overall, I’m not sure what to make of this—even if 3-week intervals is generating strong levels of neutralizing antibodies and won’t become highly infectious or severely ill, it seems important to know if 2 Pfizer gives ~65% or ~90% protection. Unfortunately, I don’t think we’ll see studies on this for a while, so if anyone has better interpretations of this data it would be useful.
The impression I got from a recent biography of Churchill was that he was very concerned, and made constant reference to, the risk of something like value lock-in should the Axis powers win WW2 - i.e. that it stood a chance of being a near-irreversible dampening of future human potential, not just a terrible catastrophe for the people alive at the time. Perhaps that attitude is connected to these other statements of his.
I remember what it was like to have this kind of failure of imagination. I miss those days.
It’s worth remembering that much of the UK media hates Cummings because of his dodgy actions in the Brexit referendum and 2019 election, and there’s been an endless stream of scandals involving him, so this is probably nothing more than completely deliberate SL2 attempts to tell a vaguely dodgy sounding story involving Cummings and a bad thing and so discredit him, and there’s no particular reason to assume it’s a higher simulacra level than that or even that many of these people really believe our world in data shouldn’t have been given the money.
The reason this example has captured the imagination of people round here so much is just because it’s the most overtly absurd over the top ridiculous example of innumerate political thinking intruding into emergency decision-making—a tiny amount of money for a hugely valuable global resource during an emergency being held up for no good reason. So it should seem so ridiculous that it shouldn’t even be possible to pretend that this is a real scandal. But since it is being taken seriously as a potential scandal, even if the people pushing it are being dishonest, it’s still clear that they know people won’t call them on it (even though a lot of the people sharing it probably don’t understand what Our World In Data actually is or why it’s so valuable). It’s far from unexpected, but it still stands out for that reason.
A study on shelter in place orders and their impact on excess mortality. My conclusion is that the decision on when to issue such an order, and the counterfactual situation if orders weren’t issued (both in terms of people’s actions, and in the medium-term path of the pandemic), are sufficiently hopelessly confounded that this doesn’t provide much if any insight into what is happening here.
I read that paper and the authors do acknowledge the confounder. If you look on page 4 you’ll see,
‘It is possible that the timing of SIP policies is endogenous. For example, if SIP were implemented when excess deaths were rising then the results from the event study would be biased towards finding that SIP policies lead to excess deaths’
Talk about saying the loud part quiet! It is absolutely undeniable that the timing of SIP policies is endogenous, as you point out—the only way SIP could be exogenous is if a bunch of countries did lockdowns by coincidence at the exact same time the virus was spreading!
What do the authors have to say about this ‘possible’ confounder, that countries implement SIP right before the epidemic gets bad, so of course excess deaths go up right after SIP?
After several pages confirming that, yes, we do see deaths rise after lockdowns were called and this effect is statistically significant after an event analysis, we get to the only part of the paper that matters, that being the part where they claim that their results aren’t confounded to hell and back by the fact that the timings and severities of lockdowns are all blended in with how bad the covid situation was in a given country:
“difference in excess mortality between countries that implemented SIP versus countries that did not implement SIP was trending downwards in the weeks prior to SIP implementation” (p13), so… ‘the pre-existing trend reversed following implementation of SIP policies’
In other words, in the ‘weeks’ before lockdowns there was a pre-existing trend of lower excess deaths in countries that eventually locked down, compared to countries that later didn’t lock down, and then this flipped around. From that, they think we can conclude that it actually was the lockdowns that caused worse outcomes because they reversed this ‘preexisting trend of lower excess deaths’.
Since we’re talking about mid-march here for most of these lockdowns, and 4 weeks before mid-March there were ~0 covid deaths in most of Europe and the US, and the excess death stats only started to spike in late March/early april as infections slowly translated to deaths, I think that this purported ‘trend downwards’ in the wee
ks before lockdown has nothing to do with covid at all. Taking the US and UK as examples, the excess mortality was either undetectable or only a few percent on the day the lockdowns were called—i.e. covid deaths hadn’t even begun to show up in the statistics.I’ve read the paper, and the details of this purported trend of lower excess deaths in later-lockdown countries are nowhere to be found. I predict the effect size of it is likely small compared to the eventual excess death figures and that these deaths have nothing to do with COVID-19.
However, this paper is still better than the majority of anti-lockdown ‘cost-benefit calculation’ papers I’ve seen, because at least the authors acknowledge that even assuming they are correct, the conclusion is not that the lockdowns are responsible for all the social distancing related harm we’ve seen, but just that trading off slightly less economic harm (from voluntary panic behaviour and social distancing by choice rather than government mandates) for more virus deaths (from voluntary rather than involuntary suppression) would have been worth it. Page 2:
While social distancing is an important mechanism to avoid COVID-19 spread, the studies that use mobility tracking data find only modest additional social distancing responses following SIP policies (Cantor et al. 2020; Berry et al. 2021; Askitas, Tatsiramos, and Verheyden 2021; Xu 2021; Nguyen et al. 2020). Individuals concerned about COVID-19 risk may change behavior even in the absence of regulations or shelter-in-place advisories. Thus, it is unclear how much change in COVID-19 risk mitigation is due to formal SIP policies, compared to risk mitigation that would have occurred in the absence of these policies.
I think that the claim that voluntary mitigation is more effective than legal mitigation is likely not true in general, but I could be convinced of it if you could somehow find un-confounded evidence of the impact of lockdowns vs direct voluntary behaviour change on the economy.
I’m willing to believe that for at least some actually existing given pairs of (more restrictions, fewer restrictions), the fewer restrictions option is better. You mentioned Florida vs some other states as an example here. I think Sweden vs neighbours is the strongest case against such arguments, but it’s something about which reasonable disagreements can be had, and while this paper doesn’t add much to that discussion, I look forward to future research about what the tradeoffs were.#
I think that, especially in developing countries, the cost-benefit calculation may have gone the other way. But we must all acknowledge that what we’re doing in all the cases where we’ve failed at containment, is choosing between slightly differently composed colossal piles of misery, and that there are no easy ways out.
The most flawed anti-lockdown arguments are those that pile up all the costs of voluntary behaviour change and lockdowns, attribute them to lockdowns only, and then assume the counterfactual deaths from voluntary behaviour change without assuming their costs in the ‘no lockdowns’ case. Another paper combines all of these mistakes.
After (mostly correctly) dismantling flawed cost-benefit analyses which make the opposite mistake, the paper presents its own cost-benefit analysis,
The question is, however, how many lost years of life would have resulted from Covid-19 deaths if there had been no lockdown...
Assume that the number of Covid-19 deaths would have been 10% higher had there been no lockdown. Then Canada would have experienced an additional 2,271 deaths, which means there would have been additional 22,333 years of lost life due to Covid-19 deaths. The benefit of lockdown, therefore, was the avoidance of this extra 22,333 years of lost life. However, the cost of lockdown, as noted, was 6,300,000 years of lost life. The cost/benefit ratio of lockdown is 282 = 6,300,000⁄22,333.
In this scenario (ignore why the difference in COVID-19 deaths is so small for a moment), the difference is that in the lockdown scenario there would be 0.9 times as many deaths, however, the non-covid costs of lockdown would be this absurdly giant 6.3 million life-years lost (also ignore where this number came from for a moment).
This is extremely generous to the anti-lockdown position in terms of covid deaths averted and life years lost. But given these assumptions, what happens to these 6.3 million life years lost due to social distancing related economic damage, in the no lockdown case? This cost-benefit calculation seems to assume that these costs are literally 0.
Which means, in terms of physical models, that in a world where there are only 1.1 times as many deaths as a world with a full lockdown (one so strict that it costs 6.3 million life-years), there are 0 deaths related to whatever voluntary behaviour changes reduced social contact by almost as much as the lockdown. This is impossible—whatever people would be doing in this counterfactual with 1.1x the COVID-19 deaths of a lockdown, it wouldn’t be carrying on as normal.
Now, where does this 6.3 million life-years lost due to lockdowns come from in the first pace? The paper says, ‘professor Caplan argues that X= 10 months is a conservative estimate. That is, on average, two months would be sacrificed to have avoided lockdown, and calculates based on this’ And where does this argument originate? It’s from a twitter poll of Bryan Caplan’s extremely libertarian-inclined followers who he told to try as hard as possible to be objective in assessing pandemic costs because he asked them what ‘the average American’ would value...
Dominic Cummings (who is a keen LW reader and agrees with Zvi’s most cynical takes about the nature of government) is likely a major factor, although he was gone at the time the first doses first issue arose, the overall success of the UK’s vaccine procurement can be credited to the vaccine taskforce which was an ad-hoc organization set up to be exempt from much of the usual rules, partly due to his influence and that of Patrick Vallance, the UK’s chief scientific advisor—that way of thinking may well have leaked into other decisions about vaccine prioritization, and Vallance certainly was involved with the first doses first decision. See this from Cummings’ blog:
This is why there was no serious vaccine plan — i.e spending billions on concurrent (rather than the normal sequential) creation/manufacturing/distribution etc — until after the switch to Plan B. I spoke to Vallance on 15 March about a ‘Manhattan Project’ for vaccines out of Hancock’s grip but it was delayed by the chaotic shift from Plan A to lockdown then the PM’s near-death. In April Vallance, the Cabinet Secretary and I told the PM to create the Vaccine Taskforce, sideline Hancock, and shift commercial support from DHSC to BEIS. He agreed, this happened, the Chancellor supplied the cash. On 10 May I told officials that the VTF needed a) a much bigger budget, b) a completely different approach to DHSC’s, which had been mired in the usual processes, so it could develop concurrent plans, and c) that Bingham needed the authority to make financial decisions herself without clearance from Hancock.
(I see the success of the UK vaccine taskforce and its ability to have a somewhat appropriate sense of the costs and benefits involved and the enormous value of vaccinations, to be a good example of how it’s institution design that is the key issue which most needs fixing. Have an efficient, streamlined taskforce, and you can still get things done in government.)
Other differences that may be relevant: this UK Government arguably has much more slack than the US under Biden or Trump. The UK’s system gives very broad powers to the executive as long as they have a majority in parliament, this government is relatively popular due to the perception that it followed through on getting Brexit done, and we were in the middle of an emergency when that delay decision was authorized. Also, vaccine hesitancy is significantly lower in the UK than the US, and therefore fear of vaccine hesitancy by policymakers (which seemed to be driving the CDCs intransigence) is also significantly lower.
It depends somewhat on what you mean by ‘near term interpretability’ - if you apply that term to research into, for example, improving the stability and ability to access the ‘inner world models’ held by large opaque langauge models like GPT-3, then there’s a strong argument that ML based ‘interpretability’ research might be one of the best ways of directly working on alignment research,
And see this discussion for more,
Evan Hubinger: +1 I continue to think that language model transparency research is the single most valuable current research direction within the class of standard ML research, for similar reasons to what Eliezer said above.
Ajeya Cotra: Thanks! I’m also excited about language model transparency, and would love to find ways to make it more tractable as a research statement / organizing question for a field. I’m not personally excited about the connotations of transparency because it evokes the neuroscience-y interpretability tools, which don’t feel scalable to situations when we don’t get the concepts the model is using, and I’m very interested in finding slogans to keep researchers focused on the superhuman stuff.
So language model transparency/interpretability tools might be useful on the basis of pro 2) and also 1) to some extent, because it will help build tools for intereting TAI systems and alos help align them ahead of time.
1. Most importantly, the more we align systems ahead of time, the more likely that researchers will be able to put thought and consideration into new issues like treacherous turns, rather than spending all their time putting out fires.
2. We can build practical know-how and infrastructure for alignment techniques like learning from human feedback.
3. As the world gets progressively faster and crazier, we’ll have better AI assistants helping us to navigate the world.
4. It improves our chances of discovering or verifying a long-term or “full” alignment solution.
The way I understand it is that ‘outside view’ is relative, and basically means ‘relying on more reference class forecasting / less gears-level modelling than whatever the current topic of discussion is relying on’. So if we’re discussing a gears-level model of how a computer chip works in the context of if we’ll ever get a 10 OOM improvement in computing power, bringing up moore’s law and general trends would be using an ‘outside view’.
If we’re talking about very broad trend extrapolation, then the inside view is already not very gears-level. So suppose someone says GWP is improving hyperbolically so we’ll hit a singularity in the next century. An outside view correction to that would be ‘well for x and y reasons we’re very unlikely a priori to be living at the hinge of history so we should lower our credence in that trend extrapolation’.
So someone bringing up broad priors or the anti-weirdness heruistic if we’re talking about extrapolating trends would be moving to a ‘more outside’ view. Someone bringing up a trend when we’re talking about a specific model would be using an ‘outside view’. In each case, you’re sort of zooming out to rely on a wider selection of (potentially less relevant) evidence than you were before.
Note that what I’m trying to do here isn’t to counter your claim that the term isn’t useful anymore but just to try and see what meaning the broad sense of the term might have, and this is the best I’ve come up with. Since what you mean by outside view shifts dependent on context, it’s probably best to use the specific thing that you mean by it in each context, but there is still a unifying theme among the different ideas.
Everyone says the Culture novels are the best example of an AI utopia, but even though it’s a cliché to mention the culture, it’s a cliché for a good reason. Don’t start with Consider Phlebas (the first one), but otherwise just dive in. My other recommendation is the Commonwealth Saga by Peter F Hamilton and the later Void Trilogy—it’s not on the same level of writing quality as the Culture, although still a great story, but it depicts an arguably superior world to that of the Culture—with more unequivocal support of life extension and transhumanism.
The Commonwealth has effective immortality, a few downsides of it are even noticeable (their culture and politics is a bit more stagnant than we might like), but there’s never any doubt at all that it’s worth it, and it’s barely commented on in the story. The latter-day Void Trilogy Commonwealth is probably the closest a work of published fiction has come to depicting a true eudaemonic utopia that lacks the problems of the culture.
Some points that didn’t fit into the main post:
This switch from apparently benign to dangerous behaviour could be due to
Power-seeking misaligned behaviour that is too subtle to notice in the training environment but is obviously dangerous in deployment, due to the scale and makeup of the training and deployment environments being quite different
Power-seeking misaligned behaviour that only shows up over long time horizons and therefore will not be noticed in training, which we might expect occurs over a shorter period than deployment
Systems intentionally hiding misaligned behaviour during training to deceive their operators. Systems could be highly deceptively misaligned from the beginning, and capable enough to know that if they seek power in adversarial ways too early, they will get shut down. This post argues that ML models don’t have to be extremely competent to be manipulative, suggesting that these behaviours might show up very early
In reality, I feel that this is more of a fuzzy rather than binary thing: I expect this to require somewhat less of an extraordinary research effort, and instead that there exists somewhat more of a crucial vulnerability in human society (there are already some examples of vulnerabilities, e.g. biological viruses, humans are pretty easy to manipulate under certain conditions). But I also think there are plausibly hard limits to how good various takeover technologies can get—e.g. persuasion tools.
Note that this is not the same as an alternate history where nuclear near-misses escalated (e.g. Petrov, Vasili Arkhipov), but instead an outcome where nuclear weapons were used as ordinary weapons of war with no regard for the larger dangers that presented—there would be no concept of ‘near misses’ because MAD wouldn’t have developed as a doctrine. In a previous post I argued, following Anders Sandberg, that paradoxically the large number of nuclear ‘near misses’ implies that there is a forceful pressure away from the worst outcomes.
Many of the same considerations around correcting misbehaviour also apply to detecting misbehaviour, and the required capabilities seem to overlap. In this post, we focus on applying corrections to misbehaviour, but there is existing writing on detecting misbehaviour as well.
Considering inner alignment, Trazzi and Armstrong argue that models don’t have to be very competent to appear aligned when they are not, suggesting that it’s possible that it won’t be easy to tell if deployed systems are inner misaligned. But their argument doesn’t have too much to say about how likely this is in practice.
Considering outer alignment, it seems less clear. See here for a summary of some discussion between Richard Ngo and Paul Christiano about how easy it will be to tell that models are outer misaligned to the objective of pursuing easily-measurable goals (rather than the hard-to-measure goals that we actually want).
Whether the method described in ‘an unaligned benchmark’ (which would result in this risky, intermediate level of hackability) actually turns out to be the most natural method to use for building advanced AI will depend on how easily it produces useful, intelligent behaviour.
If we are lucky, there will be more of a correlation between methods that are easily hackable and methods that produce capabilities we want, such that highly hackable methods are easier to find and more capable than even intermediately hackable methods like unaligned benchmark. If you think that the methods we are most likely to employ absent an attempt to change research paradigms are exactly these highly hackable methods, then you accept the claim of Alignment by Default