MTAIR project and CLR grantee, PhD student at King’s College London working on cooperative AI, Philosophy and Physics BSc, AI MSc at Edinburgh. Interested in philosophy, longtermism and AI Alignment. I write science fiction at ascentuniverse.wordpress.com
Sammy Martin
Karma: 1,317
The London School of Hygiene released a modelling paper describing some estimated effects on the UK of the Omicron wave. Mostly, it’s a lot of “the error bars on all these are giant, and we don’t have any clear idea what’s going to happen except that there will be a giant wave of infections by mid-Jan, unclear how that translates to deaths”.
If you assume no new measures and no behaviour change in the UK, you get a ridiculously enormous wave of infections that might peak with half of everyone getting infected, and that causes peak hospitalizations anywhere from 0.5-2x the UK’s previous peak from the alpha wave back in January (74k additional deaths on the high end, in the not-actually-possible-even-in-principle scenario).
Again, these scenarios aren’t really possible (since no behaviour change is assumed), but they are useful for getting a sense of what Omicron could be like and the likely timings of things in the UK.
What’s more useful for general readers is the paper’s table of assumptions about vaccine efficacy vs Omicron—this was released just after results showed VE of Pfizer + Pfizer booster against symptomatic infection is 75%,
LSHTM Vaccine Efficacy Assumptions And here’s the result from that Pfizer study:
Pfizer Booster VE Study I think the paper’s results are based on trying to guesstimate just what properties are reasonable/vaguely consistent with what we’ve seen. The highlights in yellow are the ones that will matter in the long run for most readers—efficacy for 3-dose courses on low and high assumptions about Omicron immune escape and Booster efficacy.
The range for Symptomatic Illness efficacy is 46.9 − 80.3 %, though as I said the recent Pfizer study said the efficacy was 71.4% and 75.5% protection vs. symptomatic disease for people who received AZ and Pfizer primary courses, 2 weeks post their boosters. Don’t know if that means we can bias ourselves towards the upper end of that range or not (1.9, 4.0, 5.0 times lower for the low, study result, high estimates of VE vs symptomatic infection).
The range for Hospitalization/mortality efficacy for a booster is 83.7-96.9%, or a 6.1 to 32.3 -fold reduction in mortality risk vs someone who isn’t vaccinated against omicron. Again, adjust accordingly based on your age, sex and health vs the UK population average. Also note this isn’t taking into account Omicron being intrinsically less severe, which it might well be.
Related, some guessing about which of the evasion scenarios we’re in
The lower VE numbers here might be the plausible end of a range, or might just be the authors covering all possibilities, no matter how remote or absurd.
Fingers crossed, the apparently good efficacy of a booster that wasn’t intended for anything like Omicron means that it’s giving you generally broad-based immunity that will work going forward against whatever next crazy variant evolution throws at us.
I keep seeing extremely hard-to-judge theories thrown around, generally by suspicious types, about vaccine-granted antibodies being uniquely brittle and setting us up for the emergence of immune evasive variants—for example here. Dominic Cummings recently alluded to a similar idea—vaccines only targeting the spike protein (even though that was the best way to make a highly effective vaccine), means variant evolution can more easily evade vaccine antibodies than if they were broader.
AFAIK this is something that was considered during vaccine development (the reasoning was that there’s fewer possible ways for the spike to change while keeping the covid virus still functioning, so targeting the spike would hopefully work against future variants), but which is in principle a problem for the future. Obvious caveat—you still get T-cell immunity no matter what’s going on with the antibodies, which is the underlying reason why all the assumptions in the above table are high for severe disease immunity, and since we’ll eventually end up in a scenario where most of the world has T-cell immunity to Covid from vaccines or infection, that will put a limit on the damage future waves can cause
Compare this,
[Shulman][22:18]
We’re in the Eliezerverse with huge kinks in loss graphs on automated programming/Putnam problems.
Not from scaling up inputs but from a local discovery that is much bigger in impact than the sorts of jumps we observe from things like Transformers.
[Yudkowsky][22:21]
but, sure, “huge kinks in loss graphs on automated programming / Putnam problems” sounds like something that is, if not mandated on my model, much more likely than it is in the Paulverse. though I am a bit surprised because I would not have expected Paul to be okay betting on that.
to this,
[Rohin] To the extent this is accurate, it doesn’t seem like you really get to make a bet that resolves before the end times, since you agree on basically everything until the point at which Eliezer predicts that you get the zero-to-one transition on the underlying driver of impact.
Eliezer does (at least weakly) expect more trend breaks before The End (even on metrics that aren’t qualitative measures of impressiveness/intelligence, but just things like model loss functions), despite the fact that Rohin’s summary of his view is (I think) roughly accurate.
What explains this? I think it’s something roughly like, part of the reason Eliezer expects a sudden transition when we reach the core of generality in the first place is because he thinks that’s how things usually go in the history of tech/AI progress—there’s also specific reasons to think it will happen in the case of finding the core of generality, but there are also general reasons. See e.g. this from Eliezer:
well, the Eliezerverse has more weird novel profitable things, because it has more weirdness
I take ‘more weirdness’ to mean something like more discoveries that induce sudden improvements out there in general.
So I think that’s why his view does make (weaker) differential predictions about earlier events that we can test, not because the zero-to-one core of generality hypothesis predicts anything about narrow AI progress, but because some of the beliefs that led to that hypothesis do.
We can see there’s two (connected) lines of argument and that Eliezer and Paul/Carl/Richard have different things to say on each − 1 is more localized and about seeing what we can learn about AGI specifically, and 2 is about reference class reasoning and what tech progress in general tells us about AGI:
Specific to AGI: What can we infer from human evolution and interrogating our understanding of general intelligence (?) about whether AGI will arrive suddenly?
Reference Class: What can we infer about AGI progress from the general record of technological progress, especially how common big impacts are when there’s lots of effort and investment?
My sense is that Eliezer answers
Big update for Eliezer’s view: This tells us a lot, in particular we learn evolution got to the core of generality quickly, so AI progress will probably get there quickly as well. Plus, Humans are an existence proof for the core of generality, which suggests our default expectation should be sudden progress when we hit the core.
Smaller update for Eliezer’s view: This isn’t that important—there’s no necessary connection between AGI and e.g. bridges or nukes. But, you can at least see that there’s not a strong consistent track record of continuous improvement once you understand the historical record the right way (plus the underlying assumption in 2 that there will be a lot of effort and investment is probably wrong as well). Nonetheless, if you avoid retrospective trend-fitting and look at progress in the most natural (qualitative?) way, you’ll see that early discoveries that go from 0 to 1 are all over the place—Bitcoin, the Wright flyer, nuclear weapons are at least not crazy exceptions and quite possibly the default.
While Paul and Carl(?) answer,
Smaller update for Paul’s view: The disanalogies between AI progress and Evolution all point in the direction of AI progress being smoother than evolution (we’re intelligently trying to find the capabilities we want) - we get a weak update in favour of the smooth progress view from understanding this disanalogy between AI progress and evolution, but really we don’t learn much, except that there aren’t any good reasons to think there are only a few paths to a large set of powerful world affecting capabilities. Also, the core of generality idea is wrong, so the idea that Humans are an existence proof for it or that evolution tells us something about how to find it is wrong.
Big update for Paul’s view: reasoning from the reference class of ‘technologies where there are many opportunities for improvement and many people trying different things at once’ lets us see why expecting smooth progress should be the default. It’s because as long as there are lots of paths to improvement in the underlying capability landscape (which is the default because that’s how the world works by default), and there are lots of people trying to make improvements in different ways, the incremental changes add up to smooth outputs.
So Eliezer’s claim that Paul et al’s trend-fitting must include
doing something sophisticated but wordless, where they fit a sophisticated but wordless universal model of technological permittivity to bridge lengths, then have a wordless model of cognitive scaling in the back of their minds
is sort of correct, but the model isn’t really sophisticated or wordless.
The model is: as long as the underlying ‘capability landscape’ offers many paths to improvements, not just a few really narrow ones that swamp everything else, lots of people intelligently trying lots of different approaches will lead to lots of small discoveries that add up. Additionally, most examples of tech progress look like ‘multiple ways of doing something that add up’, this is confirmed by the historical record.
And then the model of cognitive scaling consists of (among other things) specific counterarguments to the claim that AGI progress is one of those cases with a few big paths to improvement (e.g. Evolution doesn’t give us evidence that AGI progress will be sudden).
[Shulman]
As I work through sectors and the rollout of past automation I see opportunities for large-scale rollout that is not heavily blocked by regulation...[Long list of examples]
[Yudkowsky]
so… when I imagine trying to deploy this style of thought myself to predict the recent past without benefit of hindsight, it returns a lot of errors. perhaps this is because I do not know how to use this style of thought...
...”There are many possible regulatory regimes in the world, some of which would permit rapid construction of mRNA-vaccine factories well in advance of FDA approval. Given the overall urgency of the pandemic some of those extra-USA vaccines would be sold to individuals or a few countries like Israel willing to pay high prices for them, which would provide evidence of efficacy and break the usual impulse towards regulatory uniformity among developed countries...”
On Carl’s view, it sure seems like you’d just say something like “Healthcare is very overregulated, there will be an unusually strong effort anyway in lots of countries because Covid is an emergency, so it’ll be faster by some hard to predict amount but still bottlenecked by regulatory pressures.” And indeed the fastest countries got there in ~10 months instead of the multiple years predicted by superforecasters, or the ~3 months it would have taken with immediate approval.
The obvious object-level difference between Eliezer ‘applying’ Carl’s view to retrodict covid vaccine rollout and Carl’s prediction about AI is that Carl is saying there’s an enormous number of potential applications of intermediately general AI tech, and many of them aren’t blocked by regulation, while Eliezer’s attempted operating of Carl’s view for covid vaccines is saying “There are many chances for countries with lots of regulatory barriers to do the smart thing”.
The vaccine example is a different argument than AI predictions, because what Carl is saying is that there are many completely open goals for improvement like automating factories and call centres etc. not that there are many opportunities to avoid the regulatory barriers that will block everything by default.
But it seems like Eliezer is making a more outside view appeal, i.e. approach stories where big innovations are used wisely with a lot of scepticism because of our past record, even if you can tell a story about why it will be quite different this time.
If you have good news sources and follows to keep a better eye on the UK or Europe for Covid purposes, or data sources anywhere I may not have noticed, I invite you to share them in the comments.
James Ward is good for factual UK based covid news and especially as an aggregator of other news sources. His new thread on prospects for the Omicron variant is here.
Modeling Failure Modes of High-Level Machine Intelligence
Summary of why I think the post’s estimates are too low as estimates of what’s required for a system capable of seizing a decisive strategic advantage:
To be an APS-like system OmegaStar needs to be able to control robots or model real world stuff and also plan over billions, not hundreds of action steps.
Each of those problems adds on a few extra OOMs that aren’t accounted for in e.g. the setup for Omegastar (which can transfer learn across tens of thousands of games, each requiring thousands of action steps to win in a much less complicated environment than the real world).
You’d need something that can transfer learn across tens of thousands of ‘games’ each requiring billions of action steps, each one of which has way more sensory input to parse than StarCraft per time step.
When you correct Omegastar’s requirements by adding on (1) a factor for number of action steps needed to win a war Vs win a game of StarCraft, (2) a factor for the real world Vs StarCraft’s complexity of sensory input and. When you do this, the total requirement would look more like Ajeya’s reports.
I still get the intuition that OmegaStar would not just be a fancy game player! I find it hard to think about what it would be like—maybe good at gaming quite constrained systems or manipulating people?
Therefore, I think the arguments provide a strong case (unless scaling laws break—which I also think is fairly likely for technical reasons) for ‘something crazy happening by 2030’ but less strong a case for ‘AI takeover by 2030’
Summary of mine and Daniel’s disagreements:
(1) Horizon Length: Daniel thinks we’ll get a long way towards planning over a billion action steps ‘for free’ if we transfer learn over lots of games that take a thousand action steps each—so the first correction factor I gave is a lot smaller than it seems just by comparing the raw complexity of StarCraft Vs fighting a war
(2) No Robots: the complexity in sensory input difference doesn’t matter since the system won’t need to control robots [Or, as I should have also said, build robot-level models of the external world even if you’re not running the actuators yourself] - so the second correction factor isn’t an issue, because-
(3) Lower capability threshold: to take a DSA doesn’t require as many action steps as it seems or as many capabilities as often assumed. You can just do it by taking to people and over a smaller number of action steps than it would take to conquer the world yourself.
To me, it seems like Daniel’s view on horizon length reducing one of the upward corrections (1) is doing less total work than (2) and (3) in terms of shortening the timeline—hence this view looks to me like a case of plausible DSA from narrow AI with specialized abilities. Although point taken that it won’t look that narrow to most people today.
(Re scaling laws—there’s a whole debate I about how scaling laws are just a v crude observable for what’s really going on, so we shouldn’t be confident in extrapolation. This is also all conditional on the underlying assumptions of these forecasting models being correct.)
Updates on this after reflection and discussion (thanks to Rohin):
Human Evolution tells us very little about the ‘cognitive landscape of all minds’ (if that’s even a coherent idea) - it’s simply a loosely analogous individual historical example
Saying Paul’s view is that the cognitive landscape of minds might be simply incoherent isn’t quite right—at the very least you can talk about the distribution over programs implied by the random initialization of a neural network.
I could have just said ‘Paul doesn’t see this strong generality attractor in the cognitive landscape’ but it seems to me that it’s not just a disagreement about the abstraction, but that he trusts claims made on the basis of these sorts of abstractions less than Eliezer.
Also, on Paul’s view, it’s not that evolution is irrelevant as a counterexample. Rather, the specific fact of ‘evolution gave us general intelligence suddenly by evolutionary timescales’ is an unimportant surface fact, and the real truth about evolution is consistent with the continuous view.
No core of generality and extrapolation of quantitative metrics for things we care about and lack of common huge secrets in relevant tech progress reference class
These two initial claims are connected in a way I didn’t make explicit—No core of generality and lack of common secrets in the reference class together imply that there are lots of paths to improving on practical metrics (not just those that give us generality), that we are putting in lots of effort into improving such metrics and that we tend to take the best ones first, so the metric improves continuously, and trend extrapolation will be especially correct.
Core of generality and very common presence of huge secrets in relevant tech progress reference class
The first clause already implies the second clause (since “how to get the core of generality” is itself a huge secret), but Eliezer seems to use non-intelligence related examples of sudden tech progress as evidence that huge secrets are common in tech progress in general, independent of the specific reason to think generality is one such secret.
Nate’s Summary
… Eliezer was saying something like “the fact that humans go around doing something vaguely like weighting outcomes by possibility and also by attractiveness, which they then roughly multiply, is quite sufficient evidence for my purposes, as one who does not pay tribute to the gods of modesty”, while Richard protested something more like “but aren’t you trying to use your concept to carry a whole lot more weight than that amount of evidence supports?”..
And, ofc, at this point, my Eliezer-model is again saying “This is why we should be discussing things concretely! It is quite telling that all the plans we can concretely visualize for saving our skins, are scary-adjacent; and all the non-scary plans, can’t save our skins!”
Nate’s summary brings up two points I more or less ignored in my summary because I wasn’t sure what I thought—one is, just what role do the considerations about expected incompetent response/regulatory barriers/mistakes in choosing alignment strategies play? Are they necessary for a high likelihood of doom, or just peripheral assumptions? Clearly, you have to posit some level of “civilization fails to do the x-risk-minimizing thing” if you want to argue doom, but how extreme are the scenarios Eliezer is imagining where success is likely?
The other is the role that the modesty worldview plays in Eliezer’s objections.
I feel confused/suspect we might have all lost track of what Modesty epistemology is supposed to consist of—I thought it was something like “overuse of the outside view, especially in a social cognition context”.
Which of the following is:
a) probably the product of a Modesty world-view?
b) no good reason to think comes from a Modesty world-view but still bad epistemology?
c) good epistemology?
Not believing theories which don’t make new testable predictions just because they retrodict lots of things in a way that the theories proponents claim is more natural, but that you don’t understand, because that seems generally suspicious
Not believing theories which don’t make new testable predictions just because they retrodict lots of things in the world naturally (in a way you sort of get intuitively), because you don’t trust your own assessments of naturalness that much in the absence of discriminating evidence
Not believing theories which don’t make new testable predictions just because they retrodict lots of things in the world naturally (in a way you sort of get intuitively), because most powerful theories which cause conceptual revolutions also make new testable predictions, so it’s a bad sign if the newly proposed theory doesn’t.
As a general matter, accepting that there are lots of cases of theories which are knowably true independent of any new testable predictions they make because of features of the theory. Things like the implication of general relativity from the equivalence principle, or the second law of thermodynamics from Noether’s theorem, or many-worlds from QM are real, but you’ll only believe you’ve found a case like this if you’re walked through to the conclusion, so you’re sure that the underlying concepts are clear and applicable, or there’s already a scientific consensus behind it.
The above sentences, if taken (as you do) as claims about human moral psychology rather than normative ethics, are compatible with full-on moral realism. I.e. everyone’s moral attitudes are pushed around by status concerns, luckily we ended up in a community that ties status to looking for long-run implications of your beliefs and making sure they’re coherent, and so without having fundamentally different motivations to any other human being we were better able to be motivated by actual moral facts.
I know the OP is trying to say loudly and repeatedly that this isn’t the case because ‘everyone else thought that as well, don’t you know?’ with lots of vivid examples, but if that’s the only argument it seems like modesty epistemology—i.e. “most people who said the thing you said were wrong, and also said that they weren’t like all those other people who were wrong in the past for all these specific reasons, so you should believe you’re wrong too”.
I think a lot of this thread confuses moral psychology with normative ethics—most utilitarians know and understand that they aren’t solely motivated by moral concerns, and are also motivated by lots of other things. They know they don’t morally endorse those motivations in themselves, but don’t do anything about it, and don’t thereby change their moral views.
If Peter Singer goes and buys a coffee, it’s no argument at all to say “aha, by revealed preferences, you must not really think utilitarianism is true, or you’d have given the money away!” That doesn’t show that when he does donate money, he’s unmotivated by moral concerns.
Probably even this ‘pure’ motivation to act morally in cases where empathy isn’t much of an issue is itself made up of e.g. a desire not to be seen believing self-contradictory things, cognitive dissonance, basic empathy and so on. But so what? If the emotional incentives work to motivate people to form more coherent moral views, it’s the reliability of the process of forming the views that matter, not the motivation. I’m sure you could tell a similar story about the motivations that drive mathematicians to check their proofs are valid.
Holden also mentions something a bit like Eliezer’s criticism in his own write-up,
In particular, I think it’s hard to rule out the possibility of ingenuity leading to transformative AI in some far more efficient way than the “brute-force” method contemplated here.
When Holden talks about ‘ingenuity’ methods that seems consistent with Eliezer’s
They’re not going to be taking your default-imagined approach algorithmically faster, they’re going to be taking an algorithmically different approach that eats computing power in a different way than you imagine it being consumed.
I.e. if you wanted to fold this consideration into OpenAI’s estimate you’d have to do it by having a giant incredibly uncertain free-floating variable for ‘speedup factor’ because you’d be nonsensically trying to estimate the ‘speed-up’ to brain processing applied from using some completely non-Deep Learning or non-brainlike algorithm for intelligence. All your uncertainty just gets moved into that one factor, and you’re back where you started.
It’s possible that Eliezer is confident in this objection partly because of his ‘core of generality’ model of intelligence—i.e. he’s implicitly imagining enormous numbers of varied paths to improvement that end up practically in the same place, while ‘stack more layers in a brainlike DL model’ is just one of those paths (and one that probably won’t even work), so he naturally thinks estimating the difficulty of this one path we definitely won’t take (and which probably wouldn’t work even if we did try it) out of the huge numbers of varied paths to generality is useless.
However, if you don’t have this model, then perhaps you can be more confident that what we’re likely to build will look at least somewhat like a compute-limited DL system and that these other paths will have to share some properties of this path. Relatedly, it’s an implication of the model that there’s some imaginable (and not e.g. galaxy sized) model we could build right now that would be an AGI, which I think Eliezer disputes?
https://threadreaderapp.com/thread/1466076761427304453.html
Summary of some actual probabislitic guesses about Omicron’s parameters. People work fast!
There’s extensive discussion of OAS here and it’s clearly something that many immunologists have thought about deeply, yet no mention of effects on natural antibodies—https://www.statnews.com/2021/04/16/next-generation-covid-19-vaccines-are-supposed-to-be-better-some-experts-worry-they-could-be-worse/
Also I asked a similar question and got this response on a previous thread—https://www.lesswrong.com/posts/6apSCHHuWyxK635pE/omicron-variant-post-1-we-re-f-ed-it-s-never-over?commentId=gmzKDuzK3h7GqSgZf
I think it’s worth noting that a fast mutating fast spreading endemic disease with a fatality rate 10x that of the flu, on a modern Earth of 7 billion people, is also unprecedented in the history of epidemiology, just like mass vaccination while a virus is endemic.
In terms of what the evolutionary pressures are, we’d be in uncharted territory regardless of what policies we chose.
At the same time, the fact that we are in uncharted territory means you can’t use basic logic or historical precedent to rule out bad outcomes.
isn’t trying to do anything like “sketch a probability distribution over the dynamics of an AI project that is nearing AGI”. This includes all technical MIRI papers I’m familiar with.
I think this specific scenario sketch is from a mainstream AI safety perspective a case where we’ve already failed—i.e. we’ve invented a useless corrigibility intervention that we confidently but wrongly think is scalable.
And if you try training the AI out of that habit in a domain of lower complexity and intelligence, it is predicted by me that generalizing that trained AI or subsystem to a domain of sufficiently higher complexity and intelligence, but where you could still actually see overt plots, would show you the AI plotting to kill you again.
If people try this repeatedly with other corrigibility training tricks on the level where plots are easily observable, they will eventually find a try that seems to generalize to the more complicated and intelligent validation set, but which kills you on the test set.
Most AI safety researchers just don’t agree with Eliezer that there’s no (likely to be found) corrigibility interventions that won’t suddenly and invisibly fail when you increase intelligence, no matter how well you’ve validated them on low capability regimes and how carefully you try to scale up. This is because they don’t agree with/haven’t heard of Eliezer’s arguments about consequentialism being a super-strong attractor.
So they’d think the ‘die with the most dignity’ interventions would just work, while the ‘die with no dignity’ interventions are risky, and quite reasonably push for the former (since it’s far from clear we’ll take the ‘dignified’ option by default): trying corrigibility interventions at low levels of intelligence, testing the AI on validation sets to see if it plots to kill them, while scaling up.
They might be wrong about this working, but if so, the wrongness isn’t in lacking enough security mindset to see that an AI trying to kill you would just alter its own cognition to cheat its way past the tests. Rather, their mistake is not expecting the corrigibility interventions they presumably trust to suddenly break in a way that means you get no useful safety guarantees from any amount of testing at lower capability levels.
I think it’s a shame Eliezer didn’t pose the ‘validation set’ question first before answering it himself, because I think if you got rid of the difference in underlying assumptions—i.e. asked an alignment researcher “Assume there’s a strong chance your corrigibility intervention won’t work upon scaling up and the AGI might start plotting against you, so you’re going to try these transparency/validation schemes on the AGI to check if it’s safe, how could they go wrong and is this a good idea?” they’d give basically the same answer—i.e. if you try this you’re probably going to die.
You could still reasonably say, “even if the AI safety community thinks it’s not the best use of resources because ensuring knowably stable corrigibility looks a lot easier to us, shouldn’t we still be working on some strongly deception-proof method of verifying if an agent is safe, so we can avoid killing ourselves if plan A fails?”
My answer would be yes.
One of the problems here is that, as well as disagreeing about underlying world models and about the likelihoods of some pre-AGI events, Paul and Eliezer often just make predictions about different things by default. But they do (and must, logically) predict some of the same world events differently.
My very rough model of how their beliefs flow forward is:
Paul
Low initial confidence on truth/coherence of ‘core of generality’
→
Human Evolution tells us very little about the ‘cognitive landscape of all minds’ (if that’s even a coherent idea) - it’s simply a loosely analogous individual historical example. Natural selection wasn’t intelligently aiming for powerful world-affecting capabilities, and so stumbled on them relatively suddenly with humans. Therefore, we learn very little about whether there will/won’t be a spectrum of powerful intermediately general AIs from the historical case of evolution—all we know is that it didn’t happen during evolution, and we’ve got good reasons to think it’s a lot more likely to happen for AI. For other reasons (precedents already exist—MuZero is insect-brained but better at chess or go than a chimp, plus that’s the default with technology we’re heavily investing in), we should expect there will be powerful, intermediately general AIs by default (and our best guess of the timescale should be anchored to the speed of human-driven progress, since that’s where it will start) - No core of generality
Then, from there:
No core of generality and extrapolation of quantitative metrics for things we care about and lack of common huge secrets in relevant tech progress reference class → Qualitative prediction of more common continuous progress on the ‘intelligence’ of narrow AI and prediction of continuous takeoff
Eliezer
High initial confidence on truth/coherence of ‘core of generality’
→
Even though there are some disanalogies between Evolution and AI progress, the exact details of how closely analogous the two situations are don’t matter that much. Rather, we learn a generalizable fact about the overall cognitive landscape from human evolution—that there is a way to reach the core of generality quickly. This doesn’t make it certain that AGI development will go the same way, but it’s fairly strong evidence. The disanalogies between evolution and ML are indeed a slight update in Paul’s direction and suggest that AI could in principle take a smoother route to general intelligence, but we’ve never historically seen this smoother route (and it has to be not just technically ‘smooth’ but sufficiently smooth to give us a full 4-year economic doubling) or these intermediate powerful agents, so this correction is weak compared to the broader knowledge we gain from evolution. In other words, all we know is that there is a fast route to the core of generality but that it’s imaginable that there’s a slow route we’ve not yet seen—Core of generality
Then, from there:
Core of generality and very common presence of huge secrets in relevant tech progress reference class → Qualitative prediction of less common continuous progress on the ‘intelligence’ of narrow AI and prediction of discontinuous takeoff
Eliezer doesn’t have especially divergent views about benchmarks like perplexity because he thinks they’re not informative, but differs from Paul on qualitative predictions of how smoothly various practical capabilities/signs of ‘intelligence’ will emerge—he’s getting his qualitative predictions about this ultimately from interrogating his ‘cognitive landscape’ abstraction, while Paul is getting his from trend extrapolation on measures of practical capabilities and then translating those to qualitative predictions. These are very different origins, but they do eventually give different predictions about the likelihood of the same real-world events.
Since they only reach the point of discussing the same things at a very vague, qualitative level of detail, in order to get to a bet you have to back-track from both of their qualitative predictions of how likely the sudden emergence of various types of narrow intelligent behaviour are, find some clear metric for the narrow intelligent behaviour that we can apply fairly, and then there should be a difference in beliefs about the world before AI takeoff.
Israel second. The UK did first doses first and otherwise took its own path to vaccine distribution, some would say even exiting the EU for related reasons. Israel did what it had to do to get more vaccine doses faster, and give them out quickly.
Those two being the first two to ban travel does not seem remotely like a coincidence.
You could add that the UK ran essentially all the big clinical trials that discovered useful treatments, aside from those personally funded by Tyler Cowen. There’s an interesting and important discussion to be had on this topic at some future point—especially as to how these decision-making procedures got off the ground in the first place. I wrote a short summary of that here.
The large decline in Crypto is especially interesting, because we saw a similar thing in 2020 that at the time clearly didn’t reflect long term prospects and was based on the flows that happen in such situations. At the time I was too distracted to pull the trigger, which is quite sad. If the current move gets much bigger, then to the extent that one wants to be long, it seems like a potential opportunity to buy cheap (Not Investment Advice!).
Further in the realm of Not Investment Advice, I considered (but didn’t) shorting global markets when the Omicron news came out, and in the process of dithering the decline had already started. If some of the extreme high estimates (500%?!) in this thread are true then the decline we’ve seen so far looks like an under-reaction, but it’s not clear to me that there’s enough info to make a safe bet on that. A lot of trading ideas got floated in discussions about previous variants, but didn’t result in any firm consensus that I can recall (other than the short-selling that happened for original covid)
How does Original Antigenic Sin work for natural immunity vs vaccine derived immunity? Is it a stronger impediment for one vs the other?
Also, this whole topic seems (I think) to be mostly independent of the T-cell immunity that gives you the baseline immunity to severe disease—the reason for Zvi’s low estimate of full immune escape, I think.
Great and extremely valuable discussion! There’s one part that I really wished had been explored further—the fundamental difficulty of inner alignment:
Joe Carlsmith: I do have some probability that the alignment ends up being pretty easy. For example, I have some probability on hypotheses of the form “maybe they just do what you train them to do,” and “maybe if you just don’t train them to kill you, they won’t kill you.” E.g., in these worlds, non-myopic consequentialist inner misalignment doesn’t tend to crop up by default, and it’s not that hard to find training objectives that disincentivize problematically power-seeking forms of planning/cognition in practice, even if they’re imperfect proxies for human values in other ways.
...
Nate: …maybe it wouldn’t have been that hard for natural selection to train humans to be fitness maximizers, if it had been watching for goal-divergence and constructing clever training environments?
Joe Carlsmith: I think something like this is in the mix for me. That is, I don’t see the evolution example as especially strong evidence for how hard inner alignment is conditional on actually and intelligently trying to avoid inner misalignment (especially in its scariest forms).
I would very much like to see expansion (from either Nate/MIRI or Joe) on these points because they seem crucial to me. My current epistemic situation is (I think) similar to Joe’s. Different views about the fundamental difficulty of inner alignment seem to be a (the?) major driver of differences in views about how likely AI X risk is overall. I see lots of worrisome signs from indirect lines of evidence—some based on intuitions about the nature of intelligence, some from toy models and some from vague analogies to e.g. evolution. But what I don’t see is a slam dunk argument that inner misalignment is an extremely strong attractor for powerful models of the sort we’re actually going to build.
That also goes for many of the specific reasons given for inner misalignment—they often just seem to push the intuition one step further back. E.g. these from Eliezer Yudkowsky’s recent interview:
I predict that deep algorithms within the AGI will go through consequentialist dances, and model humans, and output human-manipulating actions that can’t be detected as manipulative by the humans, in a way that seems likely to bypass whatever earlier patch was imbued by gradient descent, because I doubt that earlier patch will generalize as well as the deep algorithms.
...
attempts to teach corrigibility in safe regimes are unlikely to generalize well to higher levels of intelligence and unsafe regimes (qualitatively new thought processes, things being way out of training distribution, and, the hardest part to explain, corrigibility being “anti-natural” in a certain sense that makes it incredibly hard to, eg, exhibit any coherent planning behavior (“consistent utility function”) which corresponds to being willing to let somebody else shut you off, without incentivizing you to actively manipulate them to shut you off).
seem like world models that make sense to me, given the surrounding justifications, and I wouldn’t be amazed if they were true, and I also place a decent amount of credence on them being true. But I can’t pass an ideological Turing test for someone who believes the above propositions with > 95% certainty, given the massive conceptual confusion involved with all of these concepts and the massive empirical uncertainty.
Statements like ‘corrigibility is anti-natural in a way that can’t easily be explained’ and ‘getting deep enough patches that generalize isn’t just difficult but almost impossibly difficult’ when applied to systems we don’t yet know how to build at all, don’t seem like statements about which confident beliefs either way can be formed. (Unless there’s really solid evidence out there that I’m not seeing)
This conversation seemed like another such opportunity to provide that slam-dunk justification for the extreme difficulty of inner alignment, but as in many previous cases Nate and Joe seemed happy to agree to disagree and accept that this is a hard question about which it’s difficult to reach any clear conclusion—which if true should preclude strong confidence in disaster scenarios.
(FWIW, I think there’s a good chance that until we start building systems that are already quite transformative, we’re probably going to be stuck with a lot of uncertainty about the fundamental difficulty of inner alignment—which from a future planning perspective is worse than knowing for sure how hard the problem is.)
And I think they are well enough motivated to stop their imminent annihilation, in a way that is more like avoiding mutual nuclear destruction than cosmopolitan altruistic optimal climate mitigation timing.
In my recent writeup of an investigation into AI Takeover scenarios I made an identical comparison—i.e. that the optimistic analogy looks like avoiding nuclear MAD for a while and the pessimistic analogy looks like optimal climate mitigation:
It is unrealistic to expect TAI to be deployed if first there are many worsening warning shots involving dangerous AI systems. This would be comparable to an unrealistic alternate history where nuclear weapons were immediately used by the US and Soviet Union as soon as they were developed and in every war where they might have offered a temporary advantage, resulting in nuclear annihilation in the 1950s.
Note that this is not the same as an alternate history where nuclear near-misses escalated (e.g. Petrov, Vasili Arkhipov), but instead an outcome where nuclear weapons were used as ordinary weapons of war with no regard for the larger dangers that presented—there would be no concept of ‘near misses’ because MAD wouldn’t have developed as a doctrine. In a previous post I argued, following Anders Sandberg, that paradoxically the large number of nuclear ‘near misses’ implies that there is a forceful pressure away from the worst outcomes.
https://www.ft.com/content/d4e58d38-37d6-40cd-9d72-6b9bfd0a3683
Very Good news on Boosters: first RCT of a Pfizer booster in Israel confirms 95.6% efficacy Vs infection!
https://mobile.twitter.com/DevanSinha/status/1451147345664618496
That basically takes us right back to where we started in efficacy terms
"The trial took place during a period when the Delta coronavirus variant was prevalent, and the median time between second and third doses was about 11 months, with a median follow-up time of two-and-a-half months."Plus there’s reason to think the immunity will also wane slower,
https://twitter.com/JamesWard73/status/1451147065925554183?t=bjBh1wqz3moJP8cD4tmsXg&s=08
Combined with the long covid news from last week the picture looks a lot better… Assuming they let us get boosters
To be honest, I was expecting to get pushback from libertarian-leaning types who were opposed to Orwell’s socialism, or leftwing types opposed to Churchill—he’s become controversial recently and this review was partly a defense of the key thing that I think is valuable about him. Or else pushback against my claim that you can trace EA and longtermist ideas that far back—but maybe this audience just agrees with me on all of these points!
I found the extended Fire/Nuclear Weapons analogy to be quite helpful. Here’s how I think it goes:
In 1870 the gradualist and the catastrophist physicist wonder about whether there will ever be a discontinuity in explosive power
Gradualist: we’ve already had our zero-to-one discontinuity—we’ve invented black powder, dynamite and fuses, from now on there’ll be incremental changes and inventions that increase explosive power but probably not anything qualitatively new, because that’s our default expectation with a technology like explosives where there are lots of paths to improvement and lots of effort exerted
Catastrophist: that’s all fine and good, but those priors don’t mean anything if we have already seen an existence proof for qualitatively new energy sources. What about the sun? The energy the sun outputs is overwhelming, enough to warm the entire earth. One day, we’ll discover how to release those energies ourselves, and that will give us qualitatively better explosives.
Gradualist: But we don’t know anything about how the sun works! It’s probably just be a giant ball of gas heated by gravitational collapse! One day, in some crazy distant future, we might be able to pile on enough gas that gravity implodes and heats it, but that’ll require us to be able to literally build stars, it’s not going to occur suddenly. We’ll pile up a small amount of gas, then a larger amount, and so on after we’ve given up on assembling bigger and bigger piles of explosives. There’s no secret physics there, just a lot of conventional gravitational and chemical energy in one place
Catastrophist: ah, but don’t you know Lord Kelvin calculated the Sun could only shine for a few million years under the gravitational mechanism, and we know the Earth is far older than that? So there has to be some other, incredibly powerful energy source that we’ve not yet discovered within the sun. And when we do discover it, we know it can under the right circumstances, release enough energy to power the Sun, so it seems foolhardy to assume it’ll just happen to be as powerful as our best normal explosive technologies are whenever we make the discovery. Imagine the coincidence if that was true! So I can’t say when this will happen or even exactly how powerful it’ll be, but when we discover the Sun’s power it will probably represent a qualitatively more powerful new energy source. Even if there are many ways to try to tweak our best chemical explosives to be more powerful and/or the potential new sun-power explosives to be weaker, and we’d still not hit the narrow target of the two being roughly on the same level.
Gradualist: Your logic works, but I doubt Lord Kelvin’s calculation
It seems like the AGI Gradualist sees the example of humans like my imagined Nukes Gradualist sees the sun, i.e. just a scale up of what we have now. While the AGI Catastrophist sees Humans as my imagined Nukes Catastrophist sees the sun.
The key disanalogy is that for the Sun case, there’s a very clear ‘impossibility proof’ given by the Nukes Catastrophist that the sun couldn’t just be a scale up of existing chemical and gravitational energy sources.