Sammy Martin. Philosophy and Physics BSc, AI MSc at Edinburgh, starting a PhD at King’s College London. Interested in ethics, general philosophy and AI Safety.
SDM
Karma: 844
Modern literature about immortality is written primarily by authors who expect to die, and their grapes are accordingly sour.
This is still just as true as when this essay was written, I think—even the Culture had its human citizens mostly choosing to die after a time… to the extent that I eventually decided: if you want something done properly, do it yourself.
But there are exceptions—the best example of published popular fiction that has immortality as a basic fact of life is the Commonwealth Saga by Peter F Hamilton and the later Void Trilogy (the first couple of books were out in 2007).
The Commonwealth has effective immortality, a few downsides of it are even noticable (their culture and politics is a bit more stagnant than we might like), but there’s never any doubt at all that it’s worth it, and it’s barely commented on in the story,
In truth, I suspect that if people were immortal, they would not think overmuch about the meaning that immortality gives to life.
(Incidentally, the latter-day Void Trilogy Commonwealth is probably the closest a work of published fiction has come to depicting a true eudaimonic utopia that lacks the problems of the culture)
I wonder if there’s been any harder to detect shift in how immortality is portrayed in fiction since 2007? Is it still as rare now as then to depict it as a bad thing?
The UK vaccine rollout is considered a success, and by the standards of other results, it is indeed a success. This interview explains how they did it, which was essentially ‘make deals with companies and pay them money in exchange for doses of vaccines.’
A piece of this story you may find interesting (as an example of a government minister making a decision based on object level physical considerations): multiple reports say Matt Hancock, the UK’s health Secretary, made the decision to insist on over-ordering vaccines because he saw the movie Contagion and was shocked into viscerally realising how important a speedy rollout was.
It might just be a nice piece of PR, but even if that’s the case it’s still a good metaphor for how object level physical considerations can intrude into government decision making
I agree with your argument about likelihood of DSA being higher compared to previous accelerations, due to society not being able to speed up as fast as the technology. This is sorta what I had in mind with my original argument for DSA; I was thinking that leaks/spying/etc. would not speed up nearly as fast as the relevant AI tech speeds up.
Your post on ‘against GDP as a metric’ argues more forcefully for the same thing that I was arguing for, that
‘the economic doubling time’ stops being so meaningful—technological progress speeds up abruptly but other kinds of progress that adapt to tech progress have more of a lag before the increased technological progress also affects them?
So we’re on the same page there that it’s not likely that ‘the economic doubling time’ captures everything that’s going on all that well, which leads to another problem—how do we predict what level of capability is necessary for a transformative AI to obtain a DSA (or reach the PONR for a DSA)?
I notice that in your post you don’t propose an alternative metric to GDP, which is fair enough since most of your arguments seem to lead to the conclusion that it’s almost impossibly difficult to predict in advance what level of advantage over the rest of the world in which areas are actually needed to conquer the world, since we seem to be able to analogize persuasion tools to or conquistador-analogues who had relatively small tech advantages, to the AGI situation.
I think that there is still a useful role for raw economic power measurements, in that they provide a sort of upper bound on how much capability difference is needed to conquer the world. If an AGI acquires resources equivalent to controlling >50% of the world’s entire GDP, it can probably take over the world if it goes for the maximally brute force approach of just using direct military force. Presumably the PONR for that situation would be awhile before then, but at least we know that an advantage of a certain size would be big enough given no assumptions about the effectiveness of unproven technologies of persuasion or manipulation or specific vulnerabilities in human civilization.
So we can use our estimate of how doubling time may increase, anchor on that gap and estimate down based on how soon we think the PONR is, or how many ‘cheat’ pathways that don’t involve economic growth there are.
The whole idea of using brute economic advantage as an upper limit ‘anchor’ I got from Ajeya’s Post about using biological anchors to forecast what’s required for TAI—if we could find a reasonable lower bound for the amount of advantage needed to attain DSA we could do the same kind of estimated distribution between them. We would just need a lower limit—maybe there’s a way of estimating it based on the upper limit of human ability since we know no actually existing human has used persuasion to take over the world but as you point out they’ve come relatively close.
I realize that’s not a great method, but is there any better alternative given that this is a situation we’ve never encountered before, for trying to predict what level of capability is necessary for DSA? Or perhaps you just think that anchoring your prior estimate based on economic power advantage as an upper bound is so misleading it’s worse than having a completely ignorant prior. In that case, we might have to say that there are just so many unprecedented ways that a transformative AI could obtain a DSA that we can just have no idea in advance what capability is needed, which doesn’t feel quite right to me.
Finally got round to reading your sequence and it looks like we disagree a lot less than I thought, since your first three causes are exactly what I was arguing for in my reply,
This is probably the crux. I don’t think we tend to go to higher simulacra levels now, compared to decades ago. I think it’s always been quite prevalent, and has been roughly constant through history. While signalling explanations definitely tell us a lot about particular failings, they can’t explain the reason things are worse now in certain ways, compared to before. The difference isn’t because of the perennial problem of pervasive signalling. It has more to do with economic stagnation and not enough state capacity. These flaws mean useful action gets replaced by useless action, and allow more room for wasteful signalling.
As one point in favour of this model, I think it’s worth noting that the historical comparisons aren’t ever to us actually succeeding at dealing with pandemics in the past, but to things like “WWII-style” efforts—i.e. thinking that if we could just do x as well as we once did y then things would have been a lot better.
This implies that if you made an institution analogous to e.g. the weapons researchers of WW2 and the governments that funded them, or NASA in the 1960s, without copy-pasting 1940s/1960s society wholesale, the outcome would have been better. To me that suggests it’s institution design that’s the culprit, not this more ethereal value drift or increase in overall simulacra levels.
I think you’d agree with most of that, except that you see a much more significant causal role for the cultural factors like increased fragility and social atomisation. There is pretty solid evidence for both being real problems, Jon Haidt presents the best case to take these seriously, although it’s not as definitive as you make out (E.g. Suicide rates are basically a random walk), and your explanation for how they lead to institutional problems is reasonable, but I wonder if they are even needed as explanations when your first three causes are so strong and obvious,
Essentially I see your big list like this:
Main Drivers:
Cause 1: More Real Need For Large Organizations (includes decreasing low hanging fruit) Cause 2: Laws and Regulations Favor Large Organizations Cause 3: Less Disruption of Existing Organizations Cause 5: Rent Seeking is More Widespread and Seen as Legitimate
Real but more minor:
Cause 4: Increased Demand for Illusion of Safety and Security Cause 8: Atomization and the Delegitimization of Human Social Needs Cause 7: Ignorance Cause 9: Educational System Cause 10: Vicious Cycle
No idea but should look into:
Cause 6: Big Data, Machine Learning and Internet Economics
Essentially my view is that if you directly addressed the main drivers with large legal or institutional changes the other causes of mazedom wouldn’t fight back.
I believe that the ‘obvious legible institutional risks first’ view is in line with what others who’ve written on this problem like Tyler Cowen or Sam Bowman think, but it’s a fairly minor disagreement since most of your proposed fixes are on the institutional side of things anyway.
Also, the preface is very important—these are some of the only trends that seem to be going the wrong way consistently in developed countries for a while now, and they’re exactly the forces you’d expect to be hardest to resist.
The world is better for people than it was back then. There are many things that have improved. This is not one of them.
Currently the most plausible doom scenario in my mind is maybe a version of Paul’s Type II failure. (If this is surprising to you, reread it while asking yourself what terms like “correlated automation failure” are euphemisms for.)
This is interesting, and I’d like to see you expand on this. Incidentally I agree with the statement, but I can imagine both more and less explosive, catastrophic versions of ‘correlated automation failure’. On the one hand it makes me think of things like transportation and electricity going haywire, on the other it could fit a scenario where a collection of powerful AI systems simultaneously intentionally wipe out humanity.
Clock-time leads shrink automatically as the pace of innovation speeds up, because if everyone is innovating 10x faster, then you need 10x as many hoarded ideas to have an N-year lead.
What if, as a general fact, some kinds of progress (the technological kinds more closely correlated with AI) are just much more susceptible to speed-up? I.e, what if ‘the economic doubling time’ stops being so meaningful—technological progress speeds up abruptly but other kinds of progress that adapt to tech progress have more of a lag before the increased technological progress also affects them? In that case, if the parts of overall progress that affect the likelihood of leaks, theft and spying aren’t sped up by as much as the rate of actual technology progress, the likelihood of DSA could rise to be quite high compared to previous accelerations where the order of magnitude where the speed-up occurred was fast enough to allow society to ‘speed up’ the same way.
In other words—it becomes easier to hoard more and more ideas if the ability to hoard ideas is roughly constant but the pace of progress increases. Since a lot of these ‘technologies’ for facilitating leaks and spying are more in the social realm, this seems plausible.
But if you need to generate more ideas, this might just mean that if you have a very large initial lead, you can turn it into a DSA, which you still seem to agree with:
Even if takeoff takes several years it could be unevenly distributed such that (for example) 30% of the strategically relevant research progress happens in a single corporation. I think 30% of the strategically relevant research happening in a single corporation at beginning of a multi-year takeoff would probably be enough for DSA.
I meant, ‘based on what you’ve said about Zvi’s model’ I.e. Nostalgebraist says zvi says Rt never goes below 1 - if you look at the plot he produced Rt is always above 1 given Zvi’s assumptions, which the London data falsified.
It seems better to first propose a model we know can match past data, and then add a tuning term/effect for “pandemic fatigue” for future prediction.
To get a sense of scale, here is one of the plots from my notebook:
The colored points show historical data on R vs. the 6-period average, with color indicating the date.
Thanks for actually plotting historical Rt vs infection rates!
Whereas, it seems more natural to take (3) as evidence that (1) was wrong.
In my own comment, I also identified the control system model of any kind of proportionality of Rt to infections as a problem. Based on my own observations of behaviour and government response, the MNM hypothesis seems more likely (governments hitting the panic button as imminent death approaches, i.e. hospitals begin to be overwhelmed) than a response that ramps up proportionate to recent infections. I think that explains the tight oscillations.
I’d say the dominant contributor to control systems is something like a step function at a particular level near where hospitals are overwhelmed, and individual responses proportionate to exact levels of infection are a lesser part of it.
You could maybe operationalize this by looking at past hospitalization rates, fitting a logistic curve to them at the ‘overwhelmed’ threshold and seeing if that predicts Rt. I think it would do pretty well.
This tight control was a surprise and is hard to reproduce in a model, but if our model doesn’t reproduce it, we will go on being surprised by the same thing that surprised us before.
My own predictions are essentially based on continuing to expect the ‘tight control’ to continue somehow, i.e. flattening out cases or declining a bit at a very high level after a large swing upwards.
It looks like (subsequent couple of days data seem to confirm this), Rt is currently just below 1 in London—which would outright falsify any model that claims Rt never goes below 1 for any amount of infection with the new variant, given our control system response, which according to your graph, the infections exponential model does predict.
If you ran this model on the past, what would it predict? Based on what you’ve said, Rt never goes below one, so there would be a huge first wave with a rapid rise up to partial herd immunity over weeks, based on your diagram. That’s the exact same predictive error that was made last year.
I note—outside view—that this is very similar to the predictive mistake made last Febuary/March with old Covid-19 - many around here were practically certain we were bound for an immediate (in a month or two) enormous herd immunity overshoot.
Humans have skills and motivations (such as deception, manipulation and power-hungriness) which would be dangerous in AGIs. It seems plausible that the development of many of these traits was driven by competition with other humans, and that AGIs trained to answer questions or do other limited-scope tasks would be safer and less goal-directed. I briefly make this argument here.
Note that he claims that this may be true even if single/single alignment is solved, and all AGIs involved are aligned to their respective users.
It strikes me as interesting that much of the existing work that’s been done on multiagent training, such as it is, focusses on just examining the behaviour of artificial agents in social dilemmas. The thinking seems to be—and this was also suggested in ARCHES—that it’s useful just for exploratory purposes to try to characterise how and whether RL agents cooperate in social dilemmas, what mechanism designs and what agent designs promote what types of cooperation, and if there are any general trends in terms of what kinds of multiagent failures RL tends to fall into.
For example, it’s generally known that regular RL tends to fail to cooperate in social dilemmas, ‘Unfortunately, selfish MARL agents typically fail when faced with social dilemmas’. From ARCHES:
One approach to this research area is to continually ex-amine social dilemmas through the lens of whatever is the leading AI devel-opment paradigm in a given year or decade, and attempt to classify interest-ing behaviors as they emerge. This approach might be viewed as analogous to developing “transparency for multi-agent systems”: first develop inter-esting multi-agent systems, and then try to understand them.
There seems to be an implicit assumption here that something very important and unique to multiagent situations would be uncovered—by analogy to things like the flash crash. It’s not clear to me that we’ve examined the intersection of RL and social dilemmas enough to notice if this were true, if it were true, and I think that’s the major justification for working on this area.
One thing that you didn’t account for—the method of directly scaling the Rt by the multiple on the R0 (which seems to be around 1.55), is only a rough estimate of how much the Rt will increase by when the effective Rt is lowered in a particular situation. It could be almost arbitrarily wrong—intuitively, if the hairdressers are closed, that prevents 100% of transmission in hairdressers no matter how much higher the R0 of the virus is.
For this reason, the actual epidemiological models (there aren’t any for the US for the new variant, only some for the UK), have some more complicated way of predicting the effect of control measures. This from Imperial College:
We quantified the transmission advantage of the VOC relative to non-VOC lineages in twoways: as an additive increase in R that ranged between 0.4 and 0.7, and alternatively as amultiplicative increase in R that ranged between a 50% and 75% advantage. We were not ableto distinguish between these two approaches in goodness-of-fit, and either is plausiblemechanistically. A multiplicative transmission advantage would be expected if transmissibilityhad increased in all settings and individuals, while an additive advantage might reflect increasesin transmissibility in specific subpopulations or contexts.
The multiplicative ‘increased transmissibility’ estimate will therefore tend to underestimate the effect of control measures. The actual paper did some complicated Bayesian regression to try and figure out which model of Rt change worked best, and couldn’t figure it out.
Measures like ventilation, physical distancing when you do decide to meet up, and mask use will be more multiplicative in how the new variant diminishes their effect. The parts of the behaviour response that involve people just not deciding to meet up or do things in the first place, and anything involving mandatory closures of schools, bars etc. will be less multiplicative.
I believe this is borne out in the early data. Lockdown 1 in the UK took Rt down to 0.6. The naive ‘multiplicative’ estimate would say that’s sufficient for the new variant, Rt=0.93. The second lockdown took Rt down to 0.8, which would be totally insufficient. You’d need Rt for the old variant of covid down to 0.64 on the naive multiplicative estimate—almost what was achieved in March. I have a hard time believing it was anywhere near that low in the Tier 4 regions around Christmas.
But the data that’s come in so far seems to indicate that Tier 4 + Schools closed has either levelled off or caused slow declines in infections in those regions where they were applied.
First, the random infection survey—London and South East are in decline and East of England has levelled off (page 3). The UKs symptom study, which uses a totally different methodology, confirms some levelling off and declines in those regions—page 6. It’s early days, but clearly Rt is very near 1, and likely below 1 in London. The Financial Times cottoned on to this a few days late but no-one else seems to have noticed.
I think this indicates a bunch of things—mainly that infections caused by the new variant can and will be stabilized or even reduced by lockdown measures which people are willing to obey. It’s not impossible if it’s already happening.
To start, let’s also ignore phase shifts like overloading hospitals, and ignore fatigue on the hopes that vaccines coming soon will cancel it out, although there’s an argument that in practice some people do the opposite.
I agree with ignoring fatigue, but ignoring phase shifts? If it were me I’d model the entire control system response as a phase shift with the level for the switch in reactions set near the hospital overwhelm level—at least on the policy side, there seems to be an abrupt reaction specifically to the hospital overloading question. The British government pushed the panic button a few days ago in response to that and called a full national lockdown. I’d say the dominant contributor to control systems is something like a step function at a particular level near where hospitals are overwhelmed, and individual responses proportionate to exact levels of infection are a lesser part of it.
I think the model of the control system as a continuous response is wrong, and a phased all-or-nothing response for the government side of things, plus taking into account non-multiplicative effects on the Rt, would produce overall very different results—namely that a colossal overshoot of herd immunity in a mere few weeks is probably not happening. I note—outside view—that this is very similar to the predictive mistake made last Febuary/March with old Covid-19 - many around here were practically certain we were bound for an immediate (in a month or two) enormous herd immunity overshoot.
Many of the same thoughts were in my mind when I linked when I linked that study on the previous post.
----
IMO, it would help clarify arguments about the “control system” a lot to write down the ideas in some quantitative form.
...
This tells you nothing about the maximum power of my heating system. In colder temperatures, it’d need to work harder, and at some low enough temperature T, it wouldn’t be able to sustain 70F inside. But we can’t tell what that cutoff T is until we reach it. “The indoor temperature right now oscillates around 70F” doesn’t tell you anything about T.
I agree, and in fact the main point I was getting at with my initial comment is that in the two areas I talked about—namely the control system and the overall explanation for failure, there’s an unfortunate tendency to toss out quantitative arguments or even detailed models of the world and instead resort to intuitions and qualitative arguments—and then it has a tendency to turn into a referendum on your personal opinions about human nature and the human condition, which isn’t that useful for predicting anything. You can see this in how the predictions panned out—as was pointed out by some anonymous commenter, control system ‘running out of power’ arguments generally haven’t been that predictively accurate when it comes to these questions.
The rule-of-thumb that I’ve used—the Morituri Nolumus Mori effect—has fared somewhat better than the ‘control system will run out of steam sooner or later’ rule-of-thumb, both when I wrote that post and since. The MNM tends to predict last-minute myopic decisions that mostly avoid the worst outcomes, while the ‘out of steam’ explanation led people to predict that social distancing would mostly be over by now. But neither is a proper quantitative model.
In terms of actually giving some quantitative rigour to this question—it’s not easy. I made an effort in my old post, by saying how far a society can stray from a control system equilibrium is indicated by how low they managed to get Rt—but the ‘gold standard’ is to just work off model projections trained on already existing data like I tried to do.
As to the second question—overall explanation, there is some data to work off of, but not much. We know that preexisting measures of state capacity don’t predict covid response effectiveness, which along with other evidence suggests the ‘institutional schlerosis’ hypothesis I referred to in my original post. Once again, I think that a clear mechanism - ‘institutional sclerosis as part of the great stagnation’ - is a much better starting point for unravelling all this than the ‘simulacra levels are higher now’ perspective that I see a lot around here. That claim is too abstract to easily falsify or derive genuine in-advance predictions.
I live in Southern England and so have a fair bit of personal investment in all this, but I’ll try to be objective. My first reaction, upon reading the LSHTM paper that you referred to, is ‘we can no longer win, but we can lose less’ - i.e. we are all headed for herd immunity one way or another by mid-year, but we can still do a lot to protect people. That would have been my headline—it’s over for suppression and elimination, but ‘it’s over’ isn’t quite right. Your initial reaction was different:
Are We F***ed? Is it Over?
Yeah, probably. Sure looks like it.
The twin central points last were that we were probably facing a much more infectious strain (70%), and that if we are fucked in this way, then it is effectively already over in the sense that our prevention efforts would be in vain.
The baseline scenario remains, in my mind, that the variant takes over some time in early spring, the control system kicks in as a function of hospitalizations and deaths so with several weeks lag, and likely it runs out of power before it stabilizes things at all, and we blow past herd immunity relatively quickly combining that with our vaccination efforts.
You give multiple reasons to expect this, all of which make complete sense—Lockdown fatigue, the inefficiency of prevention, lags in control systems, control systems can’t compensate etc. I could give similar reasons to expect the alternative—mainly that the MNM predicts the extreme strength of control systems and that it looks like many places in Europe/Australia did take Rt down to 0.6 or even below!
But luckily, none of that is necessary.
This preprint model via the LessWrong thread has a confidence interval for increased infectiousness of 50%-74%.
I would encourage everyone to look at the scenarios in this paper since they neatly explain exactly what we’re facing and mean we don’t have to rely on guestimate models and inference about behaviour changes. This model is likely highly robust—it successfully predicted the course of the UK’s previous lockdown, with whatever compliance we had then. They simply updated it by putting in the increased infectiousness of the new variant. Since that last lockdown was very recent, compliance isn’t going to be wildly different, weather was cold during the previous lockdown, schools were open etc. The estimate for the increase in R given in this paper seems to be the same as that given by other groups e.g. Imperial College.
So what does the paper imply? Essentially a Level 4 lockdown (median estimate) flattens out case growth but with schools closed a L4 lockdown causes cases to decline a bit (page 10). 10x-ing the vaccination rate from 200,000 to 2 million reduces the overall numbers of deaths by more than half (page 11). And they only model a one-month lockdown, but that still makes a significant difference to overall deaths (page 11). We managed 500k vaccinations the first week, and it dropped a bit the second week, but with first-doses first and the Oxford/AZ vaccine it should increase again and land somewhere between those two scenarios. Who knows where? For the US, the fundamental situation may look like the first model—no lockdowns at all, so have a look.
(Also of note is that the peak demand on the medical system even in the bad scenarios with a level 4 lockdown and schools open is less than 1.5x what was seen during the first peak. That’s certainly enough to boost the IFR and could be described as ‘healthcare system collapse’, since it means surge capacity being used, healthcare workers being wildly overstretched, but to my mind ‘collapse’ refers to demand that exceeds supply by many multiples such that most people can’t get any proper care at all—as was talked about in late feb/early march.)
(Edit: the level of accuracy of the LSHTM model should become clear in a week or two)
The nature of our situation now is such that every day of delay and every extra vaccinated person makes us incrementally better off.
This is a simpler situation than before—before we had the option of suppression, which is all-or-nothing—either you get R under 1 or you don’t. The race condition that we’re in now, where short lockdowns that temporarily hold off the virus buy us useful time, and speeding up vaccination increases herd immunity and decreases deaths and slackens the burden on the medical system, is a straightforward fight by comparison. You just do whatever you can to beat it back and vaccinate as fast as you can.
Now, I don’t think you really disagree with me here, except about some minor factual details (I reckon your pre-existing intuitions about what ‘Level 4 lockdown’ would be capable of doing are different to mine), and you mention the extreme urgency of speeding up vaccine rollout often,
We also have a vaccination crisis. WIth the new strain coming, getting as many people vaccinated as fast as possible becomes that much more important.
...
With the more reasonable version of this being “we really really really should do everything to speed up our vaccinations, everyone, and to focus them on those most likely to die of Covid-19.” That’s certainly part of the correct answer, and likely the most important one for us as a group.
But if I were writing this, my loud headline message would not have been ‘It’s over’, because none of this is over, many decisions still matter. It’s only ‘over’ for the possibility of long term suppression.
*****
There’s also the much broader point—the ‘what, precisely, is wrong with us’ question. This is very interesting and complex and deserves a long discussion of its own. I might write one at some point. I’m just giving some initial thoughts here, partly a very delayed response to your reply to me 2 weeks ago (https://www.lesswrong.com/posts/Rvzdi8RS9Bda5aLt2/covid-12-17-the-first-dose?commentId=QvYbhxS2DL4GDB6hF). I think we have a hard-to-place disagreement about some of the ultimate causes of our coronavirus failures.
We got a shout-out in Shtetl-Optimized, as he offers his “crackpot theory” that if we were a functional civilization we might have acted like one and vaccinated everyone a while ago
...
I think almost everyone on earth could have, and should have, already been vaccinated by now. I think a faster, “WWII-style” approach would’ve saved millions of lives, prevented economic destruction, and carried negligible risks compared to its benefits. I think this will be clear to future generations, who’ll write PhD theses exploring how it was possible that we invented multiple effective covid vaccines in mere days or weeks
He’s totally right on the facts, of course. The question is what to blame. I think our disagreement here, as revealed in our last discussion, is interesting. The first order answer is institutional sclerosis, inability to properly do expected value reasoning and respond rapidly to new evidence. We all agree on that and all see the problem. You said to me,
And I agree that if government is determined to prevent useful private action (e.g. “We have 2020 values”)...
Implying, as you’ve said elsewhere, that the malaise has a deeper source. When I said “2020 values” I referred to our overall greater valuation of human life, while you took it to refer to our tendency to interfere with private action—something you clearly think is deeply connected to the values we (individuals and governments) hold today.
I see a long term shift towards a greater valuation of life that has been mostly positive, and some other cause producing a terrible outcome from coronavirus in western countries, and you see a value shift towards higher S levels that has caused the bad outcomes from coronavirus and other bad things.
Unlike Robin Hanson, though, you aren’t recommending we attempt to tell people to go off and have different values—you’re simply noting that you think our tendency to make larger sacrifices is a mistake.
″...even when the trade-offs are similar, which ties into my view that simulacra and maze levels are higher, with a larger role played by fear of motive ambiguity.”
This is probably the crux. I don’t think we tend to go to higher simulacra levels now, compared to decades ago. I think it’s always been quite prevalent, and has been roughly constant through history. While signalling explanations definitely tell us a lot about particular failings, they can’t explain the reason things are worse now in certain ways, compared to before. The difference isn’t because of the perennial problem of pervasive signalling. It has more to do with economic stagnation and not enough state capacity. These flaws mean useful action gets replaced by useless action, and allow more room for wasteful signalling.
As one point in favour of this model, I think it’s worth noting that the historical comparisons aren’t ever to us actually succeeding at dealing with pandemics in the past, but to things like “WWII-style” efforts—i.e. thinking that if we could just do x as well as we once did y then things would have been a lot better.
This implies that if you made an institution analogous to e.g. the weapons researchers of WW2 and the governments that funded them, or NASA in the 1960s, without copy-pasting 1940s/1960s society wholesale, the outcome would have been better. To me that suggests it’s institution design that’s the culprit, not this more ethereal value drift or increase in overall simulacra levels. There are other independent reasons to think the value shift has been mostly good, ones I talked about in my last post.
As a corollary, I also think that your mistaken predictions in the past—that we’d give up on suppression or that the control system would fizzle out, are related to this. If you think we operate at higher S levels than in the past, you’d be more inclined to think we’ll sooner or later sleepwalk into a disaster. If you think there is a strong, consistent, S1 drag away from disaster, as I argued way back here, you’d expect strong control system effects that seem surprisingly immune to ‘fatigue’.
Update: this from public health England explicitly says Rt increases by 0.57, https://twitter.com/DevanSinha/status/1341132723105230848?s=20
“We find that Rt increases by 0.57 [95%CI: 0.25-1.25] when we use a fixed effect model for each area. Using a random effect model for each area gives an estimated additive effect of 0.74 [95%CI: 0.44- 1.29].
an area with an Rt of 0.8 without the new variant would have an Rt of 1.32 [95%CI:1.19-1.50] if only the VOC was present.”
But for R, if it’s 0.6 not 0.8 and the ratio is fixed then another march style lockdown in the UK would give R = 0.6 *(1.32/0.8)= 0.99
EDIT: doubling time would go from 17 days to 4 days (!) with the above change of numbers. This doesn’t fit given what is currently observed.
The doubling time for the new strain does appear to be around 6-7 days. And the doubling time for London overall is currently 6 days.
If the mitigated Rt is +0.66 and the growth rate is +71% figures are inconsistent with each other as you say, then perhaps the second is mistaken and +71% means that the Rt is 71% higher, not the case growth rate, which is vaguely consistent with the Rt is +58% higher estimate from the absolute increase. Or “71% higher daily growth rate” could be right and the +0.66 could be referring to the R0, as you say.
This does appear to have been summarized as ‘the new strain is 71% more infectious’ in many places, and many people have apparently inferred the R0 is >50% higher—hopefully we’re wrong.
Computer modelling of the viral spread suggests the new variant could be 70 per cent more transmissible. The modelling shows it may raise the R value of the virus — the average number of people to whom someone with Covid-19 passes the infection — by at least 0.4,
I think this is what happens when people don’t show their work.
So either ‘R number’ is actually referring to R0 and not Rt, or ‘growth rate’ isn’t referring to the daily growth rate but to the Rt/R0. I agree that the first is more plausible. All I’ll say is that a lot of people are assuming the 70% figure or something close to it is a direct multiplier to the Rt, including major news organizations like the Times and Ft. But I think you’re probably right and the R0 is more like 15% larger not 58/70% higher.
EDIT: New info from PHE seems to contradict this, https://t.co/r6GOyXFDjh?amp=1
EDIT: PHE has seemingly confirmed the higher estimate for change in R, ~65%. https://t.co/r6GOyXFDjh?amp=1
What, uh, does the “71% higher growth rate” mean
TLDR: I think that it’s probably barely 15% more infectious and the math of spread near equilibrium amplifies things.
I admit that I have not read all available documents in detail, but I presume that what they said means something like “if ancestor has a doubling time of X, then variant is estimated as having a doubling time of X/(1+0.71) = 0.58X”
In the meeting minutes, the R-value (Rt) was estimated to have increased by 0.39 to 0.93, the central estimate being +0.66 - ‘an absolute increase in the R-value of between 0.39 to 0.93’. Then we see ‘the growth rate is 71% higher than other variants’. You’re right that this is referring to the case growth rate—they’re saying the daily increase is 1.71 times higher, possibly?
I’m going to estimate the relative difference in Rt of the 2 strains from the absolute difference they provided—the relative difference in Rt (Rt(new covid now)/Rt(old covid now)) in the same region, should, I think, be the factor that tells us how more infectious the new strain is.
We need to know what the pre-existing, current, Rt of just the old strain of covid-19 is. Current central estimate for covid in the UK overall is 1.15. This guess was that the ‘old covid’ Rt was 1.13.
(0.66+1.13)/1.13 = 1.79 (Rt of new covid now)/1.13(Rt of old covid now) = 1.58, which implies that the Rt of the new covid is currently 58% higher than the old, which should be a constant factor, unless I’m missing something fundamental. (For what it’s worth, the Rt in london where the new strain makes up the majority of cases is close to that 1.79) value). So, the Rt and the R0 of the new covid is 58% higher—that would make the R0 somewhere around 4.5-5.
Something like that rough conclusion was also reached e.g. here or here or here or here or here, with discussion of ‘what if the R0 was over 5’ or ’70% more infectious’ or ‘Western-style lockdown will not suppress’ (though may be confusing the daily growth rate with the R0). This estimate from different data said the Rt was 1.66/1.13 = 47% higher which is close-ish to the 58% estimate.
I may have made a mistake somewhere here, and those sources have made the same mistake, but this seems inconsistent with your estimate that the new covid is 15% more infectious, i.e. the Rt and R0 is 15% higher not 58% higher.
This seems like a hugely consequential question. If the Rt of the new strain is more than ~66% larger than the Rt of the old strain, then March-style lockdowns which reduced Rt to 0.6 will not work, and the covid endgame will turn into a bloody managed retreat, to delay the spread and flatten the curve for as long as possible while we try to vaccinate as many people as possible. Of course, we should just go faster regardless:
Second, we do have vaccines and so in any plausible model faster viral spread implies a faster timetable for vaccine approval and distribution. And it implies we should have been faster to begin with. If you used to say “we were just slow enough,” you now have to revise that opinion and believe that greater speed is called for, both prospectively and looking backwards. In any plausible model.
If you are right then this is just a minor step up in difficulty.
Tom Chivers agrees with you, that this is an ‘amber light’, metaculus seems undecided (probability of UK 2nd wave worse than 1st; increased by 20% to 42% when this news appeared), some of the forecasters seem to agree with you or be uncertain.
On the economic front, we would have had to choose either to actually suppress the virus, in which case we get much better outcomes all around, or to accept that the virus couldn’t be stopped, *which also produces better economic outcomes. *
Our technological advancement gave us the choice to make massively larger Sacrifices to the Gods rather than deal with the situation. And as we all know, choices are bad. We also are, in my model, much more inclined to make such sacrifices now than we were in the past,
So, by ‘Sacrifices to the Gods’ I assume you’re referring to the entirety of our suppression spending—because it’s not all been wasted money, even if a large part of it has. In other places you use that phrase to refer specifically to ineffective preventative measures.
‘We also are, in my model, much more inclined to make such sacrifices now than we were in the past ’- this is a very important point that I’m glad you recognise—there has been a shift in values such that we (as individuals, as well as governments) are guaranteed to take the option of attempting to avoid getting the virus and sacrificing the economy to a greater degree than in 1919, or 1350, because our society values human life and safety differently.
And realistically, if we’d approached this with pre-2020 values and pre-2020 technology, we’d have ‘chosen’ to let the disease spread and suffered a great deal of death and destruction—but that option is no longer open to us. For better, as I think, or for worse, as you think.
You can do the abstract a cost-benefit calculation about whether the other harms of the disease have caused more damage than the disease, but it won’t tell you anything about whether the act of getting governments to stop lockdowns and suppression measures will be better or worse than having them to try. Robin Hanson directly confuses these two in his argument that we are over-preventing covid.
We see variations in both kinds of policy across space and time, due both to private and government choices, all of which seem modestly influenceable by intellectuals like Caplan, Cowen, and I...
But we should also consider the very real possibility that the political and policy worlds aren’t very capable of listening to our advice about which particular policies are more effective than others. They may well mostly just hear us say “more” or “less”, such as seems to happen in medical and education spending debates.
Here Hanson is equivocating between (correctly) identifying the entire cost of COVID-19 prevention as due to ‘both private and government choices’ and then focussing on just ‘the political and policy worlds’ in response to whether we should argue for less prevention. The claim (which may or may not be true) that ‘we overall are over-preventing covid relative to the abstract alternative where we don’t’ gets equated to ‘therefore telling people to overall reduce spending on covid prevention will be beneficial on cost-benefit terms’.
Telling governments to spend less money is much more likely to work than ordering people to have different values. So making governments spend less on covid prevention diminishes their more effective preventative actions while doing very little about the source of most of the covid prevention spending (individual action).
Like-for-like comparisons where values are similar but policy is different (like Sweden and its neighbours), make it clear that given the underlying values we have, which lead to the behaviours that we have observed this year, the imperative ‘prevent covid less’ leads to outcomes that are across the board worse.
Or consider Sweden, which had a relatively non-panicky Covid messaging, no matter what you think of their substantive policies. Sweden didn’t do any better on the gdp front, and the country had pretty typical adverse mobility reactions. (NB: These are the data that you don’t see the “overreaction” critics engage with — at all. And there is more where this came from.)
How about Brazil? While they did some local lockdowns, they have a denialist president, a weak overall response, and a population used to a high degree of risk. The country still saw a gdp plunge and lots of collateral damage. You might ponder this graph, causality is tricky and the “at what margin” question is trickier yet, but it certainly does not support what Bryan is claiming about the relevant trade-offs.
So, with the firm understanding that given the values we have, and the behaviour patterns we will inevitably adopt, telling people to prevent the pandemic less is worse economically and worse in terms of deaths, we can then ask the further, more abstract question that you ask—what if our values were different? That is, what if the option was available to us because we were actually capable of letting the virus rip.
I wanted to put that disclaimer in because discussing whether we have developed the right societal values is irrelevant for policy decisions going forward—but still important for other reasons. I’d be quite concerned if our value drift over the last century or so was revealed as overall maladapted, but it’s important to talk about the fact that this is the question that’s at stake when we ask if society is over-preventing covid. I am not asking whether lockdowns or suppression are worth it now—they are.
You seem to think that our values should be different; that it’s at least plausible that signalling is leading us astray and causing us to overvalue the direct damage of covid, like lives lost, in place of concern for overall damage. Unlike Robin Hanson, though, you aren’t recommending we attempt to tell people to go off and have different values—you’re simply noting that you think our tendency to make larger sacrifices is a mistake.
...even when the trade-offs are similar, which ties into my view that simulacra and maze levels are higher, with a larger role played by fear of motive ambiguity. We might have been willing to do challenge trials or other actual experiments, and have had a much better handle on things quicker on many levels.
There are two issues here—one is that it’s not at all clear whether the initial cost-benefit calculation about over-prevention is even correct. You don’t claim to know if we are over-preventing in this abstract sense (compared to us having different values and individually not avoiding catching the disease), and the evidence that we are over-preventing comes from a twitter poll of Bryan Caplan’s extremely libertarian-inclined followers who he told to try as hard as possible to be objective in assessing pandemic costs because he asked them what ‘the average American’ would value (Come on!!). Tyler Cowen briefly alludes to how woolly the numbers are here, ‘I don’t agree with Bryan’s numbers, but the more important point is one of logic’.
The second issue is whether our change in values is an aberration caused by runaway signalling or reflects a legitimate, correct valuation of human life. Now, the fact that a lot of our prevention spending has been wasteful counts in favour of the signalling explanation, but on the other hand there’s a ton of evidence that we in the past, in general, valued life too little. [There’s also the point that this seems like exactly a case where a signalling explanation is hard to falsify, an issue I talked about here,
I worry that there is a tendency to adopt self-justifying signalling explanations, where an internally complicated signalling explanation that’s hard to distinguish from a simpler ‘lying’ explanation, gets accepted, not because it’s a better explanation overall but just because it has a ready answer to any objections. If ‘Social cognition has been the main focus of Rationality’ is true, then we need to be careful to avoid overusing such explanations. Stefan Schubert explains how this can end up happening:
I think the correct story is that the value shift has been good and bad—valuing human life more strongly has been good, but along with that its become more valuable to credibly fake valuing human life, which has been bad.
Yeah—this is a case where how exactly the transition goes seems to make a very big difference. If it’s a fast transition to a singleton, altering the goals of the initial AI is going to be super influential. But if it’s that there are many generations of AIs that over time become the larger majority of the economy, then just control everything—predictably altering how that goes seems a lot harder at least.
Comparing the entirety of the Bostrom/Yudkowsky singleton intelligence explosion scenario to the slower more spread out scenario, it’s not clear that it’s easier to predictably alter the course of the future in the first compared to the second.
In the first, assuming you successfully set the goals of the singleton, the hard part is over and the future can be steered easily because there are, by definition, no more coordination problems to deal with. But in the first, a superintelligent AGI could explode on us out of nowhere with little warning and a ‘randomly rolled utility function’, so the amount of coordination we’d need pre-intelligence explosion might be very large.
In the second slower scenario, there are still ways to influence the development of AI—aside from massive global coordination and legislation, there may well be decision points where two developmental paths are comparable in terms of short-term usefulness but one is much better than the other in terms of alignment or the value of the long-term future.
Stuart Russell’s claim that we need to replace ‘the standard model’ of AI development is one such example—if he’s right, a concerted push now by a few researchers could alter how nearly all future AI systems are developed for the better. So different conditions have to be met for it to be possible to predictably alter the future long in advance on the slow transition model (multiple plausible AI development paths that could be universally adopted and have ethically different outcomes) compared to the fast transition model (the ability to anticipate when and where the intelligence explosion will arrive and do all the necessary alignment work in time), but its not obvious to me one is easier to meet than the other.
For this reason, I think it’s unlikely there will be a very clearly distinct “takeoff period” that warrants special attention compared to surrounding periods.
I think the period AI systems can, at least in aggregate, finally do all the stuff that people can do might be relatively distinct and critical—but, if progress in different cognitive domains is sufficiently lumpy, this point could be reached well after the point where we intuitively regard lots of AI systems as on the whole “superintelligent.”
This might be another case (like ‘the AIs utility function’) where we should just retire the term as meaningless, but I think that ‘takeoff’ isn’t always a strictly defined interval, especially if we’re towards the medium-slow end. The start of the takeoff has a precise meaning only if you believe that RSI is an all-or-nothing property. In this graph from a post of mine, the light blue curve has an obvious start to the takeoff where the gradient discontinuously changes, but what about the yellow line? There clearly is a takeoff in that progress becomes very rapid, but there’s no obvious start point, but there is still a period very different from our current period that is reached in a relatively short space of time—so not ‘very clearly distinct’ but still ‘warrants special attention’.
At this point I think it’s easier to just discard the terminology altogether. For some agents, it’s reasonable to describe them as having goals. For others, it isn’t. Some of those goals are dangerous. Some aren’t.
Daniel Dennett’s Intentional stance is either a good analogy for the problem of “can’t define what has a utility function” or just a rewording of the same issue. Dennett’s original formulation doesn’t discuss different types of AI systems or utility functions, ranging in ‘explicit goal directedness’ all the way from expected-minmax game players to deep RL to purely random agents, but instead discusses physical systems ranging from thermostats up to humans. Either way, if you agree with Dennett’s formulation of the intentional stance I think you’d also agree that it doesn’t make much sense to speak of ’the utility function as necessarily well-defined.
Much of Europe went into strict lockdown. I was and am still skeptical that they were right to keep schools open, but it was a real attempt that clearly was capable of working, and it seems to be working.
The new American restrictions are not a real attempt, and have no chance of working.
The way I understand it is that ‘being effective’ is making an efficient choice taking into account asymmetric risk and the value of information, and the long-run trade-offs. This involves things like harsh early lockdowns, throwing endless money at contact tracing, and strict enforcement of isolation. Think Taiwan, South Korea.
Then ‘trying’ is adopting policies that have a reasonable good chance of working, but not having a plan if they don’t work, not erring on the side of caution of taking into account asymmetric risk when you adopt the policies, and not responding to new evidence quickly. The schools thing is a perfect example—closing has costs (makes the lockdown less effective and therefore longer), and it wasn’t overwhelmingly clear that schools had to close to turn R under 1, so that was good enough. Partially funding tracing efforts, waiting until there’s visibly no other choice and then calling a strict lockdown—that’s ‘trying’. Think the UK and France.
And then you have ‘trying to try’, which you explain in detail.
Dolly Parton helped fund the Moderna vaccine. Neat. No idea why anyone needed to do that, but still. Neat.
It’s reassuring to know that if the administrative state and the pharmaceutical industry fails, we have Dolly Parton.
That said, I remain interested in more clarity on what you see as the biggest risks with these multi/multi approaches that could be addressed with technical research.
A (though not necessarily the most important) reason to think technical research into computational social choice might be useful is that examining specifically the behaviour of RL agents from a computational social choice perspective might alert us to ways in which coordination with future TAI might be similar or different to the existing coordination problems we face.
(i) make direct improvements in the relevant institutions, in a way that anticipates the changes brought about by AI but will most likely not look like AI research,
It seems premature to say, in advance of actually seeing what such research uncovers, whether the relevant mechanisms and governance improvements are exactly the same as the improvements we need for good governance generally, or different. Suppose examining the behaviour of current RL agents in social dilemmas leads to a general result which in turn leads us to conclude there’s a disproportionate chance TAI in the future will coordinate in some damaging way that we can resolve with a particular new regulation. It’s always possible to say, solving the single/single alignment problem will prevent anything like that from happening in the first place, but why put all your hopes on plan A, when plan B is relatively neglected?
Thanks for this long and very detailed post!
The MARL projects with the greatest potential to help are probably those that find ways to achieve cooperation between decentrally trained agents in a competitive task environment, because of its potential to minimize destructive conflicts between fleets of AI systems that cause collateral damage to humanity. That said, even this area of research risks making it easier for fleets of machines to cooperate and/or collude at the exclusion of humans, increasing the risk of humans becoming gradually disenfranchised and perhaps replaced entirely by machines that are better and faster at cooperation than humans.
In ARCHES, you mention that just examining the multiagent behaviour of RL systems (or other systems that work as toy/small-scale examples of what future transformative AI might look like) might enable us to get ahead of potential multiagent risks, or at least try to predict how transformative AI might behave in multiagent settings. The way you describe it in ARCHES, the research would be purely exploratory,
One approach to this research area is to continually ex-amine social dilemmas through the lens of whatever is the leading AI devel-opment paradigm in a given year or decade, and attempt to classify interest-ing behaviors as they emerge. This approach might be viewed as analogousto developing “transparency for multi-agent systems”: first develop inter-esting multi-agent systems, and then try to understand them.
But what you’re suggesting in this post, ‘those that find ways to achieve cooperation between decentrally trained agents in a competitive task environment’, sounds like combining computational social choice research with multiagent RL - examining the behaviour of RL agents in social dilemmas and trying to design mechanisms that work to produce the kind of behaviour we want. To do that, you’d need insights from social choice theory. There is some existing research on this, but it’s sparse and very exploratory.
OpenAI just released a paper on RL agents in social Dilemmas, https://arxiv.org/pdf/2011.05373v1.pdf and there is some previous work. This is more directly multiagent RL, but there is some consideration for things like choosing the right overall social welfare metric.
There are also two papers examining bandit algorithms in iterated voting scenarios, https://hal.archives-ouvertes.fr/hal-02641165/document and https://www.irit.fr/~Umberto.Grandi/scone/Layka.m2.pdf.
My current research is attempting to build on the second of these.
As far as I can tell, that’s more or less it in terms of examining RL agents in social dilemmas, so there may well be a lot of low-hanging fruit and interesting discoveries to be made. If the research is specifically about finding ways of achieving cooperation in multiagent systems by choosing the correct (e.g. voting) mechanism, is that not also computational social choice research, and therefore of higher priority by your metric?
In short, computational social choice research will be necessary to legitimize and fulfill governance demands for technology companies (automated and human-run companies alike) to ensure AI technologies are beneficial to and controllable by human society.
...
CSC neglect:
As mentioned above, I think CSC is still far from ready to fulfill governance demands at the ever-increasing speed and scale that will be needed to ensure existential safety in the wake of “the alignment revolution”.
I made an attempt to model intelligence explosion dynamics in this post, by attempting to make the very oversimplified exponential-returns-to-exponentially-increasing-intelligence model used by Bostrom and Yudkowsky slightly less oversimplified.
The page includes python code for the model.
This post doesn’t capture all the views of takeoff—in particular it doesn’t capture the non-hyperbolic faster growth mode scenario, where marginal intelligence improvements are exponentially increasingly difficult and therefore we get a (continuous or discontinuous switch to a) new exponential growth mode rather than runaway hyperbolic growth.
But I think that by modifying the f(I) function that determines how RSI capability varies with intelligence we can incorporate such views.
(In the context of the exponential model given in the post that would correspond to an f(I) function where
f(I)=1I(1+e−d(I(t)−IAGI))which would result in a continuous (determined by size of d) switch to a single faster exponential growth mode)
But I think the model still roughly captures the intuition behind scenarios that involve either a continuous or a discontinuous step to an intelligence explosion.