Global equity markets may have underestimated the economic effects of a potential COVID-19 pandemic because the only historical parallel to it is the 1918 flu pandemic (which is likely worse than COVID-19 due to a higher fatality rate) and stock markets didn’t drop that much. But maybe traders haven’t taken into account (and I just realized this) that there was war-time censorship in effect which strongly downplayed the pandemic and kept workers going to factories, which is a big disanalogy between the two cases, so markets could drop a lot more this time around. The upshot is that maybe it’s not too late to short the markets.
Wei_Dai(Wei Dai)
Karma: 33,780
So is this a good thing or bad thing? Is wokeness a stepping stone towards some future enlightened morality, and help bring AI/ML along to that destination (in which case perhaps we should excuse its current excesses), or will it ultimately collapse while doing a lot of damage in the meantime (like communism)?
Further alarming evidence of humanity’s inability to coordinate (especially in an emergency), and also relevant to recent discussions around terminology: ‘A bit chaotic.’ Christening of new coronavirus and its disease name create confusion
Anyone else look at the coronavirus outbreak and think this is how a future existential accident will play out, with policy always one step behind what’s necessary to control it, because governments don’t want to take the risk of “overreacting” and triggering large or very large political and economic costs “for no good reason”. So they wait until absolutely clear evidence emerge, by which time it will be too late.
Why will governments be more afraid of overreacting than underreacting? (Parallel: We don’t seem to see governments doing anything in this outbreak that could be interpreted as overreacting.) Well, every alarm up to that point will have been a false alarm. (Parallel: SARS, MERS, Ebola, every flu pandemic in recent history.)
An observation on natural language being illogical: I’ve noticed that at least some native Chinese speakers use 不一定 (literally “not certain”) to mean “I disagree”, including when I say “I think there’s 50% chance that X.” At first I was really annoyed with the person doing that (“I never said I was certain!”) but then I noticed another person doing it so now I think it’s just a standard figure of speech at this point, and I’m just generally annoyed at … cultural evolution, I guess.
From your paper:
It is interesting to note that AIXI, a Bayes-optimal reinforcement learner in general environments,is not asymptotically optimal [Orseau, 2010], and in-deed, may cease to explore [Leikeet al., 2015]. Depending on its prior and its past observations, AIXI may decide at some point that further exploration is not worth the risk. Given our result, this seems like reasonable behavior.
Given this, why is your main conclusion “Perhaps our results suggest we are in need of more theory regarding the ‘parenting’ of artificial agents” instead of “We should use Bayesian optimality instead of asymptotic optimality”?
Copy-pasting a followup to this with Robin Hanson via DM (with permission).
Robin: You can of course suspect people of many things using many weak clues. But you should hold higher standards of evidence when making public accusations that you say orgs should use to fire people, cancel speeches, etc.
Me: My instinct is to support/agree with this, but (1) it’s not an obvious interpretation of what you tweeted and (2) I think we need to understand why the standards of evidence for making public accusations and for actual firing/canceling have fallen so low (which my own comment didn’t address either) and what the leverage points are for changing that, otherwise we might just be tilting at windmills when we exhort people to raise those standards (or worse, making suicide charges, if we get lumped with “public enemies”).
I agree that this is troubling, though I think it’s similar to how I wouldn’t want the term biorisk to be expanded …
Well as I said, natural language doesn’t have to be perfectly logical, and I think “biorisk” is in somewhat in that category but there’s an explanation that makes it a bit reasonable than it might first appear, which is that the “bio” refers not to “biological” but to “bioweapon”. This is actually one of the definitions that Google gives when you search for “bio”: “relating to or involving the use of toxic biological or biochemical substances as weapons of war. ‘bioterrorism’”
I guess the analogous thing would be if we start using “AI” to mean “technical AI accidents” in a bunch of phrases, which feels worse to me than the “bio” case, maybe because “AI” is a standalone word/acronym instead of a prefix? Does this make sense to you?
Not to say that’s what you are doing with AI risk. I’m worried about what others will do with it if the term gets expanded.
But the term was expanded from the beginning. Have you actually observed it being used in ways that you fear (and which would be prevented if we were to redefine it more narrowly)?
Also, isn’t defining “AI risk” as “technical accidental AI risk” analogous to defining “apple” as “red apple” (in terms of being circular/illogical)? I realize natural language doesn’t have to be perfectly logical, but this still seems a bit too egregious.
But I am optimistic about the actual risks that you and others argue for.
Why? I actually wrote a reply that was more questioning in tone, and then changed it because I found some comments you made where you seemed to be concerned about the additional AI risks. Good thing I saved a copy of the original reply, so I’ll just paste it below:
I wonder if you would consider writing an overview of your perspective on AI risk strategy. (You do have a sequence but I’m looking for something that’s more comprehensive, that includes e.g. human safety and philosophical problems. Or let me know if there’s an existing post that I’ve missed.) I ask because you’re one of the most prolific participants here but don’t fall into one of the existing “camps” on AI risk for whom I already have good models for. It’s happened several times that I see a comment from you that seems wrong or unclear, but I’m afraid to risk being annoying or repetitive with my questions/objections. (I sometimes worry that I’ve already brought up some issue with you and then forgot your answer.) It would help a lot to have a better model of you in my head and in writing so I can refer to that to help me interpret what the most likely intended meaning of a comment is, or to predict how you would likely answer if I were to ask certain questions.
It’s notable that AI Impacts asked for people who were skeptical of AI risk (or something along those lines) and to my eye it looks like all four of the people in the newsletter independently interpreted that as accidental technical AI risk in which the AI is adversarially optimizing against you (or at least that’s what the four people argued against).
Maybe that’s because the question was asked in a way that indicated the questioner was mostly interested in technical accidental AI risk? And some of them may be fine with defining “AI risk” as “AI-caused x-risk” but just didn’t have the other risks on the top of their minds, because their personal focus is on the technical/accidental side. In other words I don’t think this is strong evidence that all 4 people would endorse defining “AI risk” as “technical accidental AI risk”. It also seems notable that I’ve been using “AI risk” in a broad sense for a while and no one has objected to that usage until now.
I would certainly support having clearer definitions and terminology if we could all agree on them.
The current situation seems to be that we have two good (relatively clear) terms “technical accidental AI risk” and “AI-caused x-risk” and the dispute is over what plain “AI risk” should be shorthand for. Does that seem fair?
AI risk is just a shorthand for “accidental technical AI risk.”
I don’t think “AI risk” was originally meant to be a shorthand for “accidental technical AI risk”. The earliest considered (i.e., not off-hand) usage I can find is in the title of Luke Muehlhauser’s AI Risk and Opportunity: A Strategic Analysis where he defined it as “the risk of AI-caused extinction”.
(He used “extinction” but nowadays we tend think in terms of “existential risk” which also includes “permanent large negative consequences”, which seems like an reasonable expansion of “AI risk”.)
However, I disagree with the idea that we should expand the word AI risk to include philosophical failures and intentional risks.
I want to include philosophical failures, as long as the consequences of the failures flow through AI, because (aside from historical usage) technical problems and philosophical problems blend into each other, and I don’t see a point in drawing an arbitrary and potentially contentious border between them. (Is UDT a technical advance or a philosophical advance? Is defining the right utility function for a Sovereign Singleton a technical problem or a philosophical problem? Why force ourselves to answer these questions?)
As for “intentional risks” it’s already common practice to include that in “AI risk”:
Dividing AI risks into misuse risks and accident risks has become a prevailing approach in the field.
Besides that, I think there’s also a large grey area between “accident risk” and “misuse” where the risk partly comes from technical/philosophical problems and partly from human nature. For example humans might be easily persuaded by wrong but psychologically convincing moral/philosophical arguments that AIs can come up with and then order their AIs to do terrible things. Even pure intentional risks might have technical solutions. Again I don’t really see the point of trying to figure out which of these problems should be excluded from “AI risk”.
It becomes unclear in conversation what people mean when they say AI risk
It seems perfectly fine to me to use that as shorthand for “AI-caused x-risk” and use more specific terms when we mean more specific risks.
Like The Singularity, it becomes a buzzword
What do you mean? Like people will use “AI risk” when their project has nothing to do with “AI-caused x-risk”? Couldn’t they do that even if we define “AI risk” to be “accidental technical AI risk”?
Journalists start projecting Terminator scenarios onto the words, and now have justification because even the researchers say that AI risk can mean a lot of different things.
Terminator scenarios seem to be scenarios of “accidental technical AI risk” (they’re just not very realistic scenarios) so I don’t see how defining “AI risk” to mean that would prevent journalists from using Terminator scenarios to illustrate “AI risk”.
It puts a whole bunch of types of risk into one basket, suggesting to outsiders that all attempts to reduce “AI risk” might be equally worthwhile.
I don’t think this is a good argument, because even within “accidental technical AI risk” there are different problems that aren’t equally worthwhile to solve, so why aren’t you already worried about outsiders thinking all those problems are equally worthwhile?
ML researchers start to distrust AI risk researchers, because people who are worried about the Terminator are using the same words as the AI risk researchers and therefore get associated with them.
See my response above regarding “Terminator scenarios”.
This can all be avoided by having a community norm to clarify that we mean technical accidental risk when we say AI risk, and when we’re talking about other types of risks we use more precise terminology.
I propose that we instead stick with historical precedent and keep “AI risk” to mean “AI-caused x-risk” and use more precise terminology to refer to more specific types of AI-caused x-risk that we might want to talk about. Aside from what I wrote above, it’s just more intuitive/commonsensical that “AI risk” means “AI-caused x-risk” in general instead of a specific kind of AI-caused x-risk.
However I appreciate that someone who works mostly on the less philosophical / less human-related problems might find it tiresome to say or type “technical accidental AI risk” all the time to describe what they do or to discuss the importance of their work, and can find it very tempting to just use “AI risk”. It would probably be good to create a (different) shorthand or acronym for it to remove this temptation and to make their lives easier.
Ok, I wasn’t sure that you’d agree, but given that you do, it seems that when you wrote the title of this newsletter “Why AI risk might be solved without additional intervention from longtermists” you must have meant “Why some forms of AI risk …”, or perhaps certain forms of AI risk just didn’t come to your mind at that time. In either case it seems worth clarifying somewhere that you don’t currently endorse interpreting “AI risk” as “AI risk in its entirety” in that sentence.
Similarly, on the inside you wrote:
The main reason I am optimistic about AI safety is that we will see problems in advance, and we will solve them, because nobody wants to build unaligned AI. A likely crux is that I think that the ML community will actually solve the problems, as opposed to applying a bandaid fix that doesn’t scale. I don’t know why there are different underlying intuitions here.
It seems worth clarifying that you’re only optimistic about certain types of AI safety problems.
(I’m basically making the same complaint/suggestion that I made to Matthew Barnett not too long ago. I don’t want to be too repetitive or annoying, so let me know if I’m starting to sound that way.)
But on the strong versions of warning shots, where there’s common knowledge that building an AGI runs a substantial risk of destroying the world, yes, I expect them to not build AGI until safety is solved. (Not to the standard you usually imagine, where we must also solve philosophical problems, but to the standard I usually imagine, where the AGI is not trying to deceive us or work against us.)
To the extent that we expect strong warning shots and ability to avoid building AGI upon receiving such warning shots, this seems like an argument for researchers/longtermists to work on / advocate for safety problems beyond the standard of “AGI is not trying to deceive us or work against us” (because that standard will likely be reached anyway). Do you agree?
The major disanalogy is that catastrophic outcomes of climate change do not personally affect the CEOs of energy companies very much, whereas AI x-risk affects everyone.
Some types of AI x-risk don’t affect everyone though (e.g., ones that reduce the long term value of the universe or multiverse without killing everyone in the near term).
In that case, it’s not really prepping anymore, but more like catch-up, since ideological fanaticism is already widespread? But along these lines, someone (who didn’t want attribution) PM’ed me the idea of “social media shaming insurance” or “cancellation insurance”, and I just came across Free Speech Union, which was apparently created just a few days ago and is offering something like this kind of insurance. Interview with founder of Free Speech Union
what needs explaining is how good epistemic norms like “free inquiry” and “civil discourse” ever became a thing.
An idea for explaining this: some group happened to adopt such norms due to a historical accident, and there happened to be enough low hanging epistemic fruit that could be picked up by a group operating under such norms that the group became successful enough for the norms to spread by conquest and emulation. This also suggests that one reason for the decay of these norms is that we are running out of low hanging fruit.
Robin Hanson also doesn’t seem to be aware of what I wrote in the parent comment:
“Other people are low-decouplers, who see ideas as inextricable from their contexts. For them =… You say ‘By X, I don’t mean Y,’ but when you say X, they will still hear Y.” More precisely, they CHOOSE to hear Y, knowing enough associates also choose that
The rest of us can choose the opposite, & we should not accept their claim that they can decide what words mean & how language works.
But why are some people low-decouplers? I think it’s because “Statements of (purported) empirical fact are often strong Bayesian evidence of the speaker’s morality and politics” so we can’t simply “choose the opposite” without understanding this and its implications.
To put it another way, a lot of times when someone says “By X, I don’t mean Y” they actually secretly do believe Y, so if another person “CHOOSE to hear Y”, that’s not a completely unreasonable heuristic, and we can’t just “not accept their claim that they can decide what words mean & how language works” without acknowledging this.
What kinds of institutions would have made it easier for a core of sanity to survive through, say, 30s Germany or 60s China?
I don’t see how any such institutions could themselves survive, since they’ll be among the highest priority targets for those regimes. It seems that sanity survived in the past only because of (1) privacy / lack of sufficiently advanced surveillance technology and (2) decentralization / lack of global coordination ability. But these are the exact things we need to develop to prevent future tech-drive x-risks. So far I don’t see a way out of this catch-22 (except low probability scenarios like it turns out to be really easy to build an Aligned Sovereign Singleton aka Friendly AI).
Statements of (purported) empirical fact are often strong Bayesian evidence of the speaker’s morality and politics (i.e., what values one holds, what political coalition one supports), and this is a root cause of most group-level bad epistemics. For example someone who thinks eugenics is immoral is less likely (than someone who doesn’t think that) to find it instrumentally useful to make a statement like (paraphrasing) “eugenics may be immoral, but is likely to work in the sense that selective breeding works for animals”, so when someone says that, it is evidence for them not thinking that eugenics is immoral and therefore not belonging to a political coalition that holds “eugenics is immoral” as a part of its ideology.
I think for many people this has been trained into intuition/emotion (you automatically think that someone is bad/evil or hate them if they express a statement of fact that your political enemies are much more likely to make than your political allies) or even reflexive policy (automatically attack anyone who makes such statements).
This seems pretty obvious to me, but some people do not seem to be aware of it (e.g., Richard Dawkins seemed surprised by people’s reactions to his tweet linked above) and I haven’t been able to find any discussion on LW about this.
(Given the above, perhaps it shouldn’t be surprising that bad social epistemics abound, and what needs explaining is how good epistemic norms like “free inquiry” and “civil discourse” ever became a thing.)
I really appreciate the pingback feature, but it looks like on LW it’s limited to the top 5 (by karma) pingbacks, while greaterwrong.com shows all of them. (I tested on these posts.) Is this intentional? If so, I don’t think it makes sense to artificially restricted the “cited by” list this way, especially without any indication that this is happening?
Also, I’d like to make a feature suggestion, namely an option in the settings to receive notifications when one of my posts or comments is linked somewhere on LW or EA Forum.
I bought some S&P 500 put options (SPXW Apr 17 2020 3000 Put (PM) to be specific) a couple of weeks ago. They were 40% underwater at some point because the market kept going up (which confused me a lot), but is up 125% as of today. (Note that it’s very easy to lose 100% of your investment when trading options. In my case, all I’d have to do is not sell the options until April 17 and S&P 500 hasn’t dropped to 3000 by then.) I had to open a brokerage account (most have no minimum I think) then apply to trade options then wait a day to be approved. You can also sell stocks short. You can also bet against foreign markets and specific stocks this way.
The above is for information purposes only. It is not intended to be investment advice. Seek a duly licensed professional for investment advice.