Philosophy PhD student, worked at AI Impacts, now works at Center on Long-Term Risk. Research interests include acausal trade, timelines, takeoff speeds & scenarios, decision theory, history, and a bunch of other stuff. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html
Daniel Kokotajlo
Karma: 8,983
Thanks, I found this helpful!
Consequentialism is convergently instrumental. Consequentialism is a (relatively) simple, effective process for accomplishing goals, so things that efficiently optimize for goals tend to approximate it.
I think this is the most important premise. I don’t have a solid justification for it yet, but I’m groping towards a non-solid justification at least in my agency sequence. I think John Wentworth’s stuff on the good regulator theorem is another line of attack that could turn into a solid justification. TurnTrout also has relevant work IIRC.
Nothing to apologize for, it was reasonably clear, I’m just trying to learn more about what you believe and why. This has been helpful, thanks!
I totally agree that in fast takeoff scenarios we are less likely to spot those things until it’s too late. I guess I agree that truthful LM work is less likely to scale gracefully to AGI in fast takeoff scenarios… so I guess I agree with your overall point… I just notice I feel a bit confused and muddle about it, is all. I can imagine plausible slow-takeoff scenarios in which truthful LM work doesn’t scale gracefully, and plausible fast-takeoff scenarios in which it does. At least, I think I can. The former scenario would be something like: It turns out the techniques we develop for making dumb AIs truthful stop working once the AIs get smart, for similar reasons that techniques we use to make small children be honest (or to put it more vividly, believe in santa) stop working once they grow up. The latter scenario would be something like: Actually that’s not the case, the techniques work all the way up past human level intelligence, and “fast takeoff” in practice means “throttled takeoff” where the leading AI project knows they have a few month lead over everyone else and is using those months to do some sort of iterated distillation and amplification, in which it’s crucial that the early stages be truthful and that the techniques scale to stage N overseeing stage N+1.
Fixed, thanks!
Thanks, these clarifications are very helpful.
FWIW I think paul slow takeoff is pretty unlikely for reasons to be found in this thread and this post. On the other hand, as someone who thinks fast takeoff (in various senses) is more likely than not, I don’t yet see why that makes Truthful LM work significantly less useful. (By contrast I totally see why Truthful LM work is significantly less useful if AGI/TAI/etc. comes from stuff that doesn’t resemble modern deep learning.)
“Catch misalignment early...” This makes it sound like misalignment is something that AIs don’t have yet but might one day have, so we need to be vigilant and notice it when it appears. But instead isn’t misalignment something that all AIs have by default?
My current view is that power-seeking misalignment will probably cause existential catastrophe, that persuasion tools happen first and have a >20% chance of destroying our ability to solve that problem, and that there are various philosophical and societal problems that could (>20%) get us even if we solve power-seeking misalignment. Does this mean I agree or disagree with “our current picture of the risks is incomplete?”
Thanks for this!
I think that working on truthful LMs has a comparative advantage in worlds where:
--We have around 10-40 years until transformative AI
--Transformative AI is built using techniques that resemble modern deep learning
--There is a slow takeoff
--Alignment does not require vastly more theoretical insight (but may require some)
--Our current picture of the risks posed by transformative AI is incomplete
Can you elaborate on what you mean by slow takeoff here?
Also, what do you mean by the current picture of the risks being incomplete? What would it even mean for our picture to be complete?
There is no clear line of argument from beginning to end but rather a disjunctive list of possibilities that all lead to similar extinction events. This leads the entire theory to not be falsifiable, cut off one road toward superintelligence or a related extinction outcome and a new one will pop up. Whatever amount of evidence mounted against it will never be enough.
Every sentence in the above is false, even if you condition on the previous sentences being true. On clarity, I guess it’s a matter of subjective judgment but Superintelligence seems significantly more clear than the average book, even the average academic book, IMO. If you don’t think it’s clear enough, check out Joseph Carlsmith’s report on power-seeking AI, which literally does it as a big conjunctive premise-conclusion-form argument. On falsifiability, this is just not how falsifiability works, as Lukas Gloor points out.* On “whatever amount of evidence mounted against it will never be enough...” well, even if it were unfalsifiable, you could still in principle provide enough evidence to change people’s minds about it, unless people are being super stubborn, which is totally possible (happens all the time, people are closed-minded about things) but not true of Bostrom et al IMO.
*I do agree that Superintelligence would be even more impressive if it had stuck its neck out and made a bunch of bold near-term predictions which had then turned out to be true. In that sense it was weakly unfalsifiable, in the same way that almost every book about almost everything is. The only bold near-term prediction I recall it making was that the recent surge of progress and interest in artificial intelligence was not going to subside into another AI winter, but rather would continue to grow and grow. At the time this was a pretty bold prediction IMO, since back then the chorus of talking heads forecasting AI Winter was louder than it is now. I don’t have my copy of Superintelligence with me now so I can’t check whether this prediction was actually made, sorry.
I typed in the questions to GPT-3 and pressed “generate” to see its answers. I used a pretty simple prompt.
Tonight my family and I played a trivia game (Wits & Wagers) with GPT-3 as one of the players! It lost, but not by much. It got 3 questions right out of 13. One of the questions it got right it didn’t get exactly right, but was the closest and so got the points. (This is interesting because it means it was guessing correctly rather than regurgitating memorized answers. Presumably the other two it got right were memorized facts.)
Anyhow, having GPT-3 playing made the whole experience more fun for me. I recommend it. :) We plan to do this every year with whatever the most advanced publicly available AI (that doesn’t have access to the internet) is.
Because if you don’t trust someone with GPT-3, you are certainly not going to trust them with an AGI.
Choosing to not release GPT-3′s weights to the whole world doesn’t imply that you don’t trust DeepMind or Anthropic or whoever. It just implies that there exists at least one person in the world you don’t trust.
I agree that releasing everything publicly would make it easier/more likely to release crucial things to key competitors when the time comes. Alas, the harms are big enough to outweigh this benefit, I think.
Thanks! In case you are interested in my opinion: I think I agree with 1 (but I expect us to fail miserably) and 2 and 6. I disagree with 3 (AGI, unlike nukes, can be used in ways that aren’t threatening and don’t hurt anyone and just make loads of money and save loads of lives. So people will find it hard to resist using it. So the more people have access to it, the more likely it is it’ll be used before it is safe.) My timelines are about 50% by 10 years; I gather from point 4 that yours are longer. I think 5 might be true but might not be; history is full of examples of different groups fighting each other yet still managing to conquer and crush some third group. For example, the Spanish conquistadors were fighting each other even as they conquered Mexico and Peru. Maybe humans will be clever enough to play the AIs off against each other in a way that lets us maintain control until we solve alignment… but I wouldn’t bet on it.
I recommend reading The Abolition of Man by C.S. Lewis. The book explores pretty much this idea! I think it should be required reading.
Yep! :) But the damage is done; thanks to OpenAI there is now a large(r) group of people who believe in FOSS AI than there otherwise would be, and there are various new actors who have entered the race who wouldn’t have if not for OpenAI publications.
To be clear, I’m not confident FOSS AI is bad. I just think it probably is, for reasons mentioned. I won’t be too surprised if I turn out to be wrong and actually (e.g. for reasons Dirichlet-to-Neumann mentioned, or because FOSS AI is harder to make a profit on and therefore will get less investment and therefore will be developed later, buying more time for safety research) the FOSS AI ethos was net-positive.
I’d be interested to hear your perspective on FOSS AI.
Yes, when we are getting really close to AGI it will be good for the leading contenders to share info with each other. Even then it won’t be a good idea for the leading contenders to publish publicly, because then there’ll be way more contenders! And now, when we are not really close to AGI, public publication accelerates research in general and thus shortens timelines, while also bringing more actors into the race.
IMO the main consideration should be whether she wants it or not. If she’s happy at school, great. If she’s being bullied or is super bored all the time, take her out. But where to draw the line? I was reasonably happy at school I guess (it’s hard to remember) but I bet if someone asked me whether I’d rather stay home and play all day I would have said yes.
I don’t have a precise definition in mind since I was parrotting Yonadav. My point was that SpaceX is way better than a random similarly-sized group of high schoolers in many many important ways, even though SpaceX consumes just as many calories/energy as the high schoolers, such that it’s totally misleading to describe them as “roughly equivalent except that SpaceX has a massive coordination advantage.” The only thing roughly equivalent about them is their energy consumption, which just goes to show energy consumption is not a useful metric here.
I totally agree that fewer, larger brains with experience advantages seem likely to outcompete many merely human-sized brains. In fact I think I agree with everything you said in this comment.
Ah, I see. I think 10^9 is not a meaningful number to be talking about; long before there are 10^9 brain-equivalents worth of compute going into AI, we’ll be past the point of no return. But if instead you are talking about an amount of compute large enough that energy companies should be able to detect it, then yeah this seems fairly plausible. Supercomputers can’t be hidden from energy companies as far as I know, and plausibly AGI will appear first in supercomputers, so plausibly wherever AGI appears, it’ll be known by some government that the project was underweigh at least.
I don’t think this meaningfully lowers the probability of treacherous turn due to a system being way above our capabilities though. That’s because I didn’t put much probability mass on secret-AGI-project-in-a-basement scenarios anyway. I guess if I had, then this would have updated me.
If I understand you correctly, you are asking something like: How many programmer-hours of effort and/or how much money was being spent specifically on scaling up large models in 2020? What about in 2025? Is the latter plausibly 4 OOMs more than the former? (You need some sort of arbitrary cutoff for what counts as large. Let’s say GPT-3 sized or bigger.)
Yeah maybe, I don’t know! I wish I did. It’s totally plausible to me that it could be +4 OOMs in this metric by 2025. It’s certainly been growing fast, and prior to GPT-3 there may not have been much of it at all.
What do you mean by “arrest FOOM?” I am quite confident that by the time the intelligence explosion starts winding down, it’ll be past the point of no return for humans. Maybe from the AIs perspective progress will have stagnated due to compute constraints, and further progress will happen only once they can design exotic new hardware, so subjectively it feels like aeons of stagnation. But I think that “human-civilization-scale collective intelligence except with much better coordination abilities” is vastly underselling it. It’s like saying caveman humans were roughly elephant-scale intelligences except with better coordination abilities. Or saying that SpaceX is “roughly equivalent to the average US high school 9th grade class, except with better coordination abilities.” Do you disagree with this?
Update: I just came across this poll, which presumably is biased etc. but still, pretty amazing, I predicted a rise in authoritarian sentiment but I would not have predicted anything nearly this extreme, especially after the virus has mostly run its course and vaccines have been deployed: