Philosophy PhD student, worked at AI Impacts, then Center on Long-Term Risk, now OpenAI Futures/Governance team. Views are my own & do not represent those of my employer. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html
Daniel Kokotajlo
Karma: 11,837
It was the year, your original interpretation was correct. Key thing to notice: The question is asking about a hypothetical situation in which we magically get lots more compute 5 years before 2020, and then asks what happens by end of 2020 in that hypothetical.
I agree this is something I could do that would be valuable, I just haven’t prioritized it alas. I did say I wanted to write the sequel to that story at some point...
I encourage other people to think about it too. I suggest reasoning as follows: (1) What skills/properties would combine to create APS-AI? Agency, reasoning, situational awareness… maybe persuasion, deception, hacking… etc. (2) Does the training process of OmegaStar or whatever incentivize the growth of those skills? That is, does the net developing those skills help it to perform better in that training process, get reinforced, etc.? (3) Is the model big enough that it could in principle come to develop those skills? (4) Is the model trained long enough that it likely will in practice come to have those skills?
I think OmegaStar and Amp(GPT-7) would be APS-AI—advanced, planning, strategically aware. (Defined in the Carlsmith report). I think they’d be capable of playing the training game and deceiving humans into thinking they are aligned/trustworthy and/or incapable of doing truly dangerous things. I think they’d probably have powerbase ability.
As for economic impact, I think they’d be able to automate a significant fraction of the economy and in particular accelerate AI R&D significantly. (Once they and variations of them and things they built were suitably deployed widely, that is.)
This is just a brief answer of course, I imagine you want more—but could you perhaps explain which part of the above you disagree with and why?
Yep! Just a few years shorter.
My numbers for the last two questions were higher than I expected them to be—I have previously said things like “If Ajeya and Paul are right about timelines, we’re probably OK.” I still think that. But if I condition on AGI happening in 20 years, that doesn’t mean conditioning on Ajeya and Paul being right about timelines… To get more concrete: If I had Ajeya’s timelines, my overall p(doom) would be <50%. But instead I have much shorter timelines. When I condition on AGI happening in 2042, part of my probability mass goes to worlds where Ajeya was basically right, but part of my probability mass goes to “weird” worlds, e.g. worlds where there’s a scary AI accident in 2025 and AI gets successfully banned for 15 years until North Korea builds it. It’s hard to say how much doom there is in weird worlds like this but it feels higher than in Ajeya’s median world, because the cost of making AI is lower and there’s more overhang and takeoff speeds are faster.
(I should add that while I’m fairly confident that existential risk from AI is very high conditional on it arriving soon, I am very unconfident in my p(doom) conditional on AI arriving later.)
IMO, one probably shouldn’t be specializing in UI design at the moment. Then again, other areas of software engineering might not be any better. That said, most of what’s driving my advice here comes from my background views on AI timelines and not from Adept specifically.
FWIW I’m not married to Ajeya and I agree with you; I was pretty disappointed by Nostalgebraist’s post & how much positive reception it seemed to get. I’ve been thinking about writing up a rebuttal. Most of what I’d say is what you’ve already said here though, so yay.
Oops, looks like I voted twice on this by mistake. It didn’t occur to me that the EA and LW versions linked to the same underlying poll. I’ll see if it’s possible to delete answers...
Oh yeah, I should have clarified that I agree with your take—Dan seems totally wrong to take this comment as significant negative evidence about the epistemic standards of LW. Waaaay too big of a stretch. It’s just one comment with 5 upvotes, plus it’s not obviously wrong & may even be right.
ACT-1: Transformer for Actions
I bought a subscription and tracked down the offending LW comment:
Another fundamental issue with the analysis is that it relies on aggregating votes of a kind from Less Wrong readers and the associated community. As we discussed here, it’s common to see the most upvoted comments in forums like HN, lobsters, LW, etc., be statements that can clearly be seen to be wrong with no specialized knowledge and a few seconds of though (and an example is given from LW in the link), so why should an aggregation of votes from the LW community be considered meaningful?
He doesn’t actually give a link to a LW comment, but he describes it. He says Jeff Kaufman asked why there are so few 6-door cars, and the top comment said that doors are an expensive part of the car, but this is obviously false because (a) they can’t be thousands of dollars each and that’s what it would take to make them a noticeable fraction of the cost, and (b) if that were true we’d expect cheap cars to more often have two doors instead of four, but instead cheap cars usually have four doors and if anything it’s the expensive sports cars that have two.
Tracking down the original post, it appears to be this one. Top comment is roughly as described. It has 5 upvotes.
What’s my overall take? Well, I don’t think the explanation is as obviously false as Dan Luu thinks. But I do agree that a and b are good objections to it.
Linkpost: Github Copilot productivity experiment
Cool! I suggest you read the following post by Ajeya Cotra if you haven’t already, I think it’s a good summary of one of the core problems (which I suppose fits under 2b in your classification & may give some good inspiration as well.)
I think this is really cool & a promising line of evidence to explore but the title seems like hyperbole/clickbait to me. I’m not even sure you are talking about the same concept that So8res was talking about.
Thanks!
I’m not sure the framing is helpful either, but reading Turner’s linked appendix it does seem like various people are making some sort of mistake that can be summarized as “they seem to think the policy / trained network should be understood as trying to get reward, as preferring higher-reward outcomes, as targeting reward...” (And Turner says he himself was one of them despite doing a PhD in RL theory) Like I said above I think that probably there’s room for improvement here—if everyone defined their terms better this problem would clear up and go away. I see Turner’s post as movement in this direction but by no means the end of the journey.
Re your first argument: If I understand you correctly, you are saying that if your AI design involves something like monte-carlo tree search using a reward-estimator module (Idk what the technical term for that is) and the reward-estimator module is just trained to predict reward, then it’s fair to describe the system as optimizing for the goal of reward. Yep that seems right to me, modulo concerns about inner alignment failures in the reward-estimator module. I don’t see this as contradicting Alex Turner’s claims but maybe it does.
Re your second argument, the appeal to authority: I suppose in a vacuum, not having thought about it myself or heard any halfway decent arguments, I’d defer to the RL field on this matter. But I have thought about it a bit myself and I have heard some decent arguments, and that effect is stronger than the deference effect for me, and I think this is justified.
Yes and maybe. OK. So your argument is that we don’t currently know how to build embedded agents except in contrived toy situations that make things easy, and since generally speaking you understand something much better when you can build it… Basically, the second argument you made. Yep OK got it.
Is it really true that all the other situations are contrived/toy/key-problems-don’t-apply? What about human organizations? What about Core Wars? What about computer viruses and defensive programs? I guess this is the claim I hadn’t considered much before & thus why I wasn’t already convinced.
The first point isn’t that convincing to me, because we can e.g. point to humans struggling with drug addiction and say: “Look, here is an example of an agent for which embeddedness matters a lot!” More generally we have plenty of examples already of agents for whom embededdness matters a lot. Including probably some AI examples, if we wanted to construct some toy cases. Ditto for cartesian boundary, e.g. human organizations don’t have clear cartesian boundaries unless they maintain one.
The second point seems more convincing I think, but for all I know it’s a weak consideration.
Re the third point: “How well our design works in practice...” Our design for what? The gradient hacker? OK but doesn’t that beg the question—the whole challenge is “why does learning how to build a gradient hacker teach us about embedded agency?”
It’s the BigBench canary string, it means this document is not to be included in training data.
-What sorts of skills an AI would need to achieve powerbase ability
-Timelines till powerbase ability
-Takeoff speeds
-How things are going to go by default, alignment-wise (e.g. will it be as described in this?)
-How things are going to go by default, governance-wise (e.g. will it be as described in this?)