Yeah, I know they don’t understand them comprehensively. Is this the point though? I mean they understand them at a level of abstraction necessary to do what they need, and the claim is they have basically the same kind of knowledge of computers. Hmm, I guess that isn’t really communicated by my phrasing though, so maybe I should edit that
Chris van Merwijk
Karma: 720
I think I communicated unclearly and it’s my fault, sorry for that: I shouldn’t have used the phrase “any easily specifiable task” for what I meant, because I didn’t mean it to include “optimize the entire human lightcone w.r.t. human values”. In fact, I was being vague and probably there isn’t really a sensible notion that I was trying to point to. However, to clarify what I really was trying to say: What I mean by “hard problem of alignment” is : “develop an AI system that keeps humanity permanently safe from misaligned AI (and maybe other x risks), and otherwise leaves humanity to figure out what it wants and do what it wants without restricting it in much of any way except some relatively small volume of behaviour around ‘things that cause existential catastrophe’ ” (maybe this ends up being to develop a second version AI that then gets free reign to optimize the universe w.r.t. human values, but I’m a bit skeptical). I agree that “solve all of human psychology and moral …” is significantly harder than that (as a technical problem). (maybe I’d call this the “even harder problem”).
Ehh, maybe I am changing my mind and also agree that even what I’m calling the hard problem is significantly more difficult than the pivotal act you’re describing, if you can really do it without modelling humans, by going to mars and doing WBE. But then still the whole thing would have to rely on the WBE, and I find it implausible to do it without it (currently, but you’ve been updating me about lack of need of human modelling so maybe I’ll update here too). Basically the pivotal act is very badly described as merely “melt the gpus”, and is much more crazy than what I thought it was meant to refer to.
Regarding “rogue”: I just looked up the meaning and I thought it meant “independent from established authority”, but it seems to mean “cheating/dishonest/mischievous”, so I take back that statement about rogueness.
I’ll respond to the “public opinion” thing later.
I’m surprised if I haven’t made this clear yet, but the thing that (from my perspective) seems different between my and your view is not that Step 1 seems easier to me than it seems to you, but that the “melt the GPUs” strategy (and possibly other pivotal acts one might come up with) seems way harder to me than it seems to you. You don’t have to convince me of “‘any easily human-specifiable task’ is asking for a really mature alignment”, because in my model this is basically equivalent to fully solving the hard problem of AI alignment.
Some reasons:
I don’t see how you can do “melt the GPUs” without having an AI that models humans. What if a government decides to send a black ops team to kill this new terrorist organization (your alignment research team), or send a bunch of icbms at your research lab, or do any of a handful of other violent things? Surely the AI needs to understand humans to a significant degree? Maybe you think we can intentionally restrict the AI’s model of humans to be only about precisely those abstractions that this alignment team considers safe and covers all the human-generated threat models such as “a black ops team comes to kill your alignment team” (e.g. the abstraction of a human as a soldier with a gun).
What if global public opinion among scientists turns against you and all ideas about “AI alignment” are from now on considered to be megalomaniacal crackpottery? Maybe part of your alignment team even has this reaction after the event, so now you’re working with a small handful of people on alignment and the world is against you, and you’ve semi-premanently destroyed any opportunity that outside researchers can effectively collaborate on alignment research. Probably your team will fail to solve alignment by themselves. It seems to me this effect alone could be enough to make the whole plan predictably backfire. You must have thought of this effect before, so maybe you consider it to be unlikely enough to take the risk, or maybe you think it doesn’t matter somehow? To me it seems almost inevitable, and could only be prevented with basically a level of secrecy and propaganda that would require your AI to model humans anyway.
These two things alone make me think that this plan doesn’t work in practice in the real world, unless you basically solve Step 1 already. Although I must say the point which I just speculated you might have, that we could somehow control the AI’s model of humans to be restricted to particular abstractions, gives me some pause and maybe I end up being wrong via something like that. This doesn’t affect the second bullet point though.
Reminder to the reader: This whole discussion is about a thought experiment that neither party actually seriously proposed as a realistic option. I want to mention this because lines might be taken out of context to give the impression that we are actually discussing whether to do this, which we aren’t.
“you” obviously is whoever would be building the AI system that ended up burning all the GPU’s (and ensuring no future GPU’s are created). I don’t know such sequence of events just as I don’t know the sequence of events for building the “burn all GPU’s” system, except at the level of granularity of “Step 1. build a superintelligent AI system that can perform basically any easily human-specifiable task without destroying the world. Step 2. make that system burn all GPU’s indefintely/build security services that prevent misaligned AI from destroying the world”.
I basically meant to say that I don’t know that “burn all the GPU’s” isn’t already as difficult as building the security services, because they both require step 1, which is basically all of the problem (with the caveat that I’m not sure, and made an edit stating a reason why it might be far from true). I basically don’t see how you execute the “burn all gpu’s” strategy without basically solving almost the entire problem.
I wonder if there is a bias induced by writing this on a year-by-year basis, as opposed to some random other time interval, like 2 years. I can somehow imagine that if you take 2 copies of a human, and ask one to do this exercise in yearly intervals, and the other to do it in 2-year intervals, they’ll basically tell the same story, but the second one’s story takes twice as long. (i.e. the second one’s prediction for 2022/2024/2026 are the same as the first one’s predictions for 2022/2023/2024). It’s probably not that extreme, but I would be surprised if there was zero such effect, which would mean these timelines are biased downwards or upwards.
yeah, I probably overstated. Nevertheless:
“CEV seems way harder to me than …”
yes, I agree it seems way harder, and I’m assuming we won’t need to do it and that we could instead “run CEV” by just actually continuing human society and having humans figure out what they want, etc. It currently seems to me that the end game is to get to an AI security service (in analogy to state security services) that protects the world from misaligned AI, and then let humanity figure out what it wants (CEV). The default is just to do CEV directly by actual human brains, but we could instead use AI, but once you’re making that choice you’ve already won. i.e. the victory condition is having a permanent defense against misaligned AI using some AI-nanotech security service, how you do CEV after that is a luxury problem. My point about your further clarification of the “melt all the GPU’s option is that it seemed to me (upon first thinking about it), that once you are able to do that, you can basically instead just make this permanent security service. (This is what I meant by “the whole alignment problem”, but I shouldn’t have put it that way). I’m not confident though, because it might be that such a security service is in fact much harder due to having to constantly monitor software for misaligned AI.Summary: My original interpretation of “melt the GPUs” was that it buys us a bit of extra time, but now I’m thinking it might be so involved and hard that if you can do that safely, you almost immediately can just create AI security services to permanently defend against misaligned AI (which seems to me to be the victory condition). (But not confident, I haven’t thought about it much).
Part of my intuition is, in order to create such a system safely, you have to (in practice, not literally logically necessary) be able to monitor an AI system for misalignment (in order to make sure your GPU melter doesn’t kill everyone), and do fully general scientific research. EDIT: maybe this doesn’t need you to do worst-case monitoring of misalignment though, so maybe that is what makes a GPU melter easier than fully general AI security services....
Ok I admit I read over it. I must say though that this makes the whole thing more involved than it sounded at fist, since it would maybe require essentially escalating a conflict with all major military powers and still coming out on top? One possible outcome of this would be that the entire global intellectual public opinion turns against you, meaning you also possibly lose access to a lot of additional humans working with you on further alignment research? I’m not sure if I’m imagining it correctly, but it seems like this plan would either require so many elements that I’m not sure if it isn’t just equivalent to solving the entire alignment problem, or otherwise it isn’t actually enough.
But assuming that law enforcement figures out that you did this, then puts you in jail, you wouldn’t be able to control the further use of such nanotech, i.e. there would just be a bunch of systems indefinitely destroying GPU’s, or maybe you set a timer or some conditions on it or something. I certainly see no reason why Iceland or anyone in iceland could get away with this unless those systems rely on completely unchecked nanosystems to which the US military has no response. Maybe all of this is what Eliezer means by “melt the GPU’s”, but I thought he did just mean “melt the GPU’s as a single act” (not weird that I thought this, given the phrasing “the pivotal act to melt all the GPU’s”). If this is what is meant, then it would be a strong enough pivotal act, and would be an extreme level of capability I agree.
Just wanna remind the reader that Eliezer isn’t actually proposing to do this, and I am not seriously discussing it as an option and nor was Eliezer (nor would I support it unless done legally), just thinking through a thought experiment.
I meant, is there a link to where you’ve written this down somewhere? Maybe you just haven’t written it down.
I would be interested in reading a draft and giving feedback (FYI I’m currently a researcher in the AI safety team at FHI).
I’m also interested to read the draft, if you’re willing to send it to me.
Here is my partial honest reaction, just two points I’m somewhat dissatisfied with (not meant to be exhaustive):
2. “A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not find it difficult to bootstrap to overpowering capabilities independent of human infrastructure.” I would like there to be an argument for this claim that doesn’t rely on nanotech, and solidly relies on actually existing amounts of compute. E.g. if the argument relies on running intractable detailed simulations of proteins, then it doesn’t count. (I’m not disagreeing with the nanotech example by the way, or saying that it relies on unrealistic amounts of compute, I’d just like to have an argument for this that is very solid and minimally reliant on speculative technology, and actually shows that it is).
6. “We need to align the performance of some large task, a ‘pivotal act’ that prevents other people from building an unaligned AGI that destroys the world.”. You name “burn all GPU’s” as an “overestimate for the rough power level of what you’d have to do”, but it seems to me that it would be too weak of a pivotal act? Assuming there isn’t some extreme change in generally held views, people would consider this an extreme act of terrorism, and shut you down, put you in jail, and then rebuild the GPU’s and go on with what they were planning to do. Moreover, now there is probably an extreme taboo on anything AI safety related. (I’m assuming here that law enforcement finds out that you were the one who did this). Maybe the idea is to burn all GPU’s indefinitely and forever (i.e. leave nanobots that continually check for GPU’s and burn them when they are created), but even this seems either insufficient or undesirable long term depending on what is counted as a GPU. Possibly I’m not getting what you mean, but it just seems completely too weak as an act.
“I have sat down to make toy models ..”
reference?
“which is to make a truly remarkable universal claim with a heavy burden of proof.”
Having thought about this way less than you, it doesn’t seem at first sight to me as remarkable as you seem to say. Note that the claim wouldn’t be that you can’t write a set of prompts to get the fully unversal reasoner, but that you can’t write a single prompt that gets you this universal reasoner. It doesn’t sound so crazy to me at all that knowledge is dispersed in the network in a way that e.g. some knowledge can only be accessed if the prompt has the feel of being generated by an american gun rights activist, or something similar. By the way, here we generate a few alternative hypotheses here.
“In order for both of the points to be true, that is equivalent to claiming that it cannot tap into the full pool under all possible conditions”
I might be misunderstanding, but it seems like this is the opposite of both my implication 1 and 2? implication 1 is that it can tap into this, in sufficiently out-of-distribution contexts. implication 2 is that with fine tuning you can make it tap into fairly quickly in specific contexts. EDIT: oh maybe you simply made a typo and meant to say “to be false”.
By the way we write some alternative hypotheses here. All of this is based on probably less than 1 hour of thinking.
Responding to this very late, but: If I recall correctly, Eric has told me in personal conversation that CAIS is a form of AGI, just not agent-like AGI. I suspect Eric would agree broadly with Richard’s definition.
“I talk about consequentialists, but not rational consequentialists”, ok this was not the impression I was getting.
Reading this post a while after it was written: I’m not going to respond to the main claim (which seems quite likely) but just to the specific arguments, which seems suspicious to me. Here are some points:
In my model of the standard debate setup with human judge, the human can just use both answers in whichever way it wants, independently of which it selects as the correct answer. The fact that one answer provides more useful information than “2+2=?” doesn’t imply a “direct” incentive for the human judge to select that as the correct answer. Upon introspection, I myself would probably say that “4” is the correct answer, while still being very interested in the other answer (the answer on AI risk). I don’t think you disagreed with this?
At a later point you say that the real reason for why the judge would nevertheless select the QIA as the correct answer is that the judge wants to train the system to do useful things. You seem to say that a rational consequentialist would make this decision. Then at a later point you say that this is probably/plausibly (?) a bad thing: “Is this definitely undesirable? I’m not sure, but probably”. But if it really is a bad thing and we can know this, then surely a rational judge would know this, and could just decide not to do it? If you were the judge, would you select the QIA, despite it being “probably undesirable”?
Given that we are talking about optimal play and the human judge is in fact not rational/safe, the debater could manipulate the judge, and so the previous argument doesn’t in fact imply that judges won’t select QIA’s. The debater could deceive and manipulate the judge into (incorrectly) thinking that it should select the QIA, even if you/we currently believe that this would be bad. I agree this kind of deception would probably happen in optimal play (if that is indeed what you meant), but it relies on the judge being irrational or manipulable, not on some argument that “it is rational for a consequentialist judge to select answers with the highest information value”.
It seems to me that either we think there is no problem with selecting QIA’s as answers, or we think that human judges will be irrational and manipulated, but I don’t see the justification in this post for saying “rational consequentialist judges will select QIA’s AND this is probably bad”.
yes, but I think your reasoning “If 2 is only talking about the map, it doesn’t imply 3” is too vague. I’d rather not go into it though, because I am currently busy with other things, so I’d suggest letting the reader decide.
Edit: reading back my response, it might come accross as a bit rude. If so, sorry for that, I didn’t mean it that way.
I think this is too vague, but I will drop this discussion and let the reader decide.
I just had a very quick look at that site, and it seems to be a collection of various chip models with pictures of them? Is there actual information on quantities sold, etc? I couldn’t find it immediately.