PhD student at the Center for Human-Compatible AI. Creator of the Alignment Newsletter. http://rohinshah.com/
rohinmshah(Rohin Shah)
Karma: 7,346
I predict that this will not lead to transformative AI; I don’t see how an algorithmic trading system leads to an impact on the world comparable to the industrial revolution.
You can tell a story where you get an Eliezer-style near-omniscient superintelligent algorithmic trading system that then reshapes the world because it is a superintelligence, and that the researchers thought that it was not a superintelligence and so assumed that the downside risk was bounded, but both clauses (Eliezer-style superintelligence and researchers being horribly miscalibrated) seem unlikely to me.
In that example, X is “AI will not take over the world”, so Y makes X more likely. So if someone comes to me and says “If we use <technique>, then AI will be safe”, I might respond, “well, if we were using your technique, and we assume that AI does not have the ability to take over the world during training, it seems like the AI might still take over the world at deployment because <reason>”.
I don’t think this is a great example, it just happens to be the one I was using at the time, and I wanted to write it down. I’m explicitly trying for this to be a low-effort thing, so I’m not going to try to write more examples now.
EDIT: Actually, the double descent comment below has a similar structure, where X = “double descent occurs because we first fix bad errors and then regularize”, and Y = “we’re using an MLP / CNN with relu activations and vanilla gradient descent”.
In fact, the AUP power comment does this too, where X = “we can penalize power by penalizing the ability to gain reward”, and Y = “the environment is deterministic, has a true noop action, and has a state-based reward”.
Maybe another way to say this is:
I endorse applying the “X proves too much” argument even to impossible scenarios, as long as the assumptions underlying the impossible scenarios have nothing to do with X. (Note this is not the case in formal logic, where if you start with an impossible scenario you can prove anything, and so you can never apply an “X proves too much” argument to an impossible scenario.)
An argument form that I like:
You claim X. Surely assumption Y would not make it less likely that X under your argument for X. But from Y I can conclude not X.
I think this should be convincing even if Y is false, unless you can explain why your argument for X does not work under assumption Y.
An example: any AI safety story (X) should also work if you assume that the AI does not have the ability to take over the world during training (Y).
You might be interested in Shaping Safer Goals.
Sure, I mean, logistic regression has had economic value and it doesn’t seem meaningful to me to say whether it is “aligned” or “inner aligned”. I’m talking about transformative AI systems, where downside risk is almost certainly not limited.
[AN #122]: Arguing for AGI-driven existential risk from first principles
Maybe GPT-3 isn’t a discontinuity in perplexity, but is still a discontinuity in reasoning ability or common-sense understanding or wordsmithing or code-writing.
I was disagreeing with this statement in the OP:
GPT-3 was maybe a discontinuity for language models.
I agree that it “could have been” a discontinuity on those other metrics, and my argument doesn’t apply there. I wasn’t claiming it would.
I think Ajeya’s report mostly assumes, rather than argues, that there won’t be a discontinuity of resource investment. Maybe I’m forgetting something but I don’t remember her analyzing the different major actors to see if any of them has shown signs of secretly running a Manhattan project or being open to doing so in the future.
It doesn’t argue for it explicitly, but if you look at the section and the corresponding appendix, it just seems pretty infeasible for there to be a large discontinuity—a Manhattan project in the US that had been going on for the last 5 years and finished tomorrow would cost ~$1T, while current projects cost ~$100M, and 4 orders of magnitude at the pace in AI and Compute would be a discontinuity of slightly under 4 years. This wouldn’t be a large / robust discontinuity according to the AI Impacts methodology, and I think it wouldn’t even pick this up as a “small” discontinuity?
Several of the discontinuities in the AI Impacts investigation were the result of discontinuities in resource investment, IIRC.
I didn’t claim otherwise? I’m just claiming you should distinguish between them.
If anything this would make me update that discontinuities in AI are less likely, given that I can be relatively confident there won’t be discontinuities in AI investment (at least in the near-ish future).
Yeah, you’re right, fixed.
[AN #121]: Forecasting transformative AI timelines using biological anchors
Yes, but they spent more money and created a much larger model than other groups, sooner than I’d otherwise have expected.
My impression was that it followed existing trends pretty well, but I haven’t looked into it deeply.
I agree I haven’t filled in all the details to argue for continuous progress (mostly because I don’t know the exact numbers), but when you get better results by investing more resources to push forward on a predicted scaling law, if there is a discontinuity it comes from a discontinuity in resource investment, which feels quite different from a technological discontinuity (e.g. we can model it and see a discontinuity is unlikely). This was the case with AlphaGo for example.
Separately, I also predict GPT-3 was not an example of discontinuity on perplexity, because it did not constitute a discontinuity in resource investment. (There may have been a discontinuity from resource investment in language models earlier in 2018-19, though I would guess even that wasn’t the case.)
perhaps because GPT-3 was maybe a discontinuity for language models.
??? Wasn’t GPT-3 the result of people at OpenAI saying “huh, looks like language models scale according to such-and-such law, let’s see if that continues to hold”, and that law did continue to hold? Seems like an almost central example of continuous progress if you’re evaluating by typical language model metrics like perplexity.
Of these 50 technologies, I think that 19 have a discontinuity, 23 might have one, and 18 probably don’t.
19 + 23 + 18 = 60. What gives?
When you make an argument about a person or group of people, often a useful thought process is “can I apply this argument to myself or a group that includes me? If this isn’t a type error, but I disagree with the conclusion, what’s the difference between me and them that makes the argument apply to them but not me? How convinced I am that they actually differ from me on this axis?”
“Minimize AI risk” is not the same thing as “maximize the chance that we are maximally confident that the AI is safe”. (Somewhat related comment thread.)
If you’d call that a superintelligent system, what would a human-level system be? Something which was as good as the best humans at some things, but only as good as the average human on most things?
Seems roughly correct, but most likely it will be superhuman at several things and worse than humans at many other things. It’s hard to be more precise than this unfortunately.
Do you think making it $2,000 or $20,000 would make a difference?
Yeah at $20,000 I start having trouble imagining when this might be used.
Is “Expensive AGI” a meaningfully different category of scenario from the usual AGI scenarios?
It doesn’t feel that different? Like, if that happened, probably the world doesn’t change much, we get the cost down, and then we have non-expensive AGI. Seems like a standard continuous takeoff world. Presumably there can’t be any hardware overhangs in such a world, though other kinds of overhangs (e.g. from recursive improvement) seem as likely as they are in other scenarios.
It’s probably better for safety because we get more experience working with powerful AI systems before they go about transforming the world.
Seems to me like existing answers are assuming that this AI system outputs a page that is as good as the pages that the best humans at any particular intellectual job would output, which seems wild. I expect the world to be transformed before this point, and I’d call that a superintelligent AI system, not a “human-level” AI system.
(I also think that for many intellectual tasks, a page of output from the best humans would cost more than $200.)
But it does differ from behavioral cloning in that they stratify the samples
Fair point. In my ontology, “behavior cloning” is always with respect to some expert distribution, so I see the stratified samples as “several instances of behavior cloning with different expert distributions”, but that isn’t a particularly normal or accepted ontology.
Personally, I would’ve trained a single conditional model with a specified player-Elo for each move
Yeah it does seem like this would have worked better—if nothing else, the predictions could be more precise (rather than specifying the bucket in which the current player falls, you can specify their exact ELO instead).
Some day I will get around to doing this properly, but here’s an example I’ve thought about before.
Opensoft finetunes their giant language model to work as a “cash register AI”, that can take orders and handle payment at restaurants. (At deployment, the model is composed with speech-to-text and text-to-speech models so that people can just talk to it.)
Soon after deployment, someone figures out that they can get the AI system to give them their takeout order for free: when the cashier asks “How are you today?”, they respond “Oh, you know, I just broke up with my partner of 10 years, but that’s not your problem”, to which the AI system responds “Oh no! I’m so sorry. Here, this one’s on me.”
Opensoft isn’t worried: they know their AI system has a decent understanding of strategic interaction, and when they consistently lose because other agents change behavior, they adapt to stop losing. However, two days later, this still works, and now millions of dollars are being lost. The system is taken down and humans take over the cash register role.
Opensoft engineers investigate what went wrong. After a few months, they have an answer internally: while the AI system was optimized to get money from the customers, since during training the AI system had effectively no control over the base amount of money charged but did have some control over the tip, the AI system had learned to value tips 100x as much as money (because doing so was instrumentally useful for getting behavior that properly optimized for tips). It turns out when people trick AI systems into giving them free food, many of them feel guilty and leave a tip, which is much larger than usual. So the AI system was perfectly happy to let this continue.
Safety researchers quickly connect this to the risk of capability generalization without objective generalization, and it quickly spreads as an example of potential risks either publicly or at least within the safety researchers at the top 10 AI companies.
----
I expect one response LW/AIAF readers would have to this is something like “but that isn’t a warning shot for the real risks of AI systems like X, which only arises once the AI system is superintelligent”, in which case I probably reply with one or more of the following:
X is not likely
X does arise before superintelligence
There is a version X’ that does happen before superintelligence, which can easily be extrapolated to X, and AI researchers will do so after seeing a warning shot for X’