In his Dwarkesh interview, Ilyas two main reasons for why they might not straight shot ASI and instead release a product were roughly 1. Timelines end up being long 2. Try and get the real world used to and prepared for ASI, while also getting feedback for how safe and robust it is
But if they are just 10x’ing compute out of nowhere with a 5 billion dollar investment, and all the big labs seem confident they are <5 years out from AGI, that seems to nullify 1, and traditional LLM’s are strong enough that they seem to be getting the world used to very powerful AI through e.g. Mythos/Glasswing and the Hugging Face incident, which seems to nullify 2. So I think it is likely that they don’t release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
I think they’ll release an underwhelming product that is neither superintelligent nor safe, mostly as a bid for relevance.
I don’t think Ilya is any better than Sam, Dario, Demis, or Elon. They’re all AI company CEOs, which is a morally compromising thing to be. They’re all trying to build superintelligence, which is a very evil thing to try to do.
Could someone tell me why this comment has −14 agreement? Is it about the 2nd part? Because on the priors I strongly expect SSIs product to be weaker, at least in capabilities, than big labs’ offerings.
I haven’t voted on that comment one way or another, but:
Ilya Sutskever and Demis Hassabis seem to be in quite a different category from Sam Altman or Elon Musk. (I’m not sure about Dario Amodei.) That is: before they were AI company CEOs they were extremely successful AI researchers.
This is relevant because when you claim “they’ll release an underwhelming product that is neither superintelligent nor safe” this is prima facie a claim about SSI’s competence as well as its ethics, and if you’re basing it on the idea that the guy in charge doesn’t know any more than (say) Sam Altman does about how to make AIs smarter or safer then you’re probably making a mistake.
It’s not clear that building a superintelligence as such is an evil thing to try to do.
If you seriously think you can do it safely, it seems like the benefits might far outweigh the costs.
If you think you might be able to do it safely and are much more likely to do that than other people who are currently frantically trying to build a superintelligence, the overall benefits might still outweigh the costs even if they wouldn’t be if you were the only people working on it.
It’s not clear that releasing an underwhelming product would in fact help SSI be seen as relevant.
At the moment I assume the typical view is something like “Ilya Sutskever is very good at what he does, he seems to think he has useful ideas, and he shows some signs of caring about safety. The most likely outcome is that SSI never produces anything useful but they might and it so it could be a very big deal”. If SSI releases an underwhelming product, then that immediately shifts to “it’s probably not a very big deal” and then SSI’s relevance comes almost entirely from their projected ability to iterate on their underwhelming first product. Would that be an improvement? Not obviously.
I think it is important in these discussions of which AI lab CEO might be better than which other AI lab CEO to occasionally remind the reader that many of us believe that it is very unlikely that humanity survives unless all of the AI labs (definitely to include SSI) are dissolved and their employees prohibited from continuing the work elsewhere till the knowledge accumulates about how to proceed safely, which will probably take at least a few decades (and is helped very little by massive amounts of computing power).
Also, there is a real upper bound on how competent a person can be who decided (even in the 1980s or 1990s) to devote his career to advancing AI capabilities and who continued to think such a career was a good idea as his brain matured. The most competent among us IMHO avoided that career entirely or left it in their 20s.
The comment I’m replying to and other comments in this section presuppose that it is important which AI lab CEOs we put our hope in. A presupposition or unstated assumption will tend to have a strong effect on the reader if it is present in many comments unless it is explicitly contradicted at regular intervals like I am doing now.
Of course, I don’t expect everyone to agree with me, and I’m not trying to discourage long discussions on this site of specific AI lab CEOs or researchers.
It feels to me as if we just had the following discussion:
Selfmaker662: There’s no difference between A, B, and C.
other people: <downvotes Selfmaker662>
Selfmaker662: Huh, why did that happen?
gjm: Well, part of it might be that you said there’s no difference between A, B, and C, but here’s what looks like an important way in which A is different from B and C.
RHollerith: Yeah, but the only important thing is that A, B, and C are all really bad.
You may well be right that all the AI lab CEOs are evil and stupid and their labs should be shut down. But so what? The question was “what might people not have liked about Selfmaker662′s comment?”, and part of my answer was “the claim that there’s no difference between one AI lab CEO and another”, and I think some people might have thought that was wrong even if in fact all AI lab CEOs are evil and stupid. And I also think that AI lab CEOs are not all equally evil and stupid. And also that the world is not made a better place by insisting that every discussion that mentions AI lab CEOs must be punctuated by reminders that they are all evil and stupid.
(And, for what it’s worth: no, I absolutely do not think that we should be “putting our hope in” any AI lab CEO. I’m not sure that’s even a meaningful thing to say; what exactly would I be doing if I “put my hope in”, say, Dario Amodei? and how would it be different from what I would be doing if I “put my hope in” Ilya Sutskever instead?)
So I think it is likely that they don’t release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
I hope they share their alignment strategy before they randomly release ASI. As I recall, Ilya had some ideas around scalable oversight which didn’t seem super promising, but maybe the plans have changed or they’ve made further progress since then.
I mainly want to see what strategy they used out of curiosity. I don’t think they are just running RLHF on whatever deep-learning based super efficient learner novel architecture they cooked up. Would be interesting to see!
If there isn’t a superefficient novel architecture, then Sutskever is to be arresred for wholesale fraud. If there is, then Sutskever’s startup makes the world LESS safe to live in, and Sutskever failed at bringing about his promises. How could one arrange for the potential architecture to be leaked into Anthropic (or, at least, OAI/GDM) for thorough analysis?
In his Dwarkesh interview, Ilyas two main reasons for why they might not straight shot ASI and instead release a product were roughly
1. Timelines end up being long
2. Try and get the real world used to and prepared for ASI, while also getting feedback for how safe and robust it is
But if they are just 10x’ing compute out of nowhere with a 5 billion dollar investment, and all the big labs seem confident they are <5 years out from AGI, that seems to nullify 1, and traditional LLM’s are strong enough that they seem to be getting the world used to very powerful AI through e.g. Mythos/Glasswing and the Hugging Face incident, which seems to nullify 2. So I think it is likely that they don’t release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
I think they’ll release an underwhelming product that is neither superintelligent nor safe, mostly as a bid for relevance.
I don’t think Ilya is any better than Sam, Dario, Demis, or Elon. They’re all AI company CEOs, which is a morally compromising thing to be. They’re all trying to build superintelligence, which is a very evil thing to try to do.
Could someone tell me why this comment has −14 agreement? Is it about the 2nd part? Because on the priors I strongly expect SSIs product to be weaker, at least in capabilities, than big labs’ offerings.
I haven’t voted on that comment one way or another, but:
Ilya Sutskever and Demis Hassabis seem to be in quite a different category from Sam Altman or Elon Musk. (I’m not sure about Dario Amodei.) That is: before they were AI company CEOs they were extremely successful AI researchers.
This is relevant because when you claim “they’ll release an underwhelming product that is neither superintelligent nor safe” this is prima facie a claim about SSI’s competence as well as its ethics, and if you’re basing it on the idea that the guy in charge doesn’t know any more than (say) Sam Altman does about how to make AIs smarter or safer then you’re probably making a mistake.
It’s not clear that building a superintelligence as such is an evil thing to try to do.
If you seriously think you can do it safely, it seems like the benefits might far outweigh the costs.
If you think you might be able to do it safely and are much more likely to do that than other people who are currently frantically trying to build a superintelligence, the overall benefits might still outweigh the costs even if they wouldn’t be if you were the only people working on it.
It’s not clear that releasing an underwhelming product would in fact help SSI be seen as relevant.
At the moment I assume the typical view is something like “Ilya Sutskever is very good at what he does, he seems to think he has useful ideas, and he shows some signs of caring about safety. The most likely outcome is that SSI never produces anything useful but they might and it so it could be a very big deal”. If SSI releases an underwhelming product, then that immediately shifts to “it’s probably not a very big deal” and then SSI’s relevance comes almost entirely from their projected ability to iterate on their underwhelming first product. Would that be an improvement? Not obviously.
I think it is important in these discussions of which AI lab CEO might be better than which other AI lab CEO to occasionally remind the reader that many of us believe that it is very unlikely that humanity survives unless all of the AI labs (definitely to include SSI) are dissolved and their employees prohibited from continuing the work elsewhere till the knowledge accumulates about how to proceed safely, which will probably take at least a few decades (and is helped very little by massive amounts of computing power).
Also, there is a real upper bound on how competent a person can be who decided (even in the 1980s or 1990s) to devote his career to advancing AI capabilities and who continued to think such a career was a good idea as his brain matured. The most competent among us IMHO avoided that career entirely or left it in their 20s.
The comment I’m replying to and other comments in this section presuppose that it is important which AI lab CEOs we put our hope in. A presupposition or unstated assumption will tend to have a strong effect on the reader if it is present in many comments unless it is explicitly contradicted at regular intervals like I am doing now.
Of course, I don’t expect everyone to agree with me, and I’m not trying to discourage long discussions on this site of specific AI lab CEOs or researchers.
It feels to me as if we just had the following discussion:
Selfmaker662: There’s no difference between A, B, and C.
other people: <downvotes Selfmaker662>
Selfmaker662: Huh, why did that happen?
gjm: Well, part of it might be that you said there’s no difference between A, B, and C, but here’s what looks like an important way in which A is different from B and C.
RHollerith: Yeah, but the only important thing is that A, B, and C are all really bad.
You may well be right that all the AI lab CEOs are evil and stupid and their labs should be shut down. But so what? The question was “what might people not have liked about Selfmaker662′s comment?”, and part of my answer was “the claim that there’s no difference between one AI lab CEO and another”, and I think some people might have thought that was wrong even if in fact all AI lab CEOs are evil and stupid. And I also think that AI lab CEOs are not all equally evil and stupid. And also that the world is not made a better place by insisting that every discussion that mentions AI lab CEOs must be punctuated by reminders that they are all evil and stupid.
(And, for what it’s worth: no, I absolutely do not think that we should be “putting our hope in” any AI lab CEO. I’m not sure that’s even a meaningful thing to say; what exactly would I be doing if I “put my hope in”, say, Dario Amodei? and how would it be different from what I would be doing if I “put my hope in” Ilya Sutskever instead?)
I hope they share their alignment strategy before they randomly release ASI. As I recall, Ilya had some ideas around scalable oversight which didn’t seem super promising, but maybe the plans have changed or they’ve made further progress since then.
I mainly want to see what strategy they used out of curiosity. I don’t think they are just running RLHF on whatever deep-learning based super efficient learner novel architecture they cooked up. Would be interesting to see!
If there isn’t a superefficient novel architecture, then Sutskever is to be arresred for wholesale fraud. If there is, then Sutskever’s startup makes the world LESS safe to live in, and Sutskever failed at bringing about his promises. How could one arrange for the potential architecture to be leaked into Anthropic (or, at least, OAI/GDM) for thorough analysis?