The obvious answer to why it’s not transparently been a high priority is:
It’s not clear specifically how it helps.
Until we’re clear on a path to impact, it seems more valuable to figure out in more depth what to aim for, than to go and do something ill-thought-through quickly.
Increasing awareness is not reversible, and there are obvious failure modes (e.g. via politicisation).
Public pressure is a blunt tool, and it’s not clear how it could best be leveraged.
Supposing we actually need to hit a precise target, it’s highly unlikely that public pressure is going to be pushing us towards that target (even if we knew the direction we wanted to push, which we currently do not).
Even for something simple such as “slow AGI development down”, the devil is in the detail: slow what down, where, how?
That said, it’s nonetheless highly unlikely that the status-quo/default situation is the ideal—so it does seem probable that some form of broader communication is desirable.
But it’s usually much easier to make things worse than to make them better—so I’d want most effort to go into figuring out the mechanics of a potentially useful plan. Most interventions don’t achieve what was intended—even the plausibly helpful ones (for instance, one might try recruiting John Carmack to work on AI safety [this strikes me as a good idea, hindsight notwithstanding], only to get him interested enough that he starts up an AGI company a few years later).
I understand that it must be frustrating for anyone who takes this very seriously and believes they cannot engage in research themselves (though I’d suggest asking how they can help). However, the goal needs to be to actually improve the situation, rather than to do something that feels good. The latter impulse is much simpler to satisfy than the former.
To find a communication strategy that actually improves the situation, we need careful analysis.
With that in mind, if you want to start useful discussion on this, I’d suggest outlining:
Some plan (in broad terms).
The mechanisms by which you’d expect it to help.
Ways in which it could fail.
Reasons to think it won’t backfire relative to the status-quo.
Reasons to think it won’t render superior future plans impossible/impractical (relative to the status-quo).
I don’t think this is easy (I have no good plan). I do think it’s required.
Increasing awareness increases resources through virtue of sheer volume. The more people hear about AI safety, the more likely someone resourceful and amenable hears about AI safety.
This is a good sentiment, but ‘resource gathering’ is an instrumentally convergent strategy. No matter what researchers end up deciding we should do, it’ll probably be best done with money and status on our side.
Politicisation is not a failure mode, it’s an optimistic outcome. Politicized issues get money. Politicized issues get studied. Other failure modes might be that we increase interest in AI in general and result in destructive AI being generated more rapidly, but there’s already a massive profit motive in that direction, so I don’t know if we can really contribute in that direction. Most other ‘failure modes’ which involve ‘bad publicity’ are infinitely preferable to the current state of affairs, given you have enough dignity to be shameless.
‘Public pressure’ isn’t really a thing as far as I can tell. The public presses back. Money talks. Wheels turn. Equilibria equilibriate or something. I’m only talking about awareness.
My current plan relies on identifying a good representative for AI safety with enough clout to be taken seriously, contacting them, and then trying to get them into public discussions with rationalist-adjacent e-celebs.
I’d expect this to increase awareness for AI safety and make people who see this content more amenable to advocacy for AI safety in the future. I’d expect a minority of people who watch this content to become very interested in AI safety and try and learn more.
Assuming that I find a good advocate and succeed in getting them into a discussion with a minor celebrity, this could go wrong in the following ways:
The advocates comes across as unhinged
The advocates comes across as unlikable
The advocate cannot explain AI safety well
The advocate cannot respond to criticism well
As far as I can see, the efficacy and reliability of this plan relies entirely on the character of the advocate. Because this is being tested in a smaller corner of the internet, I think we can believe that if inexplicably results in disaster, the effect will be relatively contained, but honestly I think a pretty small amount of screening can prevent the worst of this.
The largest issue with this approach/view is that it’s not addressing the distinction between:
Increased resources for things with “AI Safety” written on them.
Increased resources for approaches that stand a chance of working.
The problem is important in large measure due to its difficulty: that we need to hit a very small target we don’t yet understand. By default, resources allocated to anything labelled “AI safety” will not be aimed at that target.
If things are politicised, it’s a safe bet they won’t be aimed at the target; politicised issues get money thrown in their general direction, but that’s not about actually solving the problem. There’s a big difference between [more money helps, all else being equal], and claiming [action x gets more money, therefore it helps]. Politicisation would have many downsides.
Likewise, even if we get more attention/money… there are potential signal-to-noise issues. Suppose that there are 20 people involved in grant allocation with enough technical understanding to pick out promising projects. Consider two cases:
They receive 200 grant applications, 20 of which are promising.
They receive 2000 grant applications, 40 of which are promising.
In case (2) there are more promising projects, but it’s not clear that grant evaluators will find more promising projects, since the signal to noise ratio will have dropped so much.
The obvious answer is to train more grant evaluators to the point where they have the necessary expertise—but this is a slow process that’s (currently) difficult to scale (though people are working on that).
You also seem to be cherry-picking the upside possibilities from increased awareness: yes, some people may start to work on or advocate for AI safety (and some small proportion of those for some useful understanding of “AI safety”). However, some people may also:
Hear about AI safety, realise that AGI is a big deal but not buy the safety arguments, and start working on AGI.
This is not a hypothetical situation, or something that only happens to people without much ability: if John Carmack can get this badly wrong, where are you getting your confidence that most people won’t?
Realise that AGI is a big deal, think that the major issues are misuse and/or ethics, and make poor decisions on that basis.
Make sure weget AGI before them...
Put in regulations that focus the ‘safety’ resources of AI companies on ticking meaningless boxes that do nothing to mitigate x-risk. (though I’d guess the default situation looks largely like this anyway)
We need to argue that it’s net positive, not simply that there would be some positive outcomes (I don’t think anyone would argue with that).
Again, I do think that there’s some communication strategy we should be using that beats the status-quo. However, it needs to be analysed carefully, and carried out carefully—with adjustment based on empirical feedback where possible. (my guess is that the best approaches would be highly targeted—not that this says much at all)
for instance, one might try recruiting John Carmack to work on AI safety [this strikes me as a good idea, hindsight notwithstanding], only to get him interested enough that he starts up an AGI company a few years later
Is this a reference to his current personal project to work on AGI?
Edit: reading a bit more about him, I suspect if he ever got interested in alignment work he’d likely prefer working on Christiano-style stuff than MIRI-style stuff. For instance (re: metaverse):
The idea of the metaverse, Carmack says, can be “a honeypot trap for ‘architecture astronauts.’” Those are the programmers and designers who “want to only look at things from the very highest levels,” he said, while skipping the “nuts and bolts details” of how these things actually work.
These so-called architecture astronauts, Carmack said, “want to talk in high abstract terms about how we’ll have generic objects that can contain other objects that can have references to these and entitlements to that, and we can pass control from one to the other.” That kind of high-level hand-waving makes Carmack “just want to tear [his] hair out… because that’s just so not the things that are actually important when you’re building something.”
Carmack used his own experience creating Doom as an example of the value of concrete, product-based thinking. Rather than simply writing abstract game engines, he wrote games where “some of the technology… turned out to be reusable enough to be applied to other things,” he said. “But it was always driven by the technology itself, and the technology was what enabled the product and then almost accidentally enabled some other things after it.”
Building pure infrastructure and focusing on the “future-proofing and planning for broad generalizations of things,” on the other hand, risks “making it harder to do the things that you’re trying to do today in the name of things you hope to do tomorrow, and [then] it’s not actually there or doesn’t actually work right when you get around to wanting to do that,” he said.
He does seem well-informed (and no doubt he’s a very smart guy), so I do still hope that he might update pretty radically given suitable evidence. Nonetheless, if he stays on his present course greater awareness seems to have been negative (this could absolutely change).
The tl;dr of his current position is:
Doesn’t expect a fast takeoff.
Doesn’t specify a probability or timescale—unclear whether he’s saying e.g. a 2-year takeoff seems implausible; pretty clear he finds a 2-week takeoff implausible.
We should work on ethics/safety of AGI once we have a clearer idea what it looks like. (expects we’ll have time, due to 1)
Not really dismissive of current AGI safety efforts, but wouldn’t put his money there at present, since it’s unclear what can be achieved.
My take:
On 1a, his argument seems to miss the possibility that an AGI doesn’t need to copy itself to other hardware (he has reasonable points that suggest this would be impractical), but might write/train systems that can operate with very different hardware.
If we expect smooth progress, we wouldn’t expect the first system to have the capability to do this—though it may be able to bide its time (potentially requiring gradient-hacking).
However, he expects that we need a small number of key insights for AGI (seems right to me). This is hard to square with an expectation of smooth progress.
On 2, he seems overconfident on the ease of solving the safety issues once we have a clearer idea what it looks like. I guess he’s thinking that this looks broadly similar to engineering problems we’ve handled in the past, which seems wrong to me. Everything we’ve built is narrow, so we’ve never needed to solve the “point this at exactly the thing we mean” problem (capabilities didn’t generalise for any system we’ve made safe).
On 3, I’m pretty encouraged. He seems to be thinking about things reasonably, and isn’t casually dismissing other points of view. I think his current model of the situation is incorrect (as, I’m sure is mine!), but expect that he’d be keen to change direction pretty radically if his model did change.
Agreed that he’d be a much better fit for Christiano-style stuff than MIRI-style—though I also expect he could fit well with Anthropic/Redwood/Conjecture; essentially anything with an empirical slant to it. In an ideal world, he’d be a good fit for Carmack-style stuff.
I do think it’s worthwhile to consider how to engage with him further—though this is still a case where I’d want people thinking carefully about how best to communicate before acting (and whether they’re the best messenger; I quickly conclude that I am not).
So hilariously enough it looks like Carmack got into AGI 4 years ago because Sam Altman tried to recruit him for OpenAI but at the time Carmack knew barely any ML. (I just found the Lex Fidman interview).
The update you are looking for should probably flow in the other direction. Carmack is making an explicit bet on a more brain-like architecture. If he succeeds in his goal of proving out child level AGI raised in VR in 5 years, the safety community should throw out most of their now irrelevant work and focus everything on how to make said new AGI models raised in controlled VR environments safe.
There are people looking at this.
The obvious answer to why it’s not transparently been a high priority is:
It’s not clear specifically how it helps.
Until we’re clear on a path to impact, it seems more valuable to figure out in more depth what to aim for, than to go and do something ill-thought-through quickly.
Increasing awareness is not reversible, and there are obvious failure modes (e.g. via politicisation).
Public pressure is a blunt tool, and it’s not clear how it could best be leveraged.
Supposing we actually need to hit a precise target, it’s highly unlikely that public pressure is going to be pushing us towards that target (even if we knew the direction we wanted to push, which we currently do not).
Even for something simple such as “slow AGI development down”, the devil is in the detail: slow what down, where, how?
That said, it’s nonetheless highly unlikely that the status-quo/default situation is the ideal—so it does seem probable that some form of broader communication is desirable.
But it’s usually much easier to make things worse than to make them better—so I’d want most effort to go into figuring out the mechanics of a potentially useful plan. Most interventions don’t achieve what was intended—even the plausibly helpful ones (for instance, one might try recruiting John Carmack to work on AI safety [this strikes me as a good idea, hindsight notwithstanding], only to get him interested enough that he starts up an AGI company a few years later).
I understand that it must be frustrating for anyone who takes this very seriously and believes they cannot engage in research themselves (though I’d suggest asking how they can help). However, the goal needs to be to actually improve the situation, rather than to do something that feels good. The latter impulse is much simpler to satisfy than the former.
To find a communication strategy that actually improves the situation, we need careful analysis.
With that in mind, if you want to start useful discussion on this, I’d suggest outlining:
Some plan (in broad terms).
The mechanisms by which you’d expect it to help.
Ways in which it could fail.
Reasons to think it won’t backfire relative to the status-quo.
Reasons to think it won’t render superior future plans impossible/impractical (relative to the status-quo).
I don’t think this is easy (I have no good plan). I do think it’s required.
Increasing awareness increases resources through virtue of sheer volume. The more people hear about AI safety, the more likely someone resourceful and amenable hears about AI safety.
This is a good sentiment, but ‘resource gathering’ is an instrumentally convergent strategy. No matter what researchers end up deciding we should do, it’ll probably be best done with money and status on our side.
Politicisation is not a failure mode, it’s an optimistic outcome. Politicized issues get money. Politicized issues get studied. Other failure modes might be that we increase interest in AI in general and result in destructive AI being generated more rapidly, but there’s already a massive profit motive in that direction, so I don’t know if we can really contribute in that direction. Most other ‘failure modes’ which involve ‘bad publicity’ are infinitely preferable to the current state of affairs, given you have enough dignity to be shameless.
‘Public pressure’ isn’t really a thing as far as I can tell. The public presses back. Money talks. Wheels turn. Equilibria equilibriate or something. I’m only talking about awareness.
My current plan relies on identifying a good representative for AI safety with enough clout to be taken seriously, contacting them, and then trying to get them into public discussions with rationalist-adjacent e-celebs.
I’d expect this to increase awareness for AI safety and make people who see this content more amenable to advocacy for AI safety in the future. I’d expect a minority of people who watch this content to become very interested in AI safety and try and learn more.
Assuming that I find a good advocate and succeed in getting them into a discussion with a minor celebrity, this could go wrong in the following ways:
The advocates comes across as unhinged
The advocates comes across as unlikable
The advocate cannot explain AI safety well
The advocate cannot respond to criticism well
As far as I can see, the efficacy and reliability of this plan relies entirely on the character of the advocate. Because this is being tested in a smaller corner of the internet, I think we can believe that if inexplicably results in disaster, the effect will be relatively contained, but honestly I think a pretty small amount of screening can prevent the worst of this.
The largest issue with this approach/view is that it’s not addressing the distinction between:
Increased resources for things with “AI Safety” written on them.
Increased resources for approaches that stand a chance of working.
The problem is important in large measure due to its difficulty: that we need to hit a very small target we don’t yet understand. By default, resources allocated to anything labelled “AI safety” will not be aimed at that target.
If things are politicised, it’s a safe bet they won’t be aimed at the target; politicised issues get money thrown in their general direction, but that’s not about actually solving the problem. There’s a big difference between [more money helps, all else being equal], and claiming [action x gets more money, therefore it helps]. Politicisation would have many downsides.
Likewise, even if we get more attention/money… there are potential signal-to-noise issues. Suppose that there are 20 people involved in grant allocation with enough technical understanding to pick out promising projects.
Consider two cases:
They receive 200 grant applications, 20 of which are promising.
They receive 2000 grant applications, 40 of which are promising.
In case (2) there are more promising projects, but it’s not clear that grant evaluators will find more promising projects, since the signal to noise ratio will have dropped so much.
The obvious answer is to train more grant evaluators to the point where they have the necessary expertise—but this is a slow process that’s (currently) difficult to scale (though people are working on that).
You also seem to be cherry-picking the upside possibilities from increased awareness: yes, some people may start to work on or advocate for AI safety (and some small proportion of those for some useful understanding of “AI safety”).
However, some people may also:
Hear about AI safety, realise that AGI is a big deal but not buy the safety arguments, and start working on AGI.
This is not a hypothetical situation, or something that only happens to people without much ability: if John Carmack can get this badly wrong, where are you getting your confidence that most people won’t?
Realise that AGI is a big deal, think that the major issues are misuse and/or ethics, and make poor decisions on that basis.
Make sure we get AGI before them...
Put in regulations that focus the ‘safety’ resources of AI companies on ticking meaningless boxes that do nothing to mitigate x-risk. (though I’d guess the default situation looks largely like this anyway)
We need to argue that it’s net positive, not simply that there would be some positive outcomes (I don’t think anyone would argue with that).
Again, I do think that there’s some communication strategy we should be using that beats the status-quo. However, it needs to be analysed carefully, and carried out carefully—with adjustment based on empirical feedback where possible. (my guess is that the best approaches would be highly targeted—not that this says much at all)
Is this a reference to his current personal project to work on AGI?
Edit: reading a bit more about him, I suspect if he ever got interested in alignment work he’d likely prefer working on Christiano-style stuff than MIRI-style stuff. For instance (re: metaverse):
My source on this is his recent appearance on the Lex Fridman podcast.
He’s moving beyond the personal project stage.
He does seem well-informed (and no doubt he’s a very smart guy), so I do still hope that he might update pretty radically given suitable evidence. Nonetheless, if he stays on his present course greater awareness seems to have been negative (this could absolutely change).
The tl;dr of his current position is:
Doesn’t expect a fast takeoff.
Doesn’t specify a probability or timescale—unclear whether he’s saying e.g. a 2-year takeoff seems implausible; pretty clear he finds a 2-week takeoff implausible.
We should work on ethics/safety of AGI once we have a clearer idea what it looks like. (expects we’ll have time, due to 1)
Not really dismissive of current AGI safety efforts, but wouldn’t put his money there at present, since it’s unclear what can be achieved.
My take:
On 1a, his argument seems to miss the possibility that an AGI doesn’t need to copy itself to other hardware (he has reasonable points that suggest this would be impractical), but might write/train systems that can operate with very different hardware.
If we expect smooth progress, we wouldn’t expect the first system to have the capability to do this—though it may be able to bide its time (potentially requiring gradient-hacking).
However, he expects that we need a small number of key insights for AGI (seems right to me). This is hard to square with an expectation of smooth progress.
On 2, he seems overconfident on the ease of solving the safety issues once we have a clearer idea what it looks like. I guess he’s thinking that this looks broadly similar to engineering problems we’ve handled in the past, which seems wrong to me. Everything we’ve built is narrow, so we’ve never needed to solve the “point this at exactly the thing we mean” problem (capabilities didn’t generalise for any system we’ve made safe).
On 3, I’m pretty encouraged. He seems to be thinking about things reasonably, and isn’t casually dismissing other points of view. I think his current model of the situation is incorrect (as, I’m sure is mine!), but expect that he’d be keen to change direction pretty radically if his model did change.
Agreed that he’d be a much better fit for Christiano-style stuff than MIRI-style—though I also expect he could fit well with Anthropic/Redwood/Conjecture; essentially anything with an empirical slant to it. In an ideal world, he’d be a good fit for Carmack-style stuff.
I do think it’s worthwhile to consider how to engage with him further—though this is still a case where I’d want people thinking carefully about how best to communicate before acting (and whether they’re the best messenger; I quickly conclude that I am not).
So hilariously enough it looks like Carmack got into AGI 4 years ago because Sam Altman tried to recruit him for OpenAI but at the time Carmack knew barely any ML. (I just found the Lex Fidman interview).
The update you are looking for should probably flow in the other direction. Carmack is making an explicit bet on a more brain-like architecture. If he succeeds in his goal of proving out child level AGI raised in VR in 5 years, the safety community should throw out most of their now irrelevant work and focus everything on how to make said new AGI models raised in controlled VR environments safe.