Something feels off (or maybe just “sad”?) about the discussion here, although I don’t know that there’s an immediately accessible better option.
I think it’s pretty reasonable for people to be replying skeptically with “Look man it’s really easy to make dumb regulations, and you really look like you’re going to go create a huge opaque bureaucracy with lots of power, no understanding, and they’re gonna get regulatory captured and mostly bad things will happen. Meanwhile it looks like you’re overstating ‘how few AI companies are going to be affected by this’.”
(It’s even more reasonable to start that off with questions like “have you checked for X downside?”, than jumping immediately to that point)
But, also, the people saying that AFAICT also basically don’t think the problem of AI doom is especially real to begin with. So while I believe all the things they’re saying… I don’t buy that these line of argument are their primary cruxes. I feel like I’m in a murky, epistemically hostile territory. (This applies to both the critics here, who have an incentive to point out downsides but not upsides, as well as Thomas and other pro-regulation people, who have an incentive to downplay the downsides)
This recent post feels relevant, quoting the excerpt that feels most significant:
Imagine that I own a factory that I’m considering expanding onto the neighboring wetlands, and you run a local environmental protection group. The regulatory commission with the power to block the factory expansion has a mandate to protect local avian life, but not to preserve wetland area. The factory emits small amounts of Examplene gas. You argue before the regulatory commission that the expansion should be blocked because the latest Science shows that Examplene makes birds sad. I counterargue that the latest–latest Science shows that Examplene actually makes birds happy; the previous studies misheard their laughter as tears and should be retracted.
Realistically, it seems unlikely that our apparent disagreement is “really” about the effects of Examplene on avian mood regulation. More likely, what’s actually going on is a conflict rather than a disagreement: I want to expand my factory onto the wetlands, and you want me to not do that. The question of how Examplene pollution affects birds only came into it in order to persuade the regulatory commission.
It’s inefficient that our conflict is being disguised as a disagreement. We can’t both get what we want, but however the factory expansion question ultimately gets resolved, it would be better to reach that outcome without distorting Society’s shared map of the bioactive properties of Examplene. (Maybe it doesn’t affect the birds at all!) Whatever the true answer is, Society has a better shot at figuring it out if someone is allowed to point out your bias and mine (because facts about which evidence gets promoted to one’s attention are relevant to how one should update on that evidence).
Also:
Given that there’s usually “something else” going on in persistent disagreements, how do we go on, if we can’t rely on the assumption of good faith? I see two main strategies, each with their own cost–benefit profile.
One strategy is to stick the object level. Arguments can be evaluated on their merits, without addressing what the speaker’s angle is in saying it (even if you think there’s probably an angle). This delivers most of the benefits of “assume good faith” norms; the main difference I’m proposing is that speakers’ intentions be regarded as off-topic rather than presumed to be honest.
The other [Another] strategy is full-contact psychoanalysis: in addition to debating the object-level arguments, interlocutors have free reign to question each other’s motives. This is difficult to pull off, which is why most people most of the time should stick to the object level. Done well, it looks like a negotiation: in the course of discussion, pseudo-disagreements (where I argue for a belief because it’s in my interests for that belief to be on the shared map) are factorized out into real disagreements and bargaining over interests so that Pareto improvements can be located and taken, rather than both parties fighting to distort the shared map in the service of their interests.
Right now people seem to be mostly sticking to the object level, and that seems like a pretty good call. I think the counterarguments basically seem right and important to bring up. I think things’d be worse if people were psychoanalyzing each other here.
But, I guess I at least wanted to flag I expect the conversation here to be subtle warped, and a proxy-debate that’s mostly about a different topic that’s harder to make concrete claims about (i.e. “is AI x-risk important/urgent enough to be worth spinning up some kind of powerful regulatory process in the first place?”)
For myself: I think even in pretty optimistic worlds, you still need some kind of powerful tool that prevents runaway AI processes from dealing lots of damage. Any such tool that actually worked would be pretty dystopian if it were applied to most run-of-the-mill new technologies, but doesn’t feel dystopian (to me) when applied to the reference class of, i.e. nukes and bioweapons.
The tool could hypothetically be “specific regulations”, “a regulatory body”, “an aligned AI watchdog”, “a ecosystem of open source tool-AI watchdogs”. Some of those probably work better than others.
There’s separate questions that need resolving, for:
actually ensuring the tool actually points at the right thing
getting political buy-in for the tool.
I think single-regulations basically can’t work because they’re too dumb.
I can (vaguely) imagine hypothetical broad-powers-regulatory organizations, or “somehow well balanced ecosystems”, or “aligned AI pivotal watchdogs” work (although all of them have major ??? sections).
I’m pretty sympathetic to “it’s hard to make regulatory organizations do the right thing”, but, from perspective if you want x-risk folk to do something else you need to actually provide a good idea that will actually work. I expect most things that could possibly work to still feel pretty dystopian-if-misaimed and high risk.
The main problem I see with “regulatory body with broad powers” is that you do actually need someone who really fucking knows what they’re doing at the top, and the sort of people who actually know what they’re doing would probably hate the job and seem unlikely to do it. I think this is a fixable problem but, like, actually needs doing.
For what it’s worth, this is what the actual conflict looks like to me. I apologize if I sound bitter in the following.
LessWrong (et EA) has had a lot of people interested in AI, over history. A big chunk of these have been those with (1) short timelines and (2) high existential doom-percentages, but have by no means been the only people in LessWrong.
There were also people with longer timelines, or ~0.1% doom percentages, who nevertheless thought it would be good to work on as a tail risk. There were also people who were intrigued by the intellectual challenge of understanding intelligence. There were also people who were more concerned about risks from multipolar situations. There were even people just interested in rationality. All these together made up kinda the “big tent” of LW.
Over the last few months months though, there has been a concerted push to get regulations on the board now, which seems to come from people with short timelines and high p-doom. This leads to the following frictions:
I think in many cases (not merely CAIP), they are pushing for things that would shred a lot of things the “big tent” coalition in LW would care about, to guard against dangers that many people in the big tent coalition don’t think are dangers. When they talk about bad side-effects of their policies, it’s almost solely to explicitly downplay them. (I could point to other places where EAs have [imo, obviously falsely] downplayed the costs of their proposed regulations.) This feels like a betrayal of intellectual standards.
They’ve introduced terminology created for negative connotative load rather than denotative clarity and put it everywhere (“AI proliferation”), which pains me every time I read it. This feels like a betrayal of intellectual standards.
They’ve started writing a quantity of “introductory material” which is explicitly politically tilted, and I think really bad for noobs because it exists to sell a story rather than to describe the situation. I.e., I think Yud’s last meditation on LLMs is probably just harmful / confusing for a noob to ML to read; the Letter to Time obviously aims to persuade not explain; the Rational Animations “What to authorities have to say on AI risk” is for sure tilted, and even other sources (can’t find PDF at moment) sell dubious “facts” like “capabilities are growing faster than our ability to control.” This also feels like a betrayal of intellectual standards.
I’m sorry I don’t have more specific examples of the above; I’m trying to complete this comment in a limited time.
I realize in many places I’m just complaining about people on the internet being wrong. But a fair chunk of the above is coming not merely from randos on the internet but from the heads of EA-funded and EA-sponsored or now LW-sponsored organizations. And this has basically made me think, “Nope, no one in these places actually—like actually—gives a shit about what I care about. They don’t even give a shit about rationality, except inasmuch as it serves their purposes. They’re not even going to investigate downsides to what they propose.”
And it looks to me like the short timeline / high pdoom group are collectively telling what was the big tent coalition to “get with the program”—as, for instance, Zvi has chided Jack Clark, for being insufficiently repressive. And well, that’s like… not going to fly with people who weren’t convinced by your arguments in the first place. They’re going to look around at each other, be like “did you hear that?”, and try to find other places that value what they value, that make arguments that they think make sense, and that they feel are more intellectually honest.
It’s fun and intellectually engaging to be in a community where people disagree with each other. It sucks to be in a community where people are pushing for (what you think are) bad policies that you disagree with, and turning that community into a vehicle for pushing those policies. The disagreement loses the fun and savor.
I would like to be able to read political proposals from EA or LW funded institutions and not automatically anticipate that they will hide things from me. I would like to be able to read summaries of AI risk which advert to both strengths and weaknesses in such arguments. I would like things I post on LW to not feed a community whose chief legislative impact looks right now to be solely adding stupidly conceived regulations to the lawbooks.
I’m sorry I sound bitter. This is what I’m actually concerned about.
Edit: shoulda responded to your top level, whatever.
This is a good comment, and I think describes some of what is going on. I also feel concerned about some of those dynamics, though I do have high p-doom (and like 13 year timelines, which I think is maybe on the longer side these days, so not sure where I fall here in your ontology).
I disagree a lot with the examples you list that you say are deceiving or wrong. Like, I do think capabilities are growing faster than our ability to control, and that feels like a fine summary of the situation (though also not like an amazing one).
I also personally don’t care much about “the big tent” coalition. I care about saying what I believe. I don’t want to speak on behalf of others, but I also really don’t want to downplay what I believe because other people think that will make them look bad.
Independently of my commitment to not join mutual reputation protection alliances, my sense is most actions that have been taken so far by people vaguely in the LW/EA space in the public sphere and the policy sphere have been quite harmful (and e.g. involved giving huge amounts of power and legitimacy to AI capability companies), so I don’t feel much responsibility to coordinate with or help the people who made that happen. I like many of those people, and think they are smart, and I like talking to them and sometimes learn things from them, but I don’t think I owe them much in terms of coordinating our public messaging on AI, or something like that (though I do owe them not speaking on their behalf, and I do think a lot of people could do much better to speak more on behalf of themselves and less on behalf of ‘the AI safety community’).
For myself, I’ve come back to believing that AI doom is probably worth worrying about a little, and I no longer view AI doom as basically a non-problem, due to new studies.
RE viewing this as a conflict, I agree with this mindset, but with one caveat: There are also vast prior and empirical disagreements too, and while there is a large conflict of values, it’s magnified even larger by uncertainty.
I don’t think that you should be able to ignore the very real problems with the proposed policy just because you think there are other disagreements that people have.
Because those problems still remain problems, regardless of whatever other argument you want to have.
It is the job of this foundation to make good policies. If those policies are bad, thats a problem regardless of who is pointing out the problem.
Something feels off (or maybe just “sad”?) about the discussion here, although I don’t know that there’s an immediately accessible better option.
I think it’s pretty reasonable for people to be replying skeptically with “Look man it’s really easy to make dumb regulations, and you really look like you’re going to go create a huge opaque bureaucracy with lots of power, no understanding, and they’re gonna get regulatory captured and mostly bad things will happen. Meanwhile it looks like you’re overstating ‘how few AI companies are going to be affected by this’.”
(It’s even more reasonable to start that off with questions like “have you checked for X downside?”, than jumping immediately to that point)
But, also, the people saying that AFAICT also basically don’t think the problem of AI doom is especially real to begin with. So while I believe all the things they’re saying… I don’t buy that these line of argument are their primary cruxes. I feel like I’m in a murky, epistemically hostile territory. (This applies to both the critics here, who have an incentive to point out downsides but not upsides, as well as Thomas and other pro-regulation people, who have an incentive to downplay the downsides)
This recent post feels relevant, quoting the excerpt that feels most significant:
Also:
Right now people seem to be mostly sticking to the object level, and that seems like a pretty good call. I think the counterarguments basically seem right and important to bring up. I think things’d be worse if people were psychoanalyzing each other here.
But, I guess I at least wanted to flag I expect the conversation here to be subtle warped, and a proxy-debate that’s mostly about a different topic that’s harder to make concrete claims about (i.e. “is AI x-risk important/urgent enough to be worth spinning up some kind of powerful regulatory process in the first place?”)
For myself: I think even in pretty optimistic worlds, you still need some kind of powerful tool that prevents runaway AI processes from dealing lots of damage. Any such tool that actually worked would be pretty dystopian if it were applied to most run-of-the-mill new technologies, but doesn’t feel dystopian (to me) when applied to the reference class of, i.e. nukes and bioweapons.
The tool could hypothetically be “specific regulations”, “a regulatory body”, “an aligned AI watchdog”, “a ecosystem of open source tool-AI watchdogs”. Some of those probably work better than others.
There’s separate questions that need resolving, for:
actually ensuring the tool actually points at the right thing
getting political buy-in for the tool.
I think single-regulations basically can’t work because they’re too dumb.
I can (vaguely) imagine hypothetical broad-powers-regulatory organizations, or “somehow well balanced ecosystems”, or “aligned AI pivotal watchdogs” work (although all of them have major ??? sections).
I’m pretty sympathetic to “it’s hard to make regulatory organizations do the right thing”, but, from perspective if you want x-risk folk to do something else you need to actually provide a good idea that will actually work. I expect most things that could possibly work to still feel pretty dystopian-if-misaimed and high risk.
The main problem I see with “regulatory body with broad powers” is that you do actually need someone who really fucking knows what they’re doing at the top, and the sort of people who actually know what they’re doing would probably hate the job and seem unlikely to do it. I think this is a fixable problem but, like, actually needs doing.
For what it’s worth, this is what the actual conflict looks like to me. I apologize if I sound bitter in the following.
LessWrong (et EA) has had a lot of people interested in AI, over history. A big chunk of these have been those with (1) short timelines and (2) high existential doom-percentages, but have by no means been the only people in LessWrong.
There were also people with longer timelines, or ~0.1% doom percentages, who nevertheless thought it would be good to work on as a tail risk. There were also people who were intrigued by the intellectual challenge of understanding intelligence. There were also people who were more concerned about risks from multipolar situations. There were even people just interested in rationality. All these together made up kinda the “big tent” of LW.
Over the last few months months though, there has been a concerted push to get regulations on the board now, which seems to come from people with short timelines and high p-doom. This leads to the following frictions:
I think in many cases (not merely CAIP), they are pushing for things that would shred a lot of things the “big tent” coalition in LW would care about, to guard against dangers that many people in the big tent coalition don’t think are dangers. When they talk about bad side-effects of their policies, it’s almost solely to explicitly downplay them. (I could point to other places where EAs have [imo, obviously falsely] downplayed the costs of their proposed regulations.) This feels like a betrayal of intellectual standards.
They’ve introduced terminology created for negative connotative load rather than denotative clarity and put it everywhere (“AI proliferation”), which pains me every time I read it. This feels like a betrayal of intellectual standards.
They’ve started writing a quantity of “introductory material” which is explicitly politically tilted, and I think really bad for noobs because it exists to sell a story rather than to describe the situation. I.e., I think Yud’s last meditation on LLMs is probably just harmful / confusing for a noob to ML to read; the Letter to Time obviously aims to persuade not explain; the Rational Animations “What to authorities have to say on AI risk” is for sure tilted, and even other sources (can’t find PDF at moment) sell dubious “facts” like “capabilities are growing faster than our ability to control.” This also feels like a betrayal of intellectual standards.
I’m sorry I don’t have more specific examples of the above; I’m trying to complete this comment in a limited time.
I realize in many places I’m just complaining about people on the internet being wrong. But a fair chunk of the above is coming not merely from randos on the internet but from the heads of EA-funded and EA-sponsored or now LW-sponsored organizations. And this has basically made me think, “Nope, no one in these places actually—like actually—gives a shit about what I care about. They don’t even give a shit about rationality, except inasmuch as it serves their purposes. They’re not even going to investigate downsides to what they propose.”
And it looks to me like the short timeline / high pdoom group are collectively telling what was the big tent coalition to “get with the program”—as, for instance, Zvi has chided Jack Clark, for being insufficiently repressive. And well, that’s like… not going to fly with people who weren’t convinced by your arguments in the first place. They’re going to look around at each other, be like “did you hear that?”, and try to find other places that value what they value, that make arguments that they think make sense, and that they feel are more intellectually honest.
It’s fun and intellectually engaging to be in a community where people disagree with each other. It sucks to be in a community where people are pushing for (what you think are) bad policies that you disagree with, and turning that community into a vehicle for pushing those policies. The disagreement loses the fun and savor.
I would like to be able to read political proposals from EA or LW funded institutions and not automatically anticipate that they will hide things from me. I would like to be able to read summaries of AI risk which advert to both strengths and weaknesses in such arguments. I would like things I post on LW to not feed a community whose chief legislative impact looks right now to be solely adding stupidly conceived regulations to the lawbooks.
I’m sorry I sound bitter. This is what I’m actually concerned about.
Edit: shoulda responded to your top level, whatever.
This is a good comment, and I think describes some of what is going on. I also feel concerned about some of those dynamics, though I do have high p-doom (and like 13 year timelines, which I think is maybe on the longer side these days, so not sure where I fall here in your ontology).
I disagree a lot with the examples you list that you say are deceiving or wrong. Like, I do think capabilities are growing faster than our ability to control, and that feels like a fine summary of the situation (though also not like an amazing one).
I also personally don’t care much about “the big tent” coalition. I care about saying what I believe. I don’t want to speak on behalf of others, but I also really don’t want to downplay what I believe because other people think that will make them look bad.
Independently of my commitment to not join mutual reputation protection alliances, my sense is most actions that have been taken so far by people vaguely in the LW/EA space in the public sphere and the policy sphere have been quite harmful (and e.g. involved giving huge amounts of power and legitimacy to AI capability companies), so I don’t feel much responsibility to coordinate with or help the people who made that happen. I like many of those people, and think they are smart, and I like talking to them and sometimes learn things from them, but I don’t think I owe them much in terms of coordinating our public messaging on AI, or something like that (though I do owe them not speaking on their behalf, and I do think a lot of people could do much better to speak more on behalf of themselves and less on behalf of ‘the AI safety community’).
Did you swap your word ordering, or does this not belong on that list?
For myself, I’ve come back to believing that AI doom is probably worth worrying about a little, and I no longer view AI doom as basically a non-problem, due to new studies.
RE viewing this as a conflict, I agree with this mindset, but with one caveat: There are also vast prior and empirical disagreements too, and while there is a large conflict of values, it’s magnified even larger by uncertainty.
I don’t think that you should be able to ignore the very real problems with the proposed policy just because you think there are other disagreements that people have.
Because those problems still remain problems, regardless of whatever other argument you want to have.
It is the job of this foundation to make good policies. If those policies are bad, thats a problem regardless of who is pointing out the problem.
I agree. (I said that in my comment). Not sure what you’re arguing against.
(Note: there’s a bunch of background context on LessWrong on how to do a better job arguing about politics. See the LessWrong Political Prerequisites sequence)