Thank you for your elaboration, I appreciate it a lot, and upvoted for the effort. Here are your clearest points paraphrased as I understand them (sometimes just using your words), and my replies:
The FDA is net negative for health, therefore creating an FDA-for-AI would be likely net negative for the AI challenges.
I don’t think you can come to this conclusion, even if I agree with the premise. The counterfactuals are very different. With drugs the counterfactual of no FDA might be some people get more treatments, and some die but many don’t, and they were sick anyway so need to do something, and maybe fewer die than do with the FDA around, so maybe the existence of the FDA compared to the counterfactual is net bad. I won’t dispute this, I don’t know enough about it. However, the counterfactual in AI is different. If unregulated, AI progress steams on ahead, competition over the high rewards is high, and if we don’t have good safety plan (which we don’t) then maybe we all die at some point, who knows when. However, if an FDA-for-AI creates bad regulation (as long as it’s not bad enough to cause AI regulation winter) then it starts slowing down that progress. Maybe it’s bad for, idk, the diseases that could have been solved during the 10 years slowing down from when AI would have solved cancer vs not, and that kind of thing, but it’s nowhere near as bad as the counterfactual! These scenarios are different and not comparable, because the counterfactual of no FDA is not as bad as the counterfactual of no AI regulator.
Enough errors would almost certainly occur in AI regulation to make it net negative.
You gave a bunch of examples from non-AI regulation of bad regulation (I am not going to bother to think about whether I agree that they are bad regulation as it’s not cruxy) - but you didn’t explain how exactly errors lead to making AI regulation net negative? Again I think similar to the previous claim, the counterfactuals likely make this not hold.
...a field where there is bound to be vastly more misunderstanding should be at least as prone to regulation backfiring
That is an interesting claim, I am not sure what makes you think it’s obviously true, as it depends what your goal is. My understanding of the OP is that the goal of the type of regulation they advocate is simply to slow down AI development, nothing more, nothing less. If the goal is to do good regulation of AI, that’s totally different. Is there a specific way in which you imagine it backfiring for the goal of simply slowing down AI progress?
...an [oppressive] regime gaining controllable AI would produce an astronomical suffering risk.
I am unsure what point you were making in the paragraph about evil. Was it about another regime getting there first that might not do safety? For response, see the OP Objection 4 which I share and added additional reason for that not being a real worry in this world.
...unwise to think that people who take blatant actions to kill innocents for political convenience would be safe custodians of AI..
I don’t think it’s fair to say regulators would be a custodian. They have a special kind of lever called “slow things down”, and that lever does not mean that they can, for example, seize and start operating the AI. It is not in their power to do that, legally, nor do they have the capability to do anything with it. We are talking here about slowing things down before AGI, not post AGI.
the electorate does not understand AI
Answer is same as my answer to 3. and also similar to OP Objection 1.
And finally to reply to this: “hopefully this should clarify to a degree why I anticipate both severe X risks and S risks from most attempts at AI regulation”
Basically, no, it doesn’t really clarify it. You started off with a premise I agreed with or at least do not know enough to refute, that the FDA may be net negative, and then drew a conclusion that I disagree with (see 1. above), and then all your other points were assuming that conclusion, so I couldn’t really follow. I tried to pick out bits that seemed like possible key points and reply, but yeah I think you’re pretty confused.
What do you think of my reply to 1. - the counterfactuals being different. I think that’s the best way to progress the conversation.
I visited the EA Hotel last year for a few days and enjoyed my stay and think the project is good on net and would like to see it funded. But I think it could be better, namely I disagree about the vetting policy being so open if it aims to be an incubator:
The fact that it is possible to randomly get some good outcomes despite low vetting standards does not make a cost-effective way to get good outcomes. The Hotel being hits-based approach does not preclude a better vetting policy.
IMO, the acceptance policy should be: by default all rooms cost and then people working on impact projects or prep for future impact work can apply to have a room for free. If they meet the minimum standard I take them. If there are more applications that meet the minimum standard than rooms, I prioritise them. I would have no fixed amount of rooms for paying or non-paying guests. My minimum standard would vary according to the financial situation and with reflections when enough data builds up.
You said, “strong vetting is nice, but there’s no replacement for simply trying many things and seeing what works”—these are not mutually exclusive. When you do strong vetting, you typically have criteria (priors) that you update as well as updating the process as you learn what is working.
About the plans to vet post-hoc: I predict bias towards keeping people because of sunken costs on both sides.
Projects are not independent, interesting projects happening there will attract even more interesting projects (in particular complementary ones). They could be put on the website under Current Residents creating positive feedback loops. Especially since people who are planning to drop things and move to Blackpool are going to want to have some confidence in where they’re going.
I agree that tapping into the thought diversity of the larger community is good, I just think that you need some vetting—what I would like to see is a plan for a vetting process which both gets this diversity but also maximises quality. I don’t think you need a total open doors policy to get thought diversity, although acknowledge the trade off.
The comparison to https://www.cityyear.org/boston is interesting. If the goal is to create something similar to that then I take back all my points about vetting. I just think that is a quite a different goal to the one of being an incubator for high impact projects. Why do I think this. Because I think the set of people who need lots of support to “stay on track” and the set of people whose incubated projects are going to make very high positive impact in the world overlap rarely. The project aims to target these rare people, but I think this particular rare group are exceptional by definition and have already learnt how to bootstrap by themselves.
However, maybe there are people in the tails of a slightly different but similar distribution of people who will not make very high impact things but might do medium impact things, who do not yet know how to bootstrap. This population seems hard to model in my head somehow, at least the boundaries are fuzzy. The EA Hotel’s approach makes more sense to me if this is the goal. If I were them I might test the hypothesis of a very low bar, though I would still have the bar be a little bit higher than it is currently, if only for the feedback loops I talked about.
Basically I think as an incubator the model doesn’t work without increasing the bar, as a refuge it works and could be very valuable but then it shouldn’t be portrayed as an incubator. If it aims to be both, then it is really hard to model this medium potential impact target audience and I think that some work should be done to identify who is and who is not in it—and ultimately there should still be some raising of the bar. I also think the Hotel is valuable for visitors and for random events like retreats and unconferences. All round, a truly uniquely good idea.
An irrelevant aside: I don’t like the pyramid. In particular, the distinction drawn between someone who self-identifies strongly or weakly as EA seems irrelevant. Do you believe there is any correlation of interest with respect to getting a project funded? When I look at the most interesting things, some are done by people who self-identify strongly and others are done by people who keep their identity small.