I didn’t have any candidate mistakes in mind. After thinking about it for 2 minutes, here are some possible candidates (though it’s not clear to me that any of them were actually mistakes):
Eliezer’s TIME article explicitly mentions that states should “be willing to destroy a rogue datacenter by airstrike.” On one hand, it seems important to clearly/directly communicate the kind of actions that Eliezer believes the world will need to be willing to take in order to keep us safe. On the other hand, I think this particular phrasing might have made the point easier to critique/meme. On net, it’s unclear to me if this was a mistake, but it seems plausible that there could’ve been a way to rephrase this particular sentence that maintains clarity while reducing the potential for misinterpretations or deliberate attacks.
I sometimes wonder if MIRI should have a policy division that is explicitly trying to influence policymakers. It seems like we have a particularly unique window over the next (few? several?) months, where Congress is unusually-motivated to educate themselves about AI. Until recently, my understanding is that virtually no one was advocating from a MIRI perspective (e.g., alignment is difficult, we can’t just rely on dangerous capability evaluations, we need a global moratorium and the infrastructure required for it). I think the Center for AI Policy is now trying to fill the gap, and it seems plausible to me that MIRI should either start their own policy team ( or assume more of a leadership role at the Center for AI Policy—EG being directly involved in hiring, having people in-person in DC, being involved in strategy/tactics, etc. This is of course conditional on the current CAIP CEO being open to this, and I suspect he would be).
Perhaps MIRI should be critiquing some of the existing policy proposals. I think MIRI has played an important role in “breaking” alignment proposals (i.e., raising awareness about the limitations of a lot of alignment proposals, raising counterarguments, explaining failure modes). Maybe MIRI should be “breaking” some of the policy proposals or finding other ways to directly improve the ways policymakers are reasoning about AI risk.
Again, it seems plausible to me that none of these were mistakes; I’d put them more in the “hm… this seems unclear to me but seems worth considering” category.
Re: the wording about airstrikes in TIME: yeah, we did not anticipate how that was going to be received and it’s likely we would have wordsmithed it a bit more to make the meaning more clear had we realized. I’m comfortable calling that a mistake. (I was not yet employed at MIRI at the time but I was involved in editing the draft of the op-ed so it’s at least as much on me as anybody else who was involved.)
Re: policy division: we are limited by our 501(c)3 status as to how much of our budget we can spend on policy work, and here ‘budget’ includes the time of salaried employees. Malo and Eliezer both spend some fraction of their time on policy but I view it as unlikely that we’ll spin up a whole ‘division’ about that. Instead, yes, we partner with/provide technical advice to CAIP and other allied organizations. I don’t view failure-to-start-a-policy-division as a mistake and in fact I think we’re using our resources fairly well here.
Re: critiquing existing policy proposals: there is undoubtedly more we could do here, though I lean more in the direction of ‘let’s say what we think would be almost good enough’ rather than simply critiquing what’s wrong with other proposals.
Thanks for this; seems reasonable to me. One quick note is that my impression is that it’s fairly easy to set up a 501(c)4. So even if [the formal institution known as MIRI] has limits, I think MIRI would be able to start a “sister org” that de facto serves as the policy arm. (I believe this is accepted practice & lots of orgs have sister policy orgs.)
(This doesn’t matter right now, insofar as you don’t think it would be an efficient allocation of resources to spin up a whole policy division. Just pointing it out in case your belief changes and the 501(c)3 thing felt like the limiting factor).
Thanks for this response!
I didn’t have any candidate mistakes in mind. After thinking about it for 2 minutes, here are some possible candidates (though it’s not clear to me that any of them were actually mistakes):
Eliezer’s TIME article explicitly mentions that states should “be willing to destroy a rogue datacenter by airstrike.” On one hand, it seems important to clearly/directly communicate the kind of actions that Eliezer believes the world will need to be willing to take in order to keep us safe. On the other hand, I think this particular phrasing might have made the point easier to critique/meme. On net, it’s unclear to me if this was a mistake, but it seems plausible that there could’ve been a way to rephrase this particular sentence that maintains clarity while reducing the potential for misinterpretations or deliberate attacks.
I sometimes wonder if MIRI should have a policy division that is explicitly trying to influence policymakers. It seems like we have a particularly unique window over the next (few? several?) months, where Congress is unusually-motivated to educate themselves about AI. Until recently, my understanding is that virtually no one was advocating from a MIRI perspective (e.g., alignment is difficult, we can’t just rely on dangerous capability evaluations, we need a global moratorium and the infrastructure required for it). I think the Center for AI Policy is now trying to fill the gap, and it seems plausible to me that MIRI should either start their own policy team ( or assume more of a leadership role at the Center for AI Policy—EG being directly involved in hiring, having people in-person in DC, being involved in strategy/tactics, etc. This is of course conditional on the current CAIP CEO being open to this, and I suspect he would be).
Perhaps MIRI should be critiquing some of the existing policy proposals. I think MIRI has played an important role in “breaking” alignment proposals (i.e., raising awareness about the limitations of a lot of alignment proposals, raising counterarguments, explaining failure modes). Maybe MIRI should be “breaking” some of the policy proposals or finding other ways to directly improve the ways policymakers are reasoning about AI risk.
Again, it seems plausible to me that none of these were mistakes; I’d put them more in the “hm… this seems unclear to me but seems worth considering” category.
Re: the wording about airstrikes in TIME: yeah, we did not anticipate how that was going to be received and it’s likely we would have wordsmithed it a bit more to make the meaning more clear had we realized. I’m comfortable calling that a mistake. (I was not yet employed at MIRI at the time but I was involved in editing the draft of the op-ed so it’s at least as much on me as anybody else who was involved.)
Re: policy division: we are limited by our 501(c)3 status as to how much of our budget we can spend on policy work, and here ‘budget’ includes the time of salaried employees. Malo and Eliezer both spend some fraction of their time on policy but I view it as unlikely that we’ll spin up a whole ‘division’ about that. Instead, yes, we partner with/provide technical advice to CAIP and other allied organizations. I don’t view failure-to-start-a-policy-division as a mistake and in fact I think we’re using our resources fairly well here.
Re: critiquing existing policy proposals: there is undoubtedly more we could do here, though I lean more in the direction of ‘let’s say what we think would be almost good enough’ rather than simply critiquing what’s wrong with other proposals.
Thanks for this; seems reasonable to me. One quick note is that my impression is that it’s fairly easy to set up a 501(c)4. So even if [the formal institution known as MIRI] has limits, I think MIRI would be able to start a “sister org” that de facto serves as the policy arm. (I believe this is accepted practice & lots of orgs have sister policy orgs.)
(This doesn’t matter right now, insofar as you don’t think it would be an efficient allocation of resources to spin up a whole policy division. Just pointing it out in case your belief changes and the 501(c)3 thing felt like the limiting factor).