I can never tell how seriously to take those types of documents.
On one hand, AGI labs obviously have employees, including senior employees, who genuinely take the risks seriously (most notably, some very well-respected LW users, e. g. some of this specific document’s authors). I’m sure the people writing them are writing them in good faith.
On the other hand, the documents somehow never end up containing recommendations that would be drastically at odds with “race full steam ahead” (see the rather convenient Core Assumption 5 here, and subsequent “just do the standard thing plus amplified oversight” alignment plan) or opinions that could cause significant concern (see “not feeling the AGI/Singularity” in “3.6. Benefits of AGI”). And I have a nagging suspicion that if there’s ever a situation where the capability-maximizing thing to do would end up at odds with a recommendation from a published safety plan, the safety plan would be unceremoniously ignored/loopholed-around/amended. I think we saw instances of that already, and not only from OpenAI.
My current instinct is to just tune them out, on the assumption that the AGI lab in question (as opposed to the people writing the document) views them as just some nice-sounding non-binding PR.[1] Am I wrong to view it this way?
Official policy documents from AI companies can be useful in bringing certain considerations into the domain of what is allowed to be taken seriously (in particular, by the governments), as opposed to remaining weird sci-fi ideas to be ignored by most Serious People. Even declarations by AI company leaders or Turing award winners of Nobel laureates or some of the most cited AI scientists won’t by themselves have that kind of legitimizing effect. So it’s not necessary for such documents to be able to directly affect actual policies of AI companies, they can still be important in affecting these policies indirectly.
Fair point. The question of the extent to which those documents can be taken seriously as statements of company policy (as opposed to only mattering in signaling games) is still worthwhile, I think.
To be fair, while Assumption 5 is convenient, I do think some form of the assumption is at least reasonably likely to hold, and I do think something like the assumption of no software singularity being possible is a reasonable position to hold, and the nuanced articulation of that assumption is in this article:
I don’t think the assumption is so likely to hold that one can assume it as part of a safety case for AI, but I don’t think the assumption is unreasonably convenient.
I can never tell how seriously to take those types of documents.
On one hand, AGI labs obviously have employees, including senior employees, who genuinely take the risks seriously (most notably, some very well-respected LW users, e. g. some of this specific document’s authors). I’m sure the people writing them are writing them in good faith.
On the other hand, the documents somehow never end up containing recommendations that would be drastically at odds with “race full steam ahead” (see the rather convenient Core Assumption 5 here, and subsequent “just do the standard thing plus amplified oversight” alignment plan) or opinions that could cause significant concern (see “not feeling the AGI/Singularity” in “3.6. Benefits of AGI”). And I have a nagging suspicion that if there’s ever a situation where the capability-maximizing thing to do would end up at odds with a recommendation from a published safety plan, the safety plan would be unceremoniously ignored/loopholed-around/amended. I think we saw instances of that already, and not only from OpenAI.
My current instinct is to just tune them out, on the assumption that the AGI lab in question (as opposed to the people writing the document) views them as just some nice-sounding non-binding PR.[1] Am I wrong to view it this way?
Poking holes in which is still important, kudos, Zvi.
Official policy documents from AI companies can be useful in bringing certain considerations into the domain of what is allowed to be taken seriously (in particular, by the governments), as opposed to remaining weird sci-fi ideas to be ignored by most Serious People. Even declarations by AI company leaders or Turing award winners of Nobel laureates or some of the most cited AI scientists won’t by themselves have that kind of legitimizing effect. So it’s not necessary for such documents to be able to directly affect actual policies of AI companies, they can still be important in affecting these policies indirectly.
Fair point. The question of the extent to which those documents can be taken seriously as statements of company policy (as opposed to only mattering in signaling games) is still worthwhile, I think.
To be fair, while Assumption 5 is convenient, I do think some form of the assumption is at least reasonably likely to hold, and I do think something like the assumption of no software singularity being possible is a reasonable position to hold, and the nuanced articulation of that assumption is in this article:
https://epoch.ai/gradient-updates/most-ai-value-will-come-from-broad-automation-not-from-r-d
I don’t think the assumption is so likely to hold that one can assume it as part of a safety case for AI, but I don’t think the assumption is unreasonably convenient.
I agree that this isn’t an obviously unreasonable assumption to hold. But...
… that.