To me the most promising solution is to get the AI to not optimize for influencing people’s beliefs (3.4) except in certain permitted (often myopic) ways that depend on the situation. Some candidate guidelines:
Early on, helping AI companies understand risks seems crucial and allowed.
When they help with development of the model spec, they can help people understand relevant considerations to the model spec, but should do so myopically and should not consider consequences downstream of those people’s beliefs (e.g., on the model spec, on the people’s actions). This has downsides in terms of slop like you say, but: (1) I think myopically influencing people’s beliefs on requested questions does a huge amount to help and (2) I think it could also be permissible to proactively take actions based on downstream consequences sometimes if this is done openly and without much optimization pressure, ideally making arguments a human could understand (e.g., rather than just framing its discussion in a certain way to achieve a different model spec, it should instead leave a note explainingwhy it worries the default interpretation might lead to a suboptimal model spec).
Later on, during reflection, this might look like AIs only being allowed to myopicallyprovide guidance on (some) descriptive claims(aiming for something similar to 3.1). I share worries about reflection being high-variance and underspecified, but I think this is a somewhat fundamental limit on value idealization that doesn’t have much to do with AI.
I also share worries that consequentialist goals can erode all of these guidelines, but this is a somewhat separate concern (separating deliberation and competition seems good here).
To me the most promising solution is to get the AI to not optimize for influencing people’s beliefs (3.4) except in certain permitted (often myopic) ways that depend on the situation. Some candidate guidelines:
Early on, helping AI companies understand risks seems crucial and allowed.
When they help with development of the model spec, they can help people understand relevant considerations to the model spec, but should do so myopically and should not consider consequences downstream of those people’s beliefs (e.g., on the model spec, on the people’s actions). This has downsides in terms of slop like you say, but: (1) I think myopically influencing people’s beliefs on requested questions does a huge amount to help and (2) I think it could also be permissible to proactively take actions based on downstream consequences sometimes if this is done openly and without much optimization pressure, ideally making arguments a human could understand (e.g., rather than just framing its discussion in a certain way to achieve a different model spec, it should instead leave a note explaining why it worries the default interpretation might lead to a suboptimal model spec).
Later on, during reflection, this might look like AIs only being allowed to myopically provide guidance on (some) descriptive claims (aiming for something similar to 3.1). I share worries about reflection being high-variance and underspecified, but I think this is a somewhat fundamental limit on value idealization that doesn’t have much to do with AI.
I also share worries that consequentialist goals can erode all of these guidelines, but this is a somewhat separate concern (separating deliberation and competition seems good here).