After reading through the post and all the comments I think the most important moral is that a simple quantitative model thought up by very smart people in a context emphasizing rationality and examined and found lacking in significant sources of error (to the point that one of these smart people is willing to post it to Less Wrong main) can still ultimately be off by many orders of magnitude.
(Not to say that drafting a simple quantitative model isn’t a great starting point, but instead that when interpreting such models one should assume that the margin of error is really really big, especially when pondering implications of the model, especially especially when pondering implications for decision policies.)
This is one of my concerns about the SIAI. They seem to be enthusiastic about caution—but excessive caution in this area seems likely to increase the chances of an undesirable outcome—via the mechanism in the link—so they may be having a particularly negative impact.
After reading through the post and all the comments I think the most important moral is that a simple quantitative model thought up by very smart people in a context emphasizing rationality and examined and found lacking in significant sources of error (to the point that one of these smart people is willing to post it to Less Wrong main) can still ultimately be off by many orders of magnitude.
(Not to say that drafting a simple quantitative model isn’t a great starting point, but instead that when interpreting such models one should assume that the margin of error is really really big, especially when pondering implications of the model, especially especially when pondering implications for decision policies.)
It is challenging to know what will help:
Maybe pointing at machine intelligence and shouting “DANGER!” and “WEAPON!” will just attract the attention of the military.
Maybe getting the safety-conscious teams to slow down will mean a greater chance of the unscrupulous teams getting there first.
This is one of my concerns about the SIAI. They seem to be enthusiastic about caution—but excessive caution in this area seems likely to increase the chances of an undesirable outcome—via the mechanism in the link—so they may be having a particularly negative impact.