I think one standard method of improving the rhetorical value of your bullet points is to attempt to come up with a scenario that seems to generally agree with you, but disagrees with your bullet points, and imagine that scenario is rhetorically being presented to you by someone else.
Example Opposition Steel Man: Imagine researchers are attempting to use a very dumb piece of software to try to cycle through ways of generating Bacteria that clean up oil spills. The software starts cycling through possible bacteria, and it turns out that as a side effect, one of the generated bacteria spreads incredibly quickly and devours lipids in living cells in the controlled training setting. The researchers decide to not use that, since they don’t want to devour organic lipids, but they accidentally break the vial, and A worldwide pandemic ensues. When asked why they didn’t institute AI safety measures, the researchers replied that they didn’t think the software was smart enough for AI safety measures to matter, since it basically just brute forced through boring parts of the research the researchers would have done anyway.
Example Opposition Steel Man (cont): This would seem to falsify the idea that A dangerous AI will be motivated to seem safe in any controlled training setting, since the AI was too dumb to have any thing resembling purposeful motivation and was still extremely dangerous, and the researchers thought of it as not even an AI, so they did not think they would have to consider the idea that not enough effort is currently being put into designing safe AIs. I would instead say not enough effort is currently being put into designing safe software.
Then, attempt to turn that Steel Man’s argument into a bullet point:
Not enough effort is currently being put into designing safe software.
Then ask yourself: Do I have any reasons to not use this bullet point, as opposed to the bullet points the Example Opposition Steel Man disagreed with?
I think the example is weak; the software was not that dangerous, the researchers were idiots who broke a vial they knew was insanely dangerous.
I think it dilutes the argument to broaden it to software in general; it could be very dangerous under exactly those circumstances (with terrible physical safety measures), but the dangers of superhuman AGI are vastly larger IMHO and deserve to remain the focus, particularly of the ultra-reduced bullet points.
I think this is as crisp and convincing a summary as I’ve ever seen; nice work! I also liked the book, but condensing it even further is a great idea.
I think the example is weak; the software was not that dangerous, the researchers were idiots who broke a vial they knew was insanely dangerous.
As a side note, I was more convinced by my example at the time, but on rereading this I realized that I wasn’t properly remembering how poorly I had expressed the context that substantially weakened the argument (The researchers accidentally breaking the vial.)
Which actually identifies a simpler rhetoric improvement method. Have someone tell you (or pretend to have someone tell you) that you’re wrong and then reread your original point again, since rereading it when under the impression that you screwed up will give you a fresh perspective on it compared to when you are writing it. I should take this as evidence that I need to do that more often on my posts.
I think one standard method of improving the rhetorical value of your bullet points is to attempt to come up with a scenario that seems to generally agree with you, but disagrees with your bullet points, and imagine that scenario is rhetorically being presented to you by someone else.
Example Opposition Steel Man: Imagine researchers are attempting to use a very dumb piece of software to try to cycle through ways of generating Bacteria that clean up oil spills. The software starts cycling through possible bacteria, and it turns out that as a side effect, one of the generated bacteria spreads incredibly quickly and devours lipids in living cells in the controlled training setting. The researchers decide to not use that, since they don’t want to devour organic lipids, but they accidentally break the vial, and A worldwide pandemic ensues. When asked why they didn’t institute AI safety measures, the researchers replied that they didn’t think the software was smart enough for AI safety measures to matter, since it basically just brute forced through boring parts of the research the researchers would have done anyway.
Example Opposition Steel Man (cont): This would seem to falsify the idea that A dangerous AI will be motivated to seem safe in any controlled training setting, since the AI was too dumb to have any thing resembling purposeful motivation and was still extremely dangerous, and the researchers thought of it as not even an AI, so they did not think they would have to consider the idea that not enough effort is currently being put into designing safe AIs. I would instead say not enough effort is currently being put into designing safe software.
Then, attempt to turn that Steel Man’s argument into a bullet point:
Not enough effort is currently being put into designing safe software.
Then ask yourself: Do I have any reasons to not use this bullet point, as opposed to the bullet points the Example Opposition Steel Man disagreed with?
I think the example is weak; the software was not that dangerous, the researchers were idiots who broke a vial they knew was insanely dangerous.
I think it dilutes the argument to broaden it to software in general; it could be very dangerous under exactly those circumstances (with terrible physical safety measures), but the dangers of superhuman AGI are vastly larger IMHO and deserve to remain the focus, particularly of the ultra-reduced bullet points.
I think this is as crisp and convincing a summary as I’ve ever seen; nice work! I also liked the book, but condensing it even further is a great idea.
As a side note, I was more convinced by my example at the time, but on rereading this I realized that I wasn’t properly remembering how poorly I had expressed the context that substantially weakened the argument (The researchers accidentally breaking the vial.)
Which actually identifies a simpler rhetoric improvement method. Have someone tell you (or pretend to have someone tell you) that you’re wrong and then reread your original point again, since rereading it when under the impression that you screwed up will give you a fresh perspective on it compared to when you are writing it. I should take this as evidence that I need to do that more often on my posts.
Thanks! I’ll think about your general point; the specific example seems weak, though.