I think to maximize legibility to different kinds of people, it helps to include “instrumental convergence usually involves all things converging on the path of accumulating resources (including information) and self-preservation, and, by extension, anything that contributes to self-preservation and accumulating resources”.
Instrumental Convergence might seem like common language, but it certainly is not, and it really should be. Evocative examples increase word count, but they’re so helpful that cutting them out in order to lower word count is basically Goodharting. At minimum, it will help an unfamiliar reader quickly and conveniently explain it to others in many casual conversations.
Also, this post is very helpful, I have archived it in my open-source intelligence/research folder and I plan to cite it in the future. The way this post was written is actually the ideal way to explain AI safety to someone for the first time since it allows a wide variety of intelligent people to really dive into the problem in a way they understand. I recommend submitting it to the AI Safety Public Materials Contest which I recently confirmed will still read, evaluate, and implement good papers for explaining AI safety to people for the first time. Regardless of whether they have any money to award, they will still credit the author with a ranking that will build credibility.
Hmm. I see what you mean about how the post could be expanded into an intro to AI safety if I gave more examples and such. It’s not exactly the target audience I had in mind while writing it, I was more thinking someone who already had the basic intro but who might not have a strong framework to think definitely about broader questions.
I’ll look into making a beginner-friendly version of it, and ping you if I have anything. I’ll likely be busy, though, so I would be open to you adding the examples etc. for beginners and then splitting the prize for the contest if you want.
I think to maximize legibility to different kinds of people, it helps to include “instrumental convergence usually involves all things converging on the path of accumulating resources (including information) and self-preservation, and, by extension, anything that contributes to self-preservation and accumulating resources”.
Instrumental Convergence might seem like common language, but it certainly is not, and it really should be. Evocative examples increase word count, but they’re so helpful that cutting them out in order to lower word count is basically Goodharting. At minimum, it will help an unfamiliar reader quickly and conveniently explain it to others in many casual conversations.
Also, this post is very helpful, I have archived it in my open-source intelligence/research folder and I plan to cite it in the future. The way this post was written is actually the ideal way to explain AI safety to someone for the first time since it allows a wide variety of intelligent people to really dive into the problem in a way they understand. I recommend submitting it to the AI Safety Public Materials Contest which I recently confirmed will still read, evaluate, and implement good papers for explaining AI safety to people for the first time. Regardless of whether they have any money to award, they will still credit the author with a ranking that will build credibility.
Hmm. I see what you mean about how the post could be expanded into an intro to AI safety if I gave more examples and such. It’s not exactly the target audience I had in mind while writing it, I was more thinking someone who already had the basic intro but who might not have a strong framework to think definitely about broader questions.
I’ll look into making a beginner-friendly version of it, and ping you if I have anything. I’ll likely be busy, though, so I would be open to you adding the examples etc. for beginners and then splitting the prize for the contest if you want.