For those who are now worrying about missing some foundational aspects of knowledge on AI safety, could you point towards the first thing to read to check understanding?
Sure!! (also feel free to DM me if you want to discuss further. This is the kind of conversation I would be very willing to have with anyone who wants to chat)
The Problem is a good intro to misalignment risk and there is some interesting discussion in the comments. [1] I find that The Superintelligent Will holds up well even if it is sort of dense and old. I have been told by people I trust that this is a good resource to start with but I haven’t read it or maybe I have read it and just forgot.
I take misuse risks pretty seriously too but the argument can basically be contained in “models will be able to cause harm and people will try to use them to do that. Jailbreaks and fine-tuning attacks, etc are a thing so those bad actors may succeed in doing things like building bio weapons or something equally worrying.”
The hueristic I use when undergrads ask me if they should drop out of college is: 1. can you explain instrumental convergence to me and how this fits into the extinction story and 2. can you explain the difference between inner and outer alignment (maybe with some basic help in the form of a refresher of the definitions).[2] If you can’t do those things, you probably should be 100% be staying in school because you would not know what you are dropping out for. I would recommend that people reuse this test on themselves!
I think probability of extinction is less than the median MIRI researcher because I’m more optimistic about inner alignment but if the goal is to check understanding, this is a good place to start.
Maybe inner vs outer alignment isn’t the right framing (I think it is but it’s a source of controversy) but I still think this is a good question because it tests how well you can reason about different kinds of failure modes and how they may interact, etc.
For those who are now worrying about missing some foundational aspects of knowledge on AI safety, could you point towards the first thing to read to check understanding?
Sure!! (also feel free to DM me if you want to discuss further. This is the kind of conversation I would be very willing to have with anyone who wants to chat)
The Problem is a good intro to misalignment risk and there is some interesting discussion in the comments. [1] I find that The Superintelligent Will holds up well even if it is sort of dense and old. I have been told by people I trust that this is a good resource to start with but I haven’t read it or maybe I have read it and just forgot.
I take misuse risks pretty seriously too but the argument can basically be contained in “models will be able to cause harm and people will try to use them to do that. Jailbreaks and fine-tuning attacks, etc are a thing so those bad actors may succeed in doing things like building bio weapons or something equally worrying.”
The hueristic I use when undergrads ask me if they should drop out of college is: 1. can you explain instrumental convergence to me and how this fits into the extinction story and 2. can you explain the difference between inner and outer alignment (maybe with some basic help in the form of a refresher of the definitions).[2] If you can’t do those things, you probably should be 100% be staying in school because you would not know what you are dropping out for. I would recommend that people reuse this test on themselves!
I think probability of extinction is less than the median MIRI researcher because I’m more optimistic about inner alignment but if the goal is to check understanding, this is a good place to start.
Maybe inner vs outer alignment isn’t the right framing (I think it is but it’s a source of controversy) but I still think this is a good question because it tests how well you can reason about different kinds of failure modes and how they may interact, etc.