In particular, we need our AI to act sufficiently well that it:
a) wants to become better, not worse b) has (in addition to the required capabilities) sufficiently good ethical starting knowledge that it is likely to (on net balance) improve if it tries (Note that, unlike being perfectly aligned, neither of these are particularly unusual properties for a human to have.)
I.e. we need to be, not yet fully aligned, but inside the region of convergence of full alignment. It doesn’t need to gave already solved all of ethics — it merely needs to want to, and have a good starting place and the capabilities.
Also note that these two properties are quite compatible with many things often believed to be incompatible with them! i.e., an AI that can be jailbreaked to be bad (with sufficient effort) could still meet these criteria.
b) has (in addition to the required capabilities) sufficiently good ethical sense that it is likely to (on net balance) improve if it tries (Note that, unlike being perfectly aligned, neither of these are particularly unusual properties for a human to have.)
This is a very hard criterion! It requires your feedback loops for ethical reflection to be very human like. This is pretty hard and I don’t think current systems are anywhere close to it. Yes, humans tend to have this attribute, but that is drawing a target around where the arrow hit. Of course I agree with the meta-ethics of myself, that’s like, the only thing I could agree with.
LLMs understand humans extremely well. They have scoredbetter than typical humans on tests of understanding human ethical values since GPT-4
I’m confident that we could do Evolutionary Moral Psychology on a species of sapient alien — it would be challenging, but it’s not impossible. I’m sorry to any philosophers reading this to whom this might come as a shock, but Science has been studying ethics for more than half a century, and has excellent predictive hypotheses for how and why human moral intuitions evolved. Ethics is in fact no longer a philosophical question, and hasn’t been for a while — I’m afraid Science stole another of their toys. (The philosophers still have consciousness, and aesthetics… for now.)
I view my criterion a) as difficult (especially for us to be sure of), and the capabilities part of my criterion b) as difficult, but the knowing enough about human values to start in a reasonable place part of b) is easy: that’s just nuanced trivia about humans: LLMs have been superhuman at that for years. Human value are complex and fragile — and fit into O(1GB) of data.
In particular, we need our AI to act sufficiently well that it:
a) wants to become better, not worse
b) has (in addition to the required capabilities) sufficiently good ethical starting knowledge that it is likely to (on net balance) improve if it tries
(Note that, unlike being perfectly aligned, neither of these are particularly unusual properties for a human to have.)
I.e. we need to be, not yet fully aligned, but inside the region of convergence of full alignment. It doesn’t need to gave already solved all of ethics — it merely needs to want to, and have a good starting place and the capabilities.
Agreed.
Also note that these two properties are quite compatible with many things often believed to be incompatible with them! i.e., an AI that can be jailbreaked to be bad (with sufficient effort) could still meet these criteria.
And yes, reliability is also a big fat hairy problem. Especially jailbreaking, where we have an actual, mathematical proof that LLMs are, and will always be, jailbreakable.
This is a very hard criterion! It requires your feedback loops for ethical reflection to be very human like. This is pretty hard and I don’t think current systems are anywhere close to it. Yes, humans tend to have this attribute, but that is drawing a target around where the arrow hit. Of course I agree with the meta-ethics of myself, that’s like, the only thing I could agree with.
LLMs understand humans extremely well. They have scored better than typical humans on tests of understanding human ethical values since GPT-4
I’m confident that we could do Evolutionary Moral Psychology on a species of sapient alien — it would be challenging, but it’s not impossible. I’m sorry to any philosophers reading this to whom this might come as a shock, but Science has been studying ethics for more than half a century, and has excellent predictive hypotheses for how and why human moral intuitions evolved. Ethics is in fact no longer a philosophical question, and hasn’t been for a while — I’m afraid Science stole another of their toys. (The philosophers still have consciousness, and aesthetics… for now.)
I view my criterion a) as difficult (especially for us to be sure of), and the capabilities part of my criterion b) as difficult, but the knowing enough about human values to start in a reasonable place part of b) is easy: that’s just nuanced trivia about humans: LLMs have been superhuman at that for years. Human value are complex and fragile — and fit into O(1GB) of data.