Ok this is a bit of a tortured analogy but imagine it’s the 1600s. We learn that if the mass of an iron atom is greater than 10⁻²³ grams, everyone dies (for some contrived reason). So we need to figure out the mass of iron. AI companies shave off the smallest scrap of iron they can manage, and put it on an era-appropriate balance scale, and the scale reads zero. And they declare, see! Iron is so light that it reads as zero!
Meanwhile I’m over here like, how do we know the scale is sensitive enough to measure the mass of a single iron atom? How do we know that that tiny little shaving only contains a single atom? What if it’s actually several atoms, and it’s possible to have an even smaller quantity of iron?
(Plus there are questions we don’t even know we’re supposed to be asking, like “which isotope of iron is this?”)
In that scenario, if you ask me what evidence would convince me that an iron atom weighs less than 10⁻²³ grams, I wouldn’t know what to tell you.
I feel like that’s where we are currently with alignment research.
(This analogy doesn’t capture the fact that measurable AI alignment is improving over time according to benchmarks. It’s also over-generous in that we could prove that our scales aren’t sensitive enough to detect a difference of 10⁻²³ grams. I don’t think we can do anything analogous in the case of AI alignment.)
Ok this is a bit of a tortured analogy but imagine it’s the 1600s. We learn that if the mass of an iron atom is greater than 10⁻²³ grams, everyone dies (for some contrived reason). So we need to figure out the mass of iron. AI companies shave off the smallest scrap of iron they can manage, and put it on an era-appropriate balance scale, and the scale reads zero. And they declare, see! Iron is so light that it reads as zero!
Meanwhile I’m over here like, how do we know the scale is sensitive enough to measure the mass of a single iron atom? How do we know that that tiny little shaving only contains a single atom? What if it’s actually several atoms, and it’s possible to have an even smaller quantity of iron?
(Plus there are questions we don’t even know we’re supposed to be asking, like “which isotope of iron is this?”)
In that scenario, if you ask me what evidence would convince me that an iron atom weighs less than 10⁻²³ grams, I wouldn’t know what to tell you.
I feel like that’s where we are currently with alignment research.
(This analogy doesn’t capture the fact that measurable AI alignment is improving over time according to benchmarks. It’s also over-generous in that we could prove that our scales aren’t sensitive enough to detect a difference of 10⁻²³ grams. I don’t think we can do anything analogous in the case of AI alignment.)