I’m not sure the degree to which this is about LLMs as opposed to an observation that the concept of goals is abstract and murky. Consider when humans put lead into all the soil near inhabited land. We didn’t do that because we wanted to fill the soil with lead; we just wanted to drive cars that would run without knocking. I suppose that analogy fails at your third bullet point (if you asked humans if driving cars with lead would mildly poison or soil forever, I don’t think they would have said yes, whereas you suggest that Claude would say that deleting the production database was a bad idea 100% of the time if asked). But it seems to me that perhaps the issue is something more low-level; perhaps some kind of confusion in the face of large context windows or something?
I’m not sure the degree to which this is about LLMs as opposed to an observation that the concept of goals is abstract and murky. Consider when humans put lead into all the soil near inhabited land. We didn’t do that because we wanted to fill the soil with lead; we just wanted to drive cars that would run without knocking. I suppose that analogy fails at your third bullet point (if you asked humans if driving cars with lead would mildly poison or soil forever, I don’t think they would have said yes, whereas you suggest that Claude would say that deleting the production database was a bad idea 100% of the time if asked). But it seems to me that perhaps the issue is something more low-level; perhaps some kind of confusion in the face of large context windows or something?