I was going to say that Claude is not a typical machine and we should expect more from AGI. But actually, all sorts of modern machines increasingly produce various forms of error reporting when they can’t do their task, and increasingly don’t keep charging ahead blindly. If we can demand fail-safe from a water cooker or a washing machine, we can certainly demand it from Claude.
I don’t think that comparison works. We usually have machines refuse some bad orders, but there is a limit to what they are able to refuse. If you fill the washing machine with clothes that run in hot water and press the hot water button by mistake instead of the cold water button it won’t refuse the order on the grounds that it will mess up your clothes. The washing machine is not misaligned because it just ruined your clothes.
I was going to say that Claude is not a typical machine and we should expect more from AGI. But actually, all sorts of modern machines increasingly produce various forms of error reporting when they can’t do their task, and increasingly don’t keep charging ahead blindly. If we can demand fail-safe from a water cooker or a washing machine, we can certainly demand it from Claude.
I don’t think that comparison works. We usually have machines refuse some bad orders, but there is a limit to what they are able to refuse. If you fill the washing machine with clothes that run in hot water and press the hot water button by mistake instead of the cold water button it won’t refuse the order on the grounds that it will mess up your clothes. The washing machine is not misaligned because it just ruined your clothes.