I’m excited to see this cross-over into AI safety discussions. I work on what we often call “reliability engineering” in software, and I think there’s a lot of lessons there that apply here, especially the systems-based or highly-contextualized approach, since it acknowledges the same kind of failure as, say, was pointed out in The Design of Everyday Things: just because you build something to spec doesn’t mean it works if humans make mistakes using it.
I’ve not done a lot to bring that over to LW or AF, other than a half-assed post about normalization of deviance. I’m not a great explainer, so I often feel like it’s not the most valuable thing for me to do, but I think getting people pointed to this field seems neglected and valuable to get them thinking more about how real systems fail today, rather than theorizing about how AI might fail in the future, or the relatively constrained ways AI fails today.
I’m excited to see this cross-over into AI safety discussions. I work on what we often call “reliability engineering” in software, and I think there’s a lot of lessons there that apply here, especially the systems-based or highly-contextualized approach, since it acknowledges the same kind of failure as, say, was pointed out in The Design of Everyday Things: just because you build something to spec doesn’t mean it works if humans make mistakes using it.
I’ve not done a lot to bring that over to LW or AF, other than a half-assed post about normalization of deviance. I’m not a great explainer, so I often feel like it’s not the most valuable thing for me to do, but I think getting people pointed to this field seems neglected and valuable to get them thinking more about how real systems fail today, rather than theorizing about how AI might fail in the future, or the relatively constrained ways AI fails today.