I initially liked this post a lot, then saw a lot of pushback in the comments, mostly of the (very valid!) form of “we actually build reliable things out of unreliable things, particularly with computers, all the time”. I think this is a fair criticism of the post (and choice of examples/metaphors therein), but I think it may be missing (one of) the core message(s) trying to be delivered.
I wanna give an interpretation/steelman of what I think John is trying to convey here (which I don’t know whether he would endorse or not):
“There are important assumptions that need to be made for the usual kind of systems security design to work (e.g. uncorrelation of failures). Some of these assumptions will (likely) not apply with AGI. Therefor, extrapolating this kind of thinking to this domain is Bad™️.” (“Epistemological vigilance is critical”)
So maybe rather than saying “trying to build robust things out of brittle things is a bad idea”, it’s more like “we can build robust things out of certain brittle things, e.g. computers, but Godzilla is not a computer, and so you should only extrapolate from computers to Godzilla if you’re really, really sure you know what you’re doing.”
I initially liked this post a lot, then saw a lot of pushback in the comments, mostly of the (very valid!) form of “we actually build reliable things out of unreliable things, particularly with computers, all the time”. I think this is a fair criticism of the post (and choice of examples/metaphors therein), but I think it may be missing (one of) the core message(s) trying to be delivered.
I wanna give an interpretation/steelman of what I think John is trying to convey here (which I don’t know whether he would endorse or not):
“There are important assumptions that need to be made for the usual kind of systems security design to work (e.g. uncorrelation of failures). Some of these assumptions will (likely) not apply with AGI. Therefor, extrapolating this kind of thinking to this domain is Bad™️.” (“Epistemological vigilance is critical”)
So maybe rather than saying “trying to build robust things out of brittle things is a bad idea”, it’s more like “we can build robust things out of certain brittle things, e.g. computers, but Godzilla is not a computer, and so you should only extrapolate from computers to Godzilla if you’re really, really sure you know what you’re doing.”