I think this fails to say how the analogy of cryptography transfers to metaethics. What properties of cryptography as a field make it such that you cannot roll your own. Is it just that many people have the experience of trying to come up with a croptographic scheme and failing, meanwhile there are perfectly good libraries nobody has found exploits to yet?
That doesn’t seem very analogous with metaethics. As you say, it is hard to decisively show a metaethical theory is “wrong”, and as far as I know there is no well-studied metaethical theory which has no exploits yet.
The analogy is that in both fields people are by default very prone to being overconfident. In cryptography this can be seen by the phenomenon of people (especially newcomers who haven’t learned the lesson) confidently proposing new cryptographic algorithms, which end up being way easier to break than they expect. In philosophy this is a bit trickier to demonstrate, but I think can be seen via a combination of:
people confidently holding positions that are incompatible with other people’s confident positions
tendency to “bite bullets” or accepting implications that are highly counterintuitive to others or even to themselves, instead of adopting more uncertainty
the total idea/argument space being exponentially vast and underexplored due to human limitations, therefore high confidence being unjustified in light of this
At risk of committing a Bulverism, I’ve noticed a tendency for people to see ethical bullet-biting as epistemically virtuous, like a demonstration of how rational/unswayed by emotion you are (biasing them to overconfidently bullet-bite). However, this makes less sense in ethics where intuitions like repugnance are a large proportion of what everything is based on in the first place.
the total idea/argument space being exponentially vast and underexplored due to human limitations, therefore high confidence being unjustified in light of this
There’s also the thing that the idea/argument space contains dæmons/attractors exploiting shortcomings of human cognition, thus making humans hold them with higher confidence than they would if they didn’t have those limitations.
tendency to “bite bullets” or accepting implications that are highly counterintuitive to others or even to themselves, instead of adopting more uncertainty
I find this contrast between “biting bullets” and “adopting more uncertainty” strange. The two seem orthogonal to me, as in, I’ve ~just as frequently (if not more often) observed people overconfidently endorse their pretheoretic philosophical intuitions, in opposition to bullet-biting.
In my experience learning the viscereal sense that the space is dense with traps and spiders and poisonous things and what intuitively seems “basically sensible” often does not work. (I did some cryptography years ago)
The structural similarity seems to be there is a big difference in trying to do cryptography in a mode where you don’t assume what you are doing is subject to some adversarial pressure, and in the mode where it should work even if someone tries to attack it. The first one is easy, breaks easily, and it’s unclear why would you even try to do it.
In metaethics, I think it is somewhat easy to do it in the mode where you don’t assume it should be applied in some high-stakes, novel or tricky situations, like AI alignment, computer minds, multiverse, population ethics, anthropics, etc etc. The suggestions of normative ethical theories will converge for many mundane situations, so anything works, but it was not necessary to do metaethics.
I have never done cryptography, but the way I imagine working in it is that it exists in a context of extremely resourceful adversarial agents, and thus you have to give up a kind of casual, not quite noticed neglect toward extremely weird and artificial-sounding edge cases / seemingly weird and unlikely scenarios, because this is where the danger lives: your adversaries may force these weird edge cases to happen, and this is a part of the system’s behavior you haven’t sufficiently thought through.
Maybe one possible analogy with AI alignment, at least, is that we’re also talking about potential extremely resourceful agents that are adversarial until we’ve actually solved alignment, so we’re not allowed to treat weird hypothetical scenarios as unlikely edge cases and say “Come on, that’s way too far-fetched, how would it even do that?”, because it’s like pointing to a hole in a ship’s hull and saying “What are the odds the water molecules would decide to go through this hole? The ship is so big!”
I think this fails to say how the analogy of cryptography transfers to metaethics. What properties of cryptography as a field make it such that you cannot roll your own. Is it just that many people have the experience of trying to come up with a croptographic scheme and failing, meanwhile there are perfectly good libraries nobody has found exploits to yet?
That doesn’t seem very analogous with metaethics. As you say, it is hard to decisively show a metaethical theory is “wrong”, and as far as I know there is no well-studied metaethical theory which has no exploits yet.
So what exactly is the analogy?
The analogy is that in both fields people are by default very prone to being overconfident. In cryptography this can be seen by the phenomenon of people (especially newcomers who haven’t learned the lesson) confidently proposing new cryptographic algorithms, which end up being way easier to break than they expect. In philosophy this is a bit trickier to demonstrate, but I think can be seen via a combination of:
people confidently holding positions that are incompatible with other people’s confident positions
tendency to “bite bullets” or accepting implications that are highly counterintuitive to others or even to themselves, instead of adopting more uncertainty
the total idea/argument space being exponentially vast and underexplored due to human limitations, therefore high confidence being unjustified in light of this
At risk of committing a Bulverism, I’ve noticed a tendency for people to see ethical bullet-biting as epistemically virtuous, like a demonstration of how rational/unswayed by emotion you are (biasing them to overconfidently bullet-bite). However, this makes less sense in ethics where intuitions like repugnance are a large proportion of what everything is based on in the first place.
There’s also the thing that the idea/argument space contains dæmons/attractors exploiting shortcomings of human cognition, thus making humans hold them with higher confidence than they would if they didn’t have those limitations.
I find this contrast between “biting bullets” and “adopting more uncertainty” strange. The two seem orthogonal to me, as in, I’ve ~just as frequently (if not more often) observed people overconfidently endorse their pretheoretic philosophical intuitions, in opposition to bullet-biting.
In my experience learning the viscereal sense that the space is dense with traps and spiders and poisonous things and what intuitively seems “basically sensible” often does not work. (I did some cryptography years ago)
The structural similarity seems to be there is a big difference in trying to do cryptography in a mode where you don’t assume what you are doing is subject to some adversarial pressure, and in the mode where it should work even if someone tries to attack it. The first one is easy, breaks easily, and it’s unclear why would you even try to do it.
In metaethics, I think it is somewhat easy to do it in the mode where you don’t assume it should be applied in some high-stakes, novel or tricky situations, like AI alignment, computer minds, multiverse, population ethics, anthropics, etc etc. The suggestions of normative ethical theories will converge for many mundane situations, so anything works, but it was not necessary to do metaethics.
I have never done cryptography, but the way I imagine working in it is that it exists in a context of extremely resourceful adversarial agents, and thus you have to give up a kind of casual, not quite noticed neglect toward extremely weird and artificial-sounding edge cases / seemingly weird and unlikely scenarios, because this is where the danger lives: your adversaries may force these weird edge cases to happen, and this is a part of the system’s behavior you haven’t sufficiently thought through.
Maybe one possible analogy with AI alignment, at least, is that we’re also talking about potential extremely resourceful agents that are adversarial until we’ve actually solved alignment, so we’re not allowed to treat weird hypothetical scenarios as unlikely edge cases and say “Come on, that’s way too far-fetched, how would it even do that?”, because it’s like pointing to a hole in a ship’s hull and saying “What are the odds the water molecules would decide to go through this hole? The ship is so big!”