My short answer to the conundrum is that if the first thing your tool does is destroy itself, the tool is defective. That doesn’t make “rationality” defective any more than crashing your first attempt at building a car implies that “The Car” is defective.
Designing foundations for human intelligence is rather like designing foundations for artificial (general) intelligence in this respect. (I don’t know if you’ve looked at The Sequences yet, but it has a lot of material on the common fallacies the latter enterprise has often fallen into, fallacies that apply to everyday thinking as well.) That people, on the whole, do not go crazy — at least, not as crazy as the tool that blows itself up as soon as you turn it on — is a proof by example that not going crazy is possible. If your hypothetical system of thought immediately goes crazy, the design is wrong. The idea is to do better at thinking than the general run of what we can see around us. Again, we have a proof by example that this is possible: some people do think better than the general run.
Well, it sounds right. But which mistake in rationality was done in that described situation, and how can it be improved? My first idea was that there are things we shouldn’t doubt… But it is kind of dogmatic and feels wrong. So should it maybe be like “Before doubting X think of what will you become if you succeed, and take it into consideration before actually trying to doubt X”. But this still implies “There are cases when you shouldn’t doubt”, which is still suspicious and doesn’t sound “rational”. I mean, doesn’t sound like making the map reflect the territory.
It’s like repairing the foundations of a building. You can’t uproot all of them, but you can uproot any of them, as long as you take care that the building doesn’t fall down during renovations.
Welcome to Less Wrong!
My short answer to the conundrum is that if the first thing your tool does is destroy itself, the tool is defective. That doesn’t make “rationality” defective any more than crashing your first attempt at building a car implies that “The Car” is defective.
Designing foundations for human intelligence is rather like designing foundations for artificial (general) intelligence in this respect. (I don’t know if you’ve looked at The Sequences yet, but it has a lot of material on the common fallacies the latter enterprise has often fallen into, fallacies that apply to everyday thinking as well.) That people, on the whole, do not go crazy — at least, not as crazy as the tool that blows itself up as soon as you turn it on — is a proof by example that not going crazy is possible. If your hypothetical system of thought immediately goes crazy, the design is wrong. The idea is to do better at thinking than the general run of what we can see around us. Again, we have a proof by example that this is possible: some people do think better than the general run.
Well, it sounds right. But which mistake in rationality was done in that described situation, and how can it be improved? My first idea was that there are things we shouldn’t doubt… But it is kind of dogmatic and feels wrong. So should it maybe be like “Before doubting X think of what will you become if you succeed, and take it into consideration before actually trying to doubt X”. But this still implies “There are cases when you shouldn’t doubt”, which is still suspicious and doesn’t sound “rational”. I mean, doesn’t sound like making the map reflect the territory.
It’s like repairing the foundations of a building. You can’t uproot all of them, but you can uproot any of them, as long as you take care that the building doesn’t fall down during renovations.