I read your last section (“Note general failure mode: …”) with amusement as I have found myself following almost the exact train of thought several times recently.
It was an appreciated, although unpleasant, kick-in-the-teeth to realise that my thought process actually belied negative aspects to my character rather than positive ones.
Could I ask for advice then on reversing this situation? What internal monologue, or indeed actions, should be ideally followed based on a situation identical to the one given in the article.
There seem to be far too many people hung up on the mathematics which ignores the purpose of the post as I understand it.
The post is not about truth but about conviction. Eliezer is not saying that there could be a scenario in which the rules of mathematics didn’t work, but that there could be a scenario under which he was convinced of it.
Deconstructing all elements of neurology, physics and socialogy that make up the pathway from complete ignorance to solid conviction is not something I could even begin to attempt—but if one were able to list such steps as a series bullet points I could conceive that the manipulation of certain steps could lead to a different outcome, which appears to me to be the ultimate point of the post (although not hugely ground-breaking, but an interesting thought experiment).
It is not a claim that the strongly held conviction represents fact or that the conviction would not be shaken by a future event or presentation of evidence. As a fundamental believer in scientific thought and rationality there is much that I hold as firm conviction that I would not hesitate to re-think under valid contradictory evidence.