Just to be clear: by strict rules I don’t mean anything with significant subjective judgement involved, like peer reviews. I rather mean things like demanding testability, mathematical proofs, logical consistency and such. Also, not much rules governing the social life of the respective community, but rather rules applied to the hypotheses.
Also, I haven’t said that rules are sufficient. One can still publish trivial theories which nobody is interested to test, mathematical proofs of obscure unimportant theorems or logically consistent tautologies. But at least the rules remove arbitrariness and make it possible to objectively assess quality and to decide whether a hypothesis is good or bad, according to standards of the discipline.
The discipline’s standard of good hypothesis may not universally correspond to a true hypothesis, but I suspect that if the standards of the discipline are strict enough, either the correspondence is there, or it is easily visible that the discipline is based on wrong premises, because it endorses some easily identifiable falsehoods. (It would be too big a coincidence if a formal system regularly produced false statements, but no trivially false statements.)
On the other hand, when the rules aren’t enough formal, the discipline still makes complex false claims, but nobody can clearly demonstrate that their methods are unreliable because the methods (if there are any) can be always flexed to avoid producing embarrassingly trivial errors.
Just to be clear: by strict rules I don’t mean anything with significant subjective judgement involved, like peer reviews. I rather mean things like demanding testability, mathematical proofs, logical consistency and such. Also, not much rules governing the social life of the respective community, but rather rules applied to the hypotheses.
Trouble is, there are examples of fields where the standards satisfy all this, but the work is nevertheless misleading and remote from reality.
Take the example of computer science, which I’m most familiar with. In some of its subfields, the state of the art has reached a dead end, in that any obvious path for improving things hits against some sort of exponential-time or uncomputable problem, and the possible heuristics for getting around it have already been explored to death. Breaking a new path in this situation could be done only by an extraordinary stroke of genius, if it’s possible at all.
So what people do is to propose yet another complex and sophisticated but ultimately feeble heuristic wrapped into thick layers of abstruse math, and argue that it represents an improvement of some performance measure by a few percentage points. Now, if you look at a typical paper from such an area, you’ll see that the formalism is accurate mathematically and logically, the performance evaluation is carefully measured over a set of standard benchmarks according to established guidelines, and the relevant prior work is meticulously researched and cited. You have to satisfy these strict formal standards to publish.
Trouble is, nearly all this work is worthless, and quite obviously so. From a practical engineering perspective, implementing these complex algorithms in a practical system would be a Herculean task for a minuscule gain. The hypertrophied formalism often uses numerous pages of abstruse math to express ideas that could be explained intuitively and informally in a few simple sentences to someone knowledgeable in the field—and in turn would be immediately and correctly dismissed as impractical. Even the measured performance improvements are rarely evaluated truly ceteris paribus and in ways that reveal all the strengths and weaknesses of the approach. It’s simply impossible to devise a formal standard that wold ensure that reliably—these things are possible to figure out only with additional experimentation or with a practical engineering hunch.
Except perhaps in the purest mathematics, no formal standard can function well in practice if legions of extraordinarily smart people have the incentive to get around it. And if there are no easy paths to quality work, the “publish or perish” principle makes it impossible to compete and survive unless one exerts every effort to game the system.
That’s right, and I don’t disagree. Formal standards are not a panacea, never. But, do you suppose, in cases you describe, things would go better without those formal standards?
I am still not sure if we mean exactly the same thing, when talking about formal rules. Take the example of pure mathematics, which you have already mentioned. Surely, abstruse formalist descriptions of practically uninteresting and maybe trivial problems appear there too, now and then. And revolutionary breakthroughs perhaps more often result from intuitive insights of geniuses than from dilligent rigorous formal work. Much papers, in all fields, can be made more readable, accessible, and effective in dissemination of new results by shedding the lofty jargon of scientific publications. But mathematicians certainly wouldn’t do better if they got rid of mathematical proofs.
I do not suggest that all ideas in respectable fields of science should be propagated in form of publications checked against lists of formal requirements: citation index, proofs of all logical statements, p-values below 0.01, certificates of double-blindedness. Not in the slightest. Conjectures, analogies, illustrations, whatever enhances understanding is welcome. I only want a possibility to apply the formal criteria. If a conjecture is published, and it turns out interesting, there should be an ultimate method to test whether it is true. If there is an agreed method to test the results objectively, people aren’t free to publish whatever they want and expect to never be proven wrong.
If you compare the results of computer science to postmodern philosophy, you may see my point. In CS most results may be useless and incomprehensible. In postmodern philosophy, which is essentially without formal rules, all results are useless and incomprehensible, and as a bonus, meaningless or false.
I agree about the awful state of fields that don’t have any formal rules at all. However, I’m not concerned about these so much because, to put it bluntly, nobody important takes them seriously. What is in my opinion a much greater problem are fields that appear to have all the trappings of valid science and scholarship, but it’s in fact hard for an outsider to evaluate whether and to what extent they’re actually cargo-cult science. This especially because the results of some such fields (most notably economics) are used as basis for real-world decision-making with far-reaching consequences.
Regarding the role of formalism, mathematics is unique in that the internal correctness of the formalism is enough to establish the validity of the results. Sure, they may be more or less interesting, but if the formalism is valid, then it’s valid math, period.
In contrast, in areas that make claims about the real world, the important thing is not just the validity of the formalism, but also how well it corresponds to reality. Work based on a logically impeccable formalism can still be misleading garbage if the formalism is distant enough from reality. This is where the really hard problem is. The requirements about the validity of the formalism are easily enforced, since we know how to reduce those to a basically algorithmic procedure. What is really hard is ensuring that the formalism provides an accurate enough description of reality—and given an incentive to do so, smart people will inevitably figure out ways to stretch and evade this requirement, unless there is a sound common-sense judgment standing in the way.
Further, more rigorous formalism isn’t always better. It’s a trade-off. More effort put into greater formal rigor—including both the author’s effort to formulate it, and the reader’s effort to understand it—means less resources for other ways of improving the work. Physicists, for example, normally just assume that the functions are well-behaved enough in a way that would be unacceptable in mathematics, and they’re justified in doing so. In more practical technical fields like computer science, what matters is whether the results are useful in practice, and formal rigor is useful if it helps avoid confusion about complicated things, but worse than useless if applied to things where intuitive understanding is good enough to get the job done.
The crucial lesson, like in so many other things, is that whenever one deals with the real world, formalism cannot substitute for common sense. It may be tremendously helpful and enable otherwise impossible breakthroughs, but without an ultimate sanity check based on sheer common sense, any attempt at science is a house built on sand.
I don’t think we have a real disagreement. I haven’t said that more rigorous formalism is always better, quite the contrary. I was writing about objective methods of looking at the results. Physicists can ignore mathematical rigor because they have experimental tests which finally decide whether their theory is worth attention. Computer scientists can finally write down their algorithm and look whether it works. These are objective rules which validate the results.
Whether the rules are sensible or not is decided by common sense. My point is that it is easier to decide that about the rules of the whole field than about individual theories, and that’s why objective rules are useful.
Of course, saying “common sense” does in fact mean that we don’t know how did we decide, and doesn’t specify the judgement too much. One man’s common sense may be other man’s insanity.
Oh yes, I didn’t mean to imply that you disagreed with everything I wrote in the above comment. My intent was to give a self-contained summary of my position on the issue, and the specific points I raised were not necessarily in response to your claims.
Just to be clear: by strict rules I don’t mean anything with significant subjective judgement involved, like peer reviews. I rather mean things like demanding testability, mathematical proofs, logical consistency and such. Also, not much rules governing the social life of the respective community, but rather rules applied to the hypotheses.
Also, I haven’t said that rules are sufficient. One can still publish trivial theories which nobody is interested to test, mathematical proofs of obscure unimportant theorems or logically consistent tautologies. But at least the rules remove arbitrariness and make it possible to objectively assess quality and to decide whether a hypothesis is good or bad, according to standards of the discipline.
The discipline’s standard of good hypothesis may not universally correspond to a true hypothesis, but I suspect that if the standards of the discipline are strict enough, either the correspondence is there, or it is easily visible that the discipline is based on wrong premises, because it endorses some easily identifiable falsehoods. (It would be too big a coincidence if a formal system regularly produced false statements, but no trivially false statements.)
On the other hand, when the rules aren’t enough formal, the discipline still makes complex false claims, but nobody can clearly demonstrate that their methods are unreliable because the methods (if there are any) can be always flexed to avoid producing embarrassingly trivial errors.
prase:
Trouble is, there are examples of fields where the standards satisfy all this, but the work is nevertheless misleading and remote from reality.
Take the example of computer science, which I’m most familiar with. In some of its subfields, the state of the art has reached a dead end, in that any obvious path for improving things hits against some sort of exponential-time or uncomputable problem, and the possible heuristics for getting around it have already been explored to death. Breaking a new path in this situation could be done only by an extraordinary stroke of genius, if it’s possible at all.
So what people do is to propose yet another complex and sophisticated but ultimately feeble heuristic wrapped into thick layers of abstruse math, and argue that it represents an improvement of some performance measure by a few percentage points. Now, if you look at a typical paper from such an area, you’ll see that the formalism is accurate mathematically and logically, the performance evaluation is carefully measured over a set of standard benchmarks according to established guidelines, and the relevant prior work is meticulously researched and cited. You have to satisfy these strict formal standards to publish.
Trouble is, nearly all this work is worthless, and quite obviously so. From a practical engineering perspective, implementing these complex algorithms in a practical system would be a Herculean task for a minuscule gain. The hypertrophied formalism often uses numerous pages of abstruse math to express ideas that could be explained intuitively and informally in a few simple sentences to someone knowledgeable in the field—and in turn would be immediately and correctly dismissed as impractical. Even the measured performance improvements are rarely evaluated truly ceteris paribus and in ways that reveal all the strengths and weaknesses of the approach. It’s simply impossible to devise a formal standard that wold ensure that reliably—these things are possible to figure out only with additional experimentation or with a practical engineering hunch.
Except perhaps in the purest mathematics, no formal standard can function well in practice if legions of extraordinarily smart people have the incentive to get around it. And if there are no easy paths to quality work, the “publish or perish” principle makes it impossible to compete and survive unless one exerts every effort to game the system.
That’s right, and I don’t disagree. Formal standards are not a panacea, never. But, do you suppose, in cases you describe, things would go better without those formal standards?
I am still not sure if we mean exactly the same thing, when talking about formal rules. Take the example of pure mathematics, which you have already mentioned. Surely, abstruse formalist descriptions of practically uninteresting and maybe trivial problems appear there too, now and then. And revolutionary breakthroughs perhaps more often result from intuitive insights of geniuses than from dilligent rigorous formal work. Much papers, in all fields, can be made more readable, accessible, and effective in dissemination of new results by shedding the lofty jargon of scientific publications. But mathematicians certainly wouldn’t do better if they got rid of mathematical proofs.
I do not suggest that all ideas in respectable fields of science should be propagated in form of publications checked against lists of formal requirements: citation index, proofs of all logical statements, p-values below 0.01, certificates of double-blindedness. Not in the slightest. Conjectures, analogies, illustrations, whatever enhances understanding is welcome. I only want a possibility to apply the formal criteria. If a conjecture is published, and it turns out interesting, there should be an ultimate method to test whether it is true. If there is an agreed method to test the results objectively, people aren’t free to publish whatever they want and expect to never be proven wrong.
If you compare the results of computer science to postmodern philosophy, you may see my point. In CS most results may be useless and incomprehensible. In postmodern philosophy, which is essentially without formal rules, all results are useless and incomprehensible, and as a bonus, meaningless or false.
I agree about the awful state of fields that don’t have any formal rules at all. However, I’m not concerned about these so much because, to put it bluntly, nobody important takes them seriously. What is in my opinion a much greater problem are fields that appear to have all the trappings of valid science and scholarship, but it’s in fact hard for an outsider to evaluate whether and to what extent they’re actually cargo-cult science. This especially because the results of some such fields (most notably economics) are used as basis for real-world decision-making with far-reaching consequences.
Regarding the role of formalism, mathematics is unique in that the internal correctness of the formalism is enough to establish the validity of the results. Sure, they may be more or less interesting, but if the formalism is valid, then it’s valid math, period.
In contrast, in areas that make claims about the real world, the important thing is not just the validity of the formalism, but also how well it corresponds to reality. Work based on a logically impeccable formalism can still be misleading garbage if the formalism is distant enough from reality. This is where the really hard problem is. The requirements about the validity of the formalism are easily enforced, since we know how to reduce those to a basically algorithmic procedure. What is really hard is ensuring that the formalism provides an accurate enough description of reality—and given an incentive to do so, smart people will inevitably figure out ways to stretch and evade this requirement, unless there is a sound common-sense judgment standing in the way.
Further, more rigorous formalism isn’t always better. It’s a trade-off. More effort put into greater formal rigor—including both the author’s effort to formulate it, and the reader’s effort to understand it—means less resources for other ways of improving the work. Physicists, for example, normally just assume that the functions are well-behaved enough in a way that would be unacceptable in mathematics, and they’re justified in doing so. In more practical technical fields like computer science, what matters is whether the results are useful in practice, and formal rigor is useful if it helps avoid confusion about complicated things, but worse than useless if applied to things where intuitive understanding is good enough to get the job done.
The crucial lesson, like in so many other things, is that whenever one deals with the real world, formalism cannot substitute for common sense. It may be tremendously helpful and enable otherwise impossible breakthroughs, but without an ultimate sanity check based on sheer common sense, any attempt at science is a house built on sand.
I don’t think we have a real disagreement. I haven’t said that more rigorous formalism is always better, quite the contrary. I was writing about objective methods of looking at the results. Physicists can ignore mathematical rigor because they have experimental tests which finally decide whether their theory is worth attention. Computer scientists can finally write down their algorithm and look whether it works. These are objective rules which validate the results.
Whether the rules are sensible or not is decided by common sense. My point is that it is easier to decide that about the rules of the whole field than about individual theories, and that’s why objective rules are useful.
Of course, saying “common sense” does in fact mean that we don’t know how did we decide, and doesn’t specify the judgement too much. One man’s common sense may be other man’s insanity.
Oh yes, I didn’t mean to imply that you disagreed with everything I wrote in the above comment. My intent was to give a self-contained summary of my position on the issue, and the specific points I raised were not necessarily in response to your claims.