I can test reliability of win by taking a bunch of different problems with known answers that I don’t know, solving them using my current conception of rationality and solving them using the alternative conception of rationality I want to test, then checking the answers I arrived at with each conception against the right answers.
You have just limited yourself to only solving problems others have solved before (and trusting their answers). Furthermore by restricting yourself to only the category of problems with known answers you will find that you can develop heuristics that don’t apply to general problems. In particular the most effective way to solve a problem with a known answer is usually to track down that answer.
Edit: Also you’re restricting yourself to the class of problems with well-defined ‘right answers’, that’s a very restricted class of problems.
See end of second to last paragraph, I just added that before I read your comment, sorry bout that.
But regardless: I can make the problems I win very diverse, even if they are all solved problems. I think that takes care of the non-general heuristic problem. And of course, there’s a reason I specify that I’m not allowed to know the answer.
And of course, there’s a reason I specify that I’m not allowed to know the answer.
What about finding someone who does and asking them for hints?
But regardless: I can make the problems I win very diverse, even if they are all solved problems.
The probability distribution over the types of problems you encounter will still be different. For example, if a problem was a known solution, it can be solved using a known method so a good strategy is to try various known methods until one works. Whereas if a problem is unsolved especially if it has been unsolved for a long time, that’s a sign that known methods won’t work on it, so you’ll need to develop new methods.
A useful analogy, courtesy of the martial art of rationality, is the difference between a ‘friendly fight’ and a real fight, i.e., one where each opponent is trying to seriously injure or kill the other. Just as most one must practice in friendly fights before one is ready for a real fight, one must practice on problems with known solutions before one is ready to attack unsolved problems. However, without real fights to keep people honest you get schools proliferating without evidence.
Yea, I should try to have as few hints as possible from the science that’s already solved it. But even then, I won’t find new methods by working on old problems, nearly as often as if I’m working on new problems. So, you’ve shown that you’ll improve your conception of rationality faster by solving both solved and unsolved problems, but I’m still floating. I’ll edit the second paragraph to take your point into account, thanks.
You have just limited yourself to only solving problems others have solved before (and trusting their answers). Furthermore by restricting yourself to only the category of problems with known answers you will find that you can develop heuristics that don’t apply to general problems. In particular the most effective way to solve a problem with a known answer is usually to track down that answer.
Edit: Also you’re restricting yourself to the class of problems with well-defined ‘right answers’, that’s a very restricted class of problems.
See end of second to last paragraph, I just added that before I read your comment, sorry bout that.
But regardless: I can make the problems I win very diverse, even if they are all solved problems. I think that takes care of the non-general heuristic problem. And of course, there’s a reason I specify that I’m not allowed to know the answer.
What about finding someone who does and asking them for hints?
The probability distribution over the types of problems you encounter will still be different. For example, if a problem was a known solution, it can be solved using a known method so a good strategy is to try various known methods until one works. Whereas if a problem is unsolved especially if it has been unsolved for a long time, that’s a sign that known methods won’t work on it, so you’ll need to develop new methods.
A useful analogy, courtesy of the martial art of rationality, is the difference between a ‘friendly fight’ and a real fight, i.e., one where each opponent is trying to seriously injure or kill the other. Just as most one must practice in friendly fights before one is ready for a real fight, one must practice on problems with known solutions before one is ready to attack unsolved problems. However, without real fights to keep people honest you get schools proliferating without evidence.
Great use of LW metaphors.
Yea, I should try to have as few hints as possible from the science that’s already solved it. But even then, I won’t find new methods by working on old problems, nearly as often as if I’m working on new problems. So, you’ve shown that you’ll improve your conception of rationality faster by solving both solved and unsolved problems, but I’m still floating. I’ll edit the second paragraph to take your point into account, thanks.