If you pursue a goal where you have a very low chance of success, it is true that believing that Heaven’s armies are behind you might increase your chances a little bit. But it also has a negative side. For one, it increases the chance that your failure will be truly spectacular, as you dig yourself much deeper than you otherwise would. Even more commonly, you will waste resources: you will invest or spend things you would have started saving or protecting if you had a more realistic view of your chances.
Overconfidence is a pure gamble: small chance of jackpot, much bigger chance of losing your shirt to the house.
If you must go for a long-shot project, you should do so only if the cost of failure isn’t too high (i.e. “if it works, fantastic; if it doesn’t, oh well”), and then be as realistic as possible about your chances throughout the whole process. This can be fun, it can help you expand your horizons, and train your skills. But the “limited cost of failure” and “no overconfidence” caveats are critical.
As for objective ways to calibrate your confidence, yes, there is a lot of useful information. I’m currently writing a broad summary of our current understanding of motivation, in psychology and neuroscience. It should be posted in a few weeks, as a sequence (the thing will be bloody long, there is a lot of ground to cover), and it will touch upon this point more than once. But here, I’ll adapt a bit that deals with the concept of self-efficacy, which I think may help answer your question.
There are many possible, or even desirable goals we can go for, and our brain has to prioritize. We will be motivated to pursue some goals, and unmotivated to pursue others. This is not a simple function of desirability. The perceived difficulty of the goal matters, as does your perception of your abilities. Your mind will take both into account before deciding where to invest motivation.
For example, John Smith considers an acting career in Hollywood highly desirable; but the goal is extremely difficult to achieve, and he perceives his acting skills as average. Therefore, he may occasionally daydream about being an actor, but he won’t be motivated to actually pursue acting. Jane Smith, on the other hand, also finds that goal highly desirable; she also correctly sees the goal as extremely difficult. But she also perceives her acting skills as extremely good. So she moves to LA, hires an agent, and starts waiting tables while she waits for her break. She is highly motivated to pursue acting.
John and Jane have the same goal and the same outlook upon that goal. The difference in motivation comes entirely from their perception of their own abilities. John has low self-efficacy in regard to acting. Jane has high self-efficacy in regard to acting.
This applies to all areas. If we have high self-efficacy, we have more motivation to pursue a goal, we are more likely to keep trying after encountering obstacles and setbacks, and we are likely to be more productive while working towards that goal. We also feel generally better when we work in a field where our self-efficacy is high. Low self-efficacy, conversely, produces feeling of insecurity and depression.
But higher self-efficacy isn’t always better. Where a person with too low self-efficacy fails to achieve an attainable goal, a person with unrealistically high self-efficacy is likely to try for an unattainable one – and quite predictably fail. Incorrect calibration of self-efficacy also causes attribution problems. A person with low self-efficacy will attribute failure to personal lack of ability, even if failure occurred because of external factors. At the same time, a high self-efficacy person will attribute failure to external factors, even if it occurred due to insufficient skill or effort.
If we make a reasonable assumption (based on most current research) that the best, highest-growth goals are those on the very edge of our ability, it follows that the “optimal” self-efficacy should be tuned to just above the very limit of our ability. We should, in other words, believe that we can do just a little bit more than we actually can.
Paradoxically, while good, accurately calibrated self-efficacy helps motivation in its own narrow area, it can lead to significant problems elsewhere. We achieve expertise, and build up a good understanding of our abilities. In the process, we become used to the feeling of confidence while solving difficult, expert-level problems; feelings of incompetence and anxiety become much harder to bear. New challenges become demotivating, since they force us to encounter unfamiliar obstacles which we may or may not actually overcome. If failure occurs, it will compromise the picture of the competent, successful person we imagine ourselves to be. So we just focus on doing familiar things within our area, and we start avoiding new challenges and opportunities for growth.
This can be a major problem, especially common among those who have achieved extremely high levels of expertise in highly demanding fields. For example, it affects many people who work in academic research, regardless of the field. In our training, we spend years to become absolute experts in the use of a few particular (and quite difficult) experimental techniques. When we then encounter a new problem, there is an inclination against looking for the best possible approach; instead, we tend to try and solve it through convoluted and indirect ways of applying techniques we are already familiar with. I have had to drag myself out of this cognitive-motivational trap more than once
If you pursue a goal where you have a very low chance of success, it is true that believing that Heaven’s armies are behind you might increase your chances a little bit. But it also has a negative side. For one, it increases the chance that your failure will be truly spectacular, as you dig yourself much deeper than you otherwise would. Even more commonly, you will waste resources: you will invest or spend things you would have started saving or protecting if you had a more realistic view of your chances.
Overconfidence is a pure gamble: small chance of jackpot, much bigger chance of losing your shirt to the house.
If you must go for a long-shot project, you should do so only if the cost of failure isn’t too high (i.e. “if it works, fantastic; if it doesn’t, oh well”), and then be as realistic as possible about your chances throughout the whole process. This can be fun, it can help you expand your horizons, and train your skills. But the “limited cost of failure” and “no overconfidence” caveats are critical.
As for objective ways to calibrate your confidence, yes, there is a lot of useful information. I’m currently writing a broad summary of our current understanding of motivation, in psychology and neuroscience. It should be posted in a few weeks, as a sequence (the thing will be bloody long, there is a lot of ground to cover), and it will touch upon this point more than once. But here, I’ll adapt a bit that deals with the concept of self-efficacy, which I think may help answer your question.
There are many possible, or even desirable goals we can go for, and our brain has to prioritize. We will be motivated to pursue some goals, and unmotivated to pursue others. This is not a simple function of desirability. The perceived difficulty of the goal matters, as does your perception of your abilities. Your mind will take both into account before deciding where to invest motivation.
For example, John Smith considers an acting career in Hollywood highly desirable; but the goal is extremely difficult to achieve, and he perceives his acting skills as average. Therefore, he may occasionally daydream about being an actor, but he won’t be motivated to actually pursue acting. Jane Smith, on the other hand, also finds that goal highly desirable; she also correctly sees the goal as extremely difficult. But she also perceives her acting skills as extremely good. So she moves to LA, hires an agent, and starts waiting tables while she waits for her break. She is highly motivated to pursue acting.
John and Jane have the same goal and the same outlook upon that goal. The difference in motivation comes entirely from their perception of their own abilities. John has low self-efficacy in regard to acting. Jane has high self-efficacy in regard to acting.
This applies to all areas. If we have high self-efficacy, we have more motivation to pursue a goal, we are more likely to keep trying after encountering obstacles and setbacks, and we are likely to be more productive while working towards that goal. We also feel generally better when we work in a field where our self-efficacy is high. Low self-efficacy, conversely, produces feeling of insecurity and depression.
But higher self-efficacy isn’t always better. Where a person with too low self-efficacy fails to achieve an attainable goal, a person with unrealistically high self-efficacy is likely to try for an unattainable one – and quite predictably fail. Incorrect calibration of self-efficacy also causes attribution problems. A person with low self-efficacy will attribute failure to personal lack of ability, even if failure occurred because of external factors. At the same time, a high self-efficacy person will attribute failure to external factors, even if it occurred due to insufficient skill or effort.
If we make a reasonable assumption (based on most current research) that the best, highest-growth goals are those on the very edge of our ability, it follows that the “optimal” self-efficacy should be tuned to just above the very limit of our ability. We should, in other words, believe that we can do just a little bit more than we actually can.
Paradoxically, while good, accurately calibrated self-efficacy helps motivation in its own narrow area, it can lead to significant problems elsewhere. We achieve expertise, and build up a good understanding of our abilities. In the process, we become used to the feeling of confidence while solving difficult, expert-level problems; feelings of incompetence and anxiety become much harder to bear. New challenges become demotivating, since they force us to encounter unfamiliar obstacles which we may or may not actually overcome. If failure occurs, it will compromise the picture of the competent, successful person we imagine ourselves to be. So we just focus on doing familiar things within our area, and we start avoiding new challenges and opportunities for growth.
This can be a major problem, especially common among those who have achieved extremely high levels of expertise in highly demanding fields. For example, it affects many people who work in academic research, regardless of the field. In our training, we spend years to become absolute experts in the use of a few particular (and quite difficult) experimental techniques. When we then encounter a new problem, there is an inclination against looking for the best possible approach; instead, we tend to try and solve it through convoluted and indirect ways of applying techniques we are already familiar with. I have had to drag myself out of this cognitive-motivational trap more than once