If there are patterns in your thinking that are consistently causing you to think things that are not true, metacognition is the general tool by which you can notice that and try to correct the situation.
To be more specific, I can very easily imagine AI researchers not believing that AI safety is an issue due to something like cognitive dissonance: if they admitted that AI safety was an issue, they’d be admitting that what they’re working on is dangerous and maybe shouldn’t be worked on, which contradicts their desire to work on it. The easiest way to resolve the cognitive dissonance, and the most socially acceptable way barring people like Stuart Russell publicly pumping in the other direction, is to dismiss the concern as Luddite fear-mongering. This is the sort of thing you can try to notice and correct about yourself with the right metacognitive tools.
To make another analogy with math, I have never once heard a mathematics graduate student or professor speculate, publicly or privately, about the extent to which pure mathematics is mostly useless and overfunded. This is unsayable among mathematicians, maybe even unthinkable.
If there are patterns in your thinking that are consistently causing you to think things that are not true, metacognition is the general tool by which you can notice that and try to correct the situatio
And it there isnt that problem, there is no need for that solution. For your argument to go through, you need to
show that people likely to be impactive on AI safety are likely to have cognitive problems that affect them when they are doing AI safety. (Saying something like “academics are irrational because some of them believe in God” isn’t enough.” Compartmentalised beliefs are unimpactive because compartmentalised. Instrumental rationality is not epistemic rationality ).
To be more specific, I can very easily imagine AI researchers not believing that AI safety is an issue due to something like cognitive dissonance:
if they admitted that AI safety was an issue, they’d be admitting that what they’re working on is dangerous and maybe shouldn’t be worked on, which contradicts their desire to work on it.
I can easily imagine an AI safety researcher maintaining a false belief that AI safety is a huge deal, because if they didn’t they would be a nobody working on a non-problem. Funny how you can make logic run in more than one direction.
Can you answer the question : why is metacognition needed for AI safety?
If there are patterns in your thinking that are consistently causing you to think things that are not true, metacognition is the general tool by which you can notice that and try to correct the situation.
To be more specific, I can very easily imagine AI researchers not believing that AI safety is an issue due to something like cognitive dissonance: if they admitted that AI safety was an issue, they’d be admitting that what they’re working on is dangerous and maybe shouldn’t be worked on, which contradicts their desire to work on it. The easiest way to resolve the cognitive dissonance, and the most socially acceptable way barring people like Stuart Russell publicly pumping in the other direction, is to dismiss the concern as Luddite fear-mongering. This is the sort of thing you can try to notice and correct about yourself with the right metacognitive tools.
To make another analogy with math, I have never once heard a mathematics graduate student or professor speculate, publicly or privately, about the extent to which pure mathematics is mostly useless and overfunded. This is unsayable among mathematicians, maybe even unthinkable.
And it there isnt that problem, there is no need for that solution. For your argument to go through, you need to show that people likely to be impactive on AI safety are likely to have cognitive problems that affect them when they are doing AI safety. (Saying something like “academics are irrational because some of them believe in God” isn’t enough.” Compartmentalised beliefs are unimpactive because compartmentalised. Instrumental rationality is not epistemic rationality ).
I dare say
I can easily imagine an AI safety researcher maintaining a false belief that AI safety is a huge deal, because if they didn’t they would be a nobody working on a non-problem. Funny how you can make logic run in more than one direction.