will likely require the invention of new fundamental math
I wish you’d stop saying that (without justification or clarification). Modern math seems quite powerful enough to express most problems, so the words “new” and “fundamental” sound somewhat suspicious. Is this “new fundamental math” something like the invention of category theory? Probably not. Clarifying the topic of Friendly AI would almost certainly involve nontrivial mathematical developments, but in the current state of utter confusion it seems premature to characterize these developments as “fundamental”.
We don’t know how it turns out, what we know is that only a mathematical theory would furnish an accurate enough understanding of the topic, and so it seems to be a good heuristic to have mathematicians work on the problem, because non-mathematicians probably won’t be able to develop a mathematical theory. In addition, we have some idea about the areas where additional training might be helpful, such as logic, type theory, formal languages, probability and computability.
You’re right, the word “fundamental” might suggest the wrong kinds of things. I’m not at all confident that Friendly AI will require the invention of something like category theory. So, I’ve removed the word “fundamental” from the above comment.
I wish you’d stop saying that (without justification or clarification). Modern math seems quite powerful enough to express most problems, so the words “new” and “fundamental” sound somewhat suspicious. Is this “new fundamental math” something like the invention of category theory? Probably not. Clarifying the topic of Friendly AI would almost certainly involve nontrivial mathematical developments, but in the current state of utter confusion it seems premature to characterize these developments as “fundamental”.
We don’t know how it turns out, what we know is that only a mathematical theory would furnish an accurate enough understanding of the topic, and so it seems to be a good heuristic to have mathematicians work on the problem, because non-mathematicians probably won’t be able to develop a mathematical theory. In addition, we have some idea about the areas where additional training might be helpful, such as logic, type theory, formal languages, probability and computability.
You’re right, the word “fundamental” might suggest the wrong kinds of things. I’m not at all confident that Friendly AI will require the invention of something like category theory. So, I’ve removed the word “fundamental” from the above comment.
Developing new fundamental math is hard. SI may have to do it, but keep it to a minimum if you want to win!