I get that change is scary, but this is honestly the most exciting time to be alive as a mathematician. even if all LLM use were banned tomorrow, each of these discoveries would still be talked about in 100 years. And, capabilities-wise, this is the worst it is ever going to be.
given that AI-associated x-risk largely depends on our level of mathematical competence, this is also arguably the most important time in history for the social role of mathematicians.
AI-associated x-risk largely depends on our level of mathematical competence
So far, mathematics have been used mainly to make the global situation worse, i.e., by enabling AI development to proceed in a way that maximizes capabilities with almost no regard for how easy it will prove to maintain control or alignment of the capabilities.
It would take a lot to persuade me that that trend will reverse any time soon.
I will concede that doing mathematics is probably good training for alignment research because it teaches mental moves that are useful outside mathematics, but I wouldn’t recommend mathematics as the majority of the training time.
I say this as someone who really enjoys doing mathematics.
given that AI-associated x-risk largely depends on our level of mathematical competence, this is also arguably the most important time in history for the social role of mathematicians.
I think controlling AI x-risk requires insight into the behavior of complex mathematical objects. To be a little less fuzzy, we need a working theory of mind. It is our mathematical competence which will determine how much control we have here.
I think RSI (a.k.a. self-improving AI autoresearchers) is inevitable for a wide variety of commercial, military, and/or geopolitical reasons, no matter what treaties are signed, no matter what promises are made. At least the NSA will do it. This is a curse in that it speeds the timeline. This is a blessing in that it gives us a chance. Again, though, the key to bounding the alignment delta between versions is going to be a theory of mind and its associated complex mathematics. If the delta is unbounded or even just too large, we lose and die. If the delta is small, we have a chance.
I think just waiting, or rather just researching, allows successive model versions to drift apart in mind space and makes it more difficult to bound the alignment delta. An all round catastrophe. We need to cognitively enhance each mathematician we have and put them to work developing (the mathematics behind) a theory of mind and/or on their favorite alignment subproblem.
given that AI-associated x-risk largely depends on our level of mathematical competence, this is also arguably the most important time in history for the social role of mathematicians.
I get that change is scary, but this is honestly the most exciting time to be alive as a mathematician. even if all LLM use were banned tomorrow, each of these discoveries would still be talked about in 100 years. And, capabilities-wise, this is the worst it is ever going to be.
given that AI-associated x-risk largely depends on our level of mathematical competence, this is also arguably the most important time in history for the social role of mathematicians.
So far, mathematics have been used mainly to make the global situation worse, i.e., by enabling AI development to proceed in a way that maximizes capabilities with almost no regard for how easy it will prove to maintain control or alignment of the capabilities.
It would take a lot to persuade me that that trend will reverse any time soon.
I will concede that doing mathematics is probably good training for alignment research because it teaches mental moves that are useful outside mathematics, but I wouldn’t recommend mathematics as the majority of the training time.
I say this as someone who really enjoys doing mathematics.
Why do you think this?
I think controlling AI x-risk requires insight into the behavior of complex mathematical objects. To be a little less fuzzy, we need a working theory of mind. It is our mathematical competence which will determine how much control we have here.
I think RSI (a.k.a. self-improving AI autoresearchers) is inevitable for a wide variety of commercial, military, and/or geopolitical reasons, no matter what treaties are signed, no matter what promises are made. At least the NSA will do it. This is a curse in that it speeds the timeline. This is a blessing in that it gives us a chance. Again, though, the key to bounding the alignment delta between versions is going to be a theory of mind and its associated complex mathematics. If the delta is unbounded or even just too large, we lose and die. If the delta is small, we have a chance.
I think just waiting, or rather just researching, allows successive model versions to drift apart in mind space and makes it more difficult to bound the alignment delta. An all round catastrophe. We need to cognitively enhance each mathematician we have and put them to work developing (the mathematics behind) a theory of mind and/or on their favorite alignment subproblem.
Strongly doubt that.