I’ve been thinking about this for a long time. Doesn’t mean my thoughts are right, just well worn. Hopefully deep in a truth-tracking direction.
My best guess is that mathematicians will need to make their job be about enriching subjective experience rather than generating proofs.
If humans’ value is defined by their automatable output, then the economy absolutely will replace humans. It seems to have maybe found a way to do that with AI.
If we think there’s something wrong with that, then we’d better start defining humans’ value in terms of whatever it is that gives us that sense of wrongness.
I don’t know exactly how we do that. But I I think I see at least part of what to do.
If we get that part right, I think maybe the economy stops actively maligning AI against human existence.
I feel like mathematicians are being damned by faint praise here. The mathematical project is about precision and certainty, but it is a common experience for students to realize that for the first time they are seeing (for want of better terms) mathematical objects and relationships to which they were formerly blind. and much of the joy of mathematics comes from the ability to now discuss these new entities with precision. and further, part of being a professor of mathematics is exactly inventing and codifying new languages to carry out such discussion. so that a mathematician’s job can be understood as the invention and learning of new languages to discuss previously unseen entities and associated phenomena. this strikes me as an extraordinarily valuable social role, far beyond what mathematicians are usually granted.
the economy stops actively maligning AI against human existence
I think the path here is to decouple human value from economic output. mere human existence will be, for all practical purposes, of negligible cost. with luxuries not so very much more.
I feel like mathematicians are being damned by faint praise here.
I don’t understand how. Mind clarifying?
I think the path here is to decouple human value from economic output.
I see two interpretations here. One I might agree with, the other I fairly strongly disagree with.
The one I might agree with: Economic output is often going to be better done by something nonhuman, so if human value is tied to output, we’re setting humanity up for replacement.
I say I might agree because it really depends on what we mean by “output”. In some sense, all signals are a kind of output. Even if we find some way of valuing human existence inherently, signs that something is human will be an output that gets valued in whatever we might want to call an economy.
The one I fairly strongly disagree with: We have to decouple human value from the economy. Humans have to be seen as valuable for their mere existence.
I think that can’t work. Human existence is a cost, even if negligible. An economy that does anything like optimization will have pressure in such a situation to wipe out humans to recover the cost. If the thing stopping the economy from doing that is some kind of rigid rule around “value”, then the economy will try to find some way to hack around that rule.
It seems to be that for humanity to survive this, human existence needs to be valued in a way that’s intrinsic to the economy.
Said differently, I think it needs to be on the table that what humanity most values and cherishes is best accomplished without humanity’s survival. If it can’t be, then the reason it can’t be needs to be what’s baked into the Great Optimizer (i.e. the thing that showed up as evolution, then the economy, now AI) in order for humanity to survive this juncture.
> > I feel like mathematicians are being damned by faint praise here.
> I don’t understand how. Mind clarifying?
Just that you said …
> mathematicians will need to make their job be about enriching subjective experience rather than generating proofs
This seems to imply that generating proofs (if of value at all) does not enrich or cannot compete with enriching “subjective experience.” Even being generous, it damns the supposed role of the mathematician (“generating proofs”) with faint praise (they’ll be forced to do something useful, like enrich subjective experience). Of course as discussed, proof generation is already a most human of activities entailing language invention and novel expression.
Perhaps further, while proof generation is an important act in mathematics, able to give confidence when taking and refraining from project actions and time allocation, it is universally acknowledged among mathematicians that particular points in proof space are less interesting than the charts produced on the journey to those points. That unlocking new reasoning techniques is more important. Again, referencing generating proofs without the larger context of enhanced human cognition is “damning with faint praise.”
Sorry I’m ranting a bit, but I’m on something of a mission to see mathematics held in at least as high esteem as literature. No one admits to illiteracy (or at least pre-TikTok this was a shameful thing), but will casually assert innumeracy even though it is just as intellectually impoverishing.
> > I think the path here is to decouple human value from economic output.
> I see two interpretations here. One I might agree with, the other I fairly strongly disagree with.
I confess I am unable to distinguish between “economic output” and “the economy,” so I will refrain from sowing further confusion on this topic.
a mathematician’s job can be understood as the invention and learning of new languages to discuss previously unseen entities and associated phenomena
Would you say that LLMs are already good at the ‘invention’ part? My mostly-outsider impression is that they have a decent grasp of already existing languages and can apply them in appropriate circumstances that humans have overlooked, which is what’s driving the current action.
Would you say that LLMs are already good at the ‘invention’ part?
it is hard to measure how “good” they are without a truly formidable experimentation program. however, models have been observed inventing languages to solve a hard problem (and others in its class), which is already astounding to me, and is literally the definition of intelligence used by some computer scientists.
for example see section 6.4.5 in the Claude Sonnet 5 System Card …
the behavior is somewhat forced by limited prompt context and hours long runs, but is still super interesting.
> they have a decent grasp of already existing languages and can apply them in appropriate circumstances that humans have overlooked
”overlooked” makes it sound like everything was right before the expert and they still failed to notice it. I think that there is a little more going on in mathematics in that “proof space” is so vast that an expert could search it forever and still never encounter the particular region with the key that unlocks the solution. while the patterns of mathematics and the LLM’s brute force speed makes up for some of this, it is still only a few of orders of magnitude, so that’s where the mysterious “research taste” comes in, which somehow predicts productive areas to look with only a preliminary survey of the problem (both in humans and LLM-based autoresearchers!).
I’ve been thinking about this for a long time. Doesn’t mean my thoughts are right, just well worn. Hopefully deep in a truth-tracking direction.
My best guess is that mathematicians will need to make their job be about enriching subjective experience rather than generating proofs.
If humans’ value is defined by their automatable output, then the economy absolutely will replace humans. It seems to have maybe found a way to do that with AI.
If we think there’s something wrong with that, then we’d better start defining humans’ value in terms of whatever it is that gives us that sense of wrongness.
I don’t know exactly how we do that. But I I think I see at least part of what to do.
If we get that part right, I think maybe the economy stops actively maligning AI against human existence.
I feel like mathematicians are being damned by faint praise here. The mathematical project is about precision and certainty, but it is a common experience for students to realize that for the first time they are seeing (for want of better terms) mathematical objects and relationships to which they were formerly blind. and much of the joy of mathematics comes from the ability to now discuss these new entities with precision. and further, part of being a professor of mathematics is exactly inventing and codifying new languages to carry out such discussion. so that a mathematician’s job can be understood as the invention and learning of new languages to discuss previously unseen entities and associated phenomena. this strikes me as an extraordinarily valuable social role, far beyond what mathematicians are usually granted.
I think the path here is to decouple human value from economic output. mere human existence will be, for all practical purposes, of negligible cost. with luxuries not so very much more.
Totally agree. I think mathematics is dramatically and tragically misunderstood.
I don’t understand how. Mind clarifying?
I see two interpretations here. One I might agree with, the other I fairly strongly disagree with.
The one I might agree with: Economic output is often going to be better done by something nonhuman, so if human value is tied to output, we’re setting humanity up for replacement.
I say I might agree because it really depends on what we mean by “output”. In some sense, all signals are a kind of output. Even if we find some way of valuing human existence inherently, signs that something is human will be an output that gets valued in whatever we might want to call an economy.
The one I fairly strongly disagree with: We have to decouple human value from the economy. Humans have to be seen as valuable for their mere existence.
I think that can’t work. Human existence is a cost, even if negligible. An economy that does anything like optimization will have pressure in such a situation to wipe out humans to recover the cost. If the thing stopping the economy from doing that is some kind of rigid rule around “value”, then the economy will try to find some way to hack around that rule.
It seems to be that for humanity to survive this, human existence needs to be valued in a way that’s intrinsic to the economy.
Said differently, I think it needs to be on the table that what humanity most values and cherishes is best accomplished without humanity’s survival. If it can’t be, then the reason it can’t be needs to be what’s baked into the Great Optimizer (i.e. the thing that showed up as evolution, then the economy, now AI) in order for humanity to survive this juncture.
> > I feel like mathematicians are being damned by faint praise here.
> I don’t understand how. Mind clarifying?
Just that you said …
> mathematicians will need to make their job be about enriching subjective experience rather than generating proofs
This seems to imply that generating proofs (if of value at all) does not enrich or cannot compete with enriching “subjective experience.” Even being generous, it damns the supposed role of the mathematician (“generating proofs”) with faint praise (they’ll be forced to do something useful, like enrich subjective experience). Of course as discussed, proof generation is already a most human of activities entailing language invention and novel expression.
Perhaps further, while proof generation is an important act in mathematics, able to give confidence when taking and refraining from project actions and time allocation, it is universally acknowledged among mathematicians that particular points in proof space are less interesting than the charts produced on the journey to those points. That unlocking new reasoning techniques is more important. Again, referencing generating proofs without the larger context of enhanced human cognition is “damning with faint praise.”
Sorry I’m ranting a bit, but I’m on something of a mission to see mathematics held in at least as high esteem as literature. No one admits to illiteracy (or at least pre-TikTok this was a shameful thing), but will casually assert innumeracy even though it is just as intellectually impoverishing.
> > I think the path here is to decouple human value from economic output.
> I see two interpretations here. One I might agree with, the other I fairly strongly disagree with.
I confess I am unable to distinguish between “economic output” and “the economy,” so I will refrain from sowing further confusion on this topic.
Would you say that LLMs are already good at the ‘invention’ part? My mostly-outsider impression is that they have a decent grasp of already existing languages and can apply them in appropriate circumstances that humans have overlooked, which is what’s driving the current action.
it is hard to measure how “good” they are without a truly formidable experimentation program. however, models have been observed inventing languages to solve a hard problem (and others in its class), which is already astounding to me, and is literally the definition of intelligence used by some computer scientists.
for example see section 6.4.5 in the Claude Sonnet 5 System Card …
https://www.anthropic.com/claude-sonnet-5-system-card
the behavior is somewhat forced by limited prompt context and hours long runs, but is still super interesting.
> they have a decent grasp of already existing languages and can apply them in appropriate circumstances that humans have overlooked
”overlooked” makes it sound like everything was right before the expert and they still failed to notice it. I think that there is a little more going on in mathematics in that “proof space” is so vast that an expert could search it forever and still never encounter the particular region with the key that unlocks the solution. while the patterns of mathematics and the LLM’s brute force speed makes up for some of this, it is still only a few of orders of magnitude, so that’s where the mysterious “research taste” comes in, which somehow predicts productive areas to look with only a preliminary survey of the problem (both in humans and LLM-based autoresearchers!).