Charles Harding
charding
Karma: 21
How often can frontier models autonomously solve math problems that required human-AI collaboration only a few months ago? This seems evaluable and may be useful as an early warning.
Frontier models are resolving many open problems and saturating many benchmarks. At this point, would it also be worth tracking how often the newest AI models can zero-shot proofs and counterexamples that people recently obtained in collaboration with older AI models?
For the sake of argument, suppose that many recent human contributions turn out to be redundant to the capabilities of AI models released weeks or months later, in the sense that results which previously required humans and AIs to work collaboratively can now be obtained by AI alone. This would likely feel unpleasant. However, if and when this occurs, it could provide an important early warning that we are exiting the “centaur phase,” during which human-AI collaborations are typically stronger than humans alone and AIs alone. So it would be good to know.
I’ve hesitated to make this suggestion because I’m not a mathematician. I couldn’t do this tracking and I can’t even fully assess whether it is a good idea. However, others have surely had similar thoughts by now. If the idea isn’t feasible, then I’d like understand more—what would prevent it? I’d guess that others would like to know too.
Some obstacles do seem addressable. If the details interest you, please see below the folds.
Knowledge cutoffs
For this to work, tracking would need to focus on problems solved by human-AI collaboration after the knowledge cutoff date of the newest AI model (but before that model’s release date). Otherwise, solutions that a model obtained de novo could not be distinguished from solutions that it remembered from training data. Evaluations would be performed with web search disabled. For simplicity, it would also help if original solutions were Lean-verified.
Sample size and misclassification
When I looked through Erdos problems a month ago, it seemed like 10-20 recent solutions met both the knowledge cutoff and Lean criteria. This is not a big sample, but is large enough to give a rough sense of whether the human part of the human-AI collaborations could be replaced by subsequent models working on their own. It is not smaller than the sample used in First Proof, for example. More problems may meet the criteria today.
However, it is slippery to figure out which solutions actually involved meaningful human and AI contributions. I may have misjudged. Also, a user could easily make derivations in extended interaction with AI, without realizing that AI could have provided the same results alone. To address this, it would be useful to run the original model on the problem alone, to verify that it fails to resolve the problem. (This is only possible until the original model is deprecated.) Care would be needed to avoid blame in such situations. I think it could happen to any of us.
Interpretation
Another obstacle is that proper interpretation would be subtle. The class of problem that is included in tracking may greatly affect how often newer AI can replace the human contribution to recent derivations. Further, even if it is found that newer AI was able to fully replace human contributions, it does not strictly follow that we are leaving the centaur phase. For example, it remains possible that the mathematicians aimed for low-hanging fruit with their recent contributions, which were therefore easiest for newer models to replicate, and that mathematicians could also reach higher fruit that AI still cannot obtain. Additionally, solving problems does not touch other important aspects of math, like theory building and exercising taste.
Obstacles of interpretation are also faced by First Proof, Erdos problem tracking, and trends in AI-solved conjectures. All are valuable indicators of AI capability nonetheless.
Continual learning
A last obstacle is that the suggested tracking could become impossible when continual learning arrives, owing to the removal of knowledge cutoffs. The opportunity to do this work may disappear, but that is all the more reason to do it now.
charding’s Shortform
Good points, thanks!
1. General capabilities...
You have many informative analyses for questions the predecessors got wrong, but results could also be interesting (and different) for questions the predecessors got right.
Let the true positive rate (TPR, often termed ‘recall’) be the proportion of erroneous predecessor responses that the simulator replicated successfully.* Let the false positive rate (FPR) be the proportion of correct predecessor responses that the simulator failed to replicate (predecessor: right answer, simulator: wrong answer).
On latent reasoning and other subjects, you discovered interventions that increase TPR, with thought-provoking implications. It could be useful to know if those same interventions reduce FPR, leave FPR unchanged, or increase FPR.
The last option might seem theoretical, but is typical in fields like diagnostic testing (see sensitivity-specificity tradeoffs). In those fields, studying TPR by itself can produce skeptical reactions or the absence of attention, since any reported TPR changes may be outweighed by unstudied FPR changes.
In your case, an example might be a prompt that increases TPR but does not increase simulation ability. This could happen if the prompt operates by hinting to the model that erroneous responses should be more common than the model’s raw intuition, thereby making the model more open to suggesting erroneous responses, which may be seen from increased FPR.
However, I agree TPR is important by itself too, and it could even be that your FPRs would be low enough (near 0) to be inconsequential.
2. Exposure to more model traces...
I hadn’t considered benchmaxed models, but I agree they could be a useful test case.
Online traces… are largely using the “assistant” persona. We can train models with other, differentiated personas for monitoring.
Reasoning and other intelligence scaling paradigms that don’t rely on a better/newer/larger model can boost the predecessor into a regime where it is actually more intelligent than its successor.
Solid points, and I appreciate that you have some evidence for the latter from your work (claim 2).
* To be more exact, you have two TPR varieties. In the first, simulation is successful if simulator and predecessor both get the question wrong. In the second, simulation is only successful if simulator and predecessor both select the same erroneous answer. However, the distinction between TPRs mostly isn’t relevant to my comment.
What an excellent topic for study through experiments! Thanks for posting.
Upcoming models will have higher general capability. During training, they’ll also be exposed to more outputs from previous models. Do you have thoughts on how these two factors would affect the simulation performance of models in the near future, especially any thoughts based on your experiment results?
If I tried to simulate another person’s answers to questions, I’d benefit from having higher capability with the subject matter, but I’d also benefit from more individual experience with the person (their personality, habits, areas of competence...). In this way, capability and familiarity seem like relevant factors that are distinct from each other.
Models are already exposed to many predecessor responses during training, intentionally and unintentionally. However, effective learning can require huge data volumes. Also, to learn an error pattern, in some sense the new model may need both erroneous responses from the predecessor model and correct ground truth, which is a rarer data combination. For these reasons, data insufficiency could still be a limiting factor to simulator performance. I wonder if your experiment findings give you any insight into this?
If exposure to even more predecessor responses during training could be sufficient (by itself) to greatly improve simulator performance, then I tend to think this suggests fundamental weaknesses with the idea of using earlier models to monitor newer models.
P.S. I enjoy the data-cones plot! Cool idea.
The text you quoted has many strong ideas, which makes it unfortunate that the text is buried down in the post-scripts to a long article.
It’s good that you posted here, to give the text a little more prominence.
Fair enough & I appreciate the follow-up.
Though I will say—It seems we need to find lessons somewhere in history, in part because we aren’t smart enough as a species to reason purely from first principles. I’m certainly not smart enough for that, anyway.
When looking for lessons on AI, nuclear development may be the least worst historical analogy.
Thank you for explaining.
The societal readiness plot doesn’t seem to have a log-ish y axis, considering the shapes of the trend lines.
If the alignment plots were also drawn without a log-ish y axis, then they might look as bad as the societal readiness plot or—if not equally bad—then at least substantially worse than they do now.
I’m questioning plotting decisions for fake plots …I know. This may seem like splitting hairs. However, to me, there is a major difference between requiring linear vs.
exponentialpower law improvement in alignment, to take us to “what we need”.
The plots state that their x axes show capabilities on log scale, but what scales were intended for the y axes?
We might expect that the y axes are on linear (untransformed) scale. However, this would imply that multiplicative increases in AI capability can be addressed safely by making only additive amounts of progress in alignment (dashed green line on plots).
In general, multiplicative outmatches additive. How can we be confident that additive alignment progress would be enough?
Alternatively, we could view y axes as being on log scale. Yet, then the gap between “actual” (solid green line) and “what we need” (dashed green line) can be much larger than visually apparent on the plots, especially for dates in the future, leaving a big space for the region helpfully labeled, “A lot of bad shit can still happen [here].”
Many people may feel we are in your scenario A under a log y axis, with a big gap between “actual” and “what we need” appearing in future if current trends persist. In particular, people with fragile world concerns may place more “Everybody dies” dots in that gap, considering how large it could be.
Thank you for the plots! I hope I did not misinterpret them.
Thanks & no apology needed : )
Thank you to your dad for offering to answer questions.
Sometimes people make the argument that the U.S. needs to race toward AGI-ASI as rapidly as possible, because if China obtains it first, then the risks to the U.S. are unacceptably high. However, this argument can also be an appealing excuse for people in the U.S. who would wish to go full-speed toward AGI-ASI even if there were no competition from China.
I imagine similar arguments are also made in China, with the roles of the U.S. and China reversed.
Does your dad have thoughts about these kinds of arguments, considering that analogous arguments were made about the nuclear arms race? How does your dad think about the interaction of people who make these arguments genuinely vs. those who use these arguments as an excuse?
Thank you for your response!
The go/no-go model is not meant to show that a P(doom) of up to 97% is “acceptable”
I should have been clearer, yes. I meant that the 97% deals are acceptable to your go/no-go model, not to you or your later models.
However, I think my arguments apply equally to your later models, just with P(doom) different from 97%. (See below.)
a series of more complicated models are introduced that take into account some of these other factors.
Thank you for running the more complicated models. (And, in case unclear, I did read all your article before making any comments.)
Do the models help us understand how we should act? Here is how I look at it --
It’s difficult to get intuition about how good or bad the optimal ASI launch times are, because the envisioned situation is so far from experience.
In each model, there is a remaining P(doom) at the model-optimal launch time, call it R%. R% is sometimes large, sometimes small.
From a perspective that stays entirely within the specific person-affecting framework used by the models, the following deals have equal value
Deal 1: At ASI launch, all humans are killed with R% probability, but life expectancy becomes superhuman with probability 100% - R%.
Deal 2: At ASI launch, R% of humans are killed with certainty, which is the cost of providing superhuman life expectancy for the remaining 100% - R%.
What happens if we put one foot outside model assumptions and begin to care some about future generations? Are lessons of the models robust to a small step away from their assumptions?
If we care about future generations, this tiebreaks. Deal 1 seems worse than Deal 2, since under Deal 1 there may easily be no future generations, while under Deal 2 future generations can repopulate, have happiness, and so on. Even if one believes Deal 1 is not strictly worse than Deal 2, Deal 1 hardly seems much better.
Let’s now return to viewing things however we actually see them, without adopting a specific framework whose rules we restrict ourselves to follow. It’s difficult for me to get an intuitive grasp on Deal 1, but Deal 2 is easier to understand. There are a lots of similar historical precedents. Deal 2 is explicitly killing a large proportion of humans to let the others prosper more. It’s horrible.
We have seen that stepping even slightly outside the model’s person-affecting stance, Deal 1 is worse than Deal 2. Yet, Deal 2 is plainly awful. So what is the value of simulating optimal times for models to accept Deal 1? It is like modeling the ″best″ time to enact horror.
I should mention that the models often find high R% to be optimal. For example, with 50% initial P(doom) and 5%/yr safety progress, the model you used in Table 3 would launch when P(doom) remains 32% with overall life expectancy of 774 years, instead of waiting a handful of decades for P(doom) to fall to near 0. I noticed this when I coded your Appendix A model for the sensitivity analysis in my other comment (which also found a small error in your Table 3 -- see that comment for detail.)
Some people …. care about inequality of outcomes, not only inequality of chances
I neglected this, thank you for raising it. I don’t think it undermines my argument, but let me know if you disagree.
Infinite regress issue?
I’m apologize if you addressed this somewhere or if I misunderstand, but is there an infinite regress issue with your models?
Take three times, T0, T1, and T2, each a year apart. Let average life expectancy be 40 years among the living at each time. Suppose the model is run at T0 and it says “Delay ASI launch 3 years”. However, if the model is run again at T1, it will still say “Delay ASI launch 3 years” because the model explicitly only cares about the living. And the same at T2 or later. So one would never reach the time to launch ASI.
Bostrom’s Footnote 21 seems innocuous, but to me it unravels a lot of the argument Bostrom is making.[1]
Bostrom’s central go/no-go model suggests a P(doom) of up to 97% is acceptable if life expectancy rises to 1,400 years post-AGI.
Footnote 21 clarifies, ``more generally, we could take P(doom) to be the expected fraction of the human population that dies when superintelligence is launched.″
So, suppose one could extend life 1,400 years for 3% of humans at the cost of killing 97% right now. How should we reply to this deal? Bostrom’s go/no-go model says to accept, but I think people would overwhelmingly find the deal morally reprehensible.
It’s straightforward to observe that the two deals, Bostrom’s and mine, are mathematically identical for the go/no-go model, which assumes as person-affecting framework. But I think something more subtle happens if we step slightly outside the person-affecting framework.
If you have even slight impersonal preferences, I think that you should decide Bostrom’s deal is strictly worse than mine. Therefore, I think that you should support Bostrom’s deal only if (a) you solely follow selfish motivations (which most people think is bad) or (b) you support my deal (and my deal seems horrible).
Why is Bostrom’s deal strictly worse than mine, if you have slight impersonal preferences? Under Bostrom’s deal, in 97% of cases everyone is dead. Under my deal the surviving 3% could have further generations who continue aspects of life we value, potentially flourishing for a long time.
One could disagree and propose ways my deal seems worse than Bostrom’s, but I think my deal can simply be modified to accommodate until my deal is better. For example, if you feel its more fair for everyone to face the same risk, suppose the 3% are selected by random sampling in my deal. If you favor Bostrom’s deal because it doesn’t split families and friends across living vs. killed, suppose the sampling in my deal is by large clusters of families and friends. If you favor Bostrom’s scenario because of non-lifespan ASI benefits, suppose an ASI confers those same benefits to the surviving 3% in my deal. Wherever you end up, my deal is still killing 97% of people to benefit 3%, so my deal is still horrible. Yet, Bostrom’s deal remains worse.
I am sympathetic that Bostrom set out reasonably to investigate ASI within a person-affecting framework. Unfortunately—and irrespective of Bostrom’s initial intentions—I think that his article has mainly turned out to be a list of justifications for the small, probably malevolent group of people who would accept a deal that kills 97% of humans to extend the lives of 3%.
- ^
Always read the footnotes!
- ^
Bostrom’s results seem very sensitive to deviations from a wholly person-affecting perspective. To investigate, I coded up the model from Appendix A with one modification: I supposed that, instead of being wholly self interested, people are willing to sacrifice 10% of life expectancy for the sake of all future generations.
My method was to calculate the launch time that is later than the optimal time-point according to a selfish view, but only so much that life expectancy is reduced 10% from the selfish optimum.[^1] This method is crude, but illustrates how rush-to-launch loses support if one walks mildly away from a person-affecting view.
For example, with 20% $P_{doom}$ and 10%/yr safety progress, the selfishly optimal launch time is 8 months (Bostrom’s Table 3), which offers you 1,120 years of life expectancy. If you are willing to sacrifice 10% of that life expectancy (leaving you with 1,008 expected years of life!) for future generations, you would wait 11 years before launch to help safety become established. More generally, all superintelligence launch times from Table 3 were delayed by at least 4 years (none were ASAP anymore) and many were delayed 10-20 years. The rush to superintelligence was ameliorated.
Last, I estimated delays under more sacrifice. If people are willing to lose half of life expectancy to help ensure the existence of future generations, then superintelligence launch times would be delayed by at least 28 years for all scenarios covered in Bostrom’s Table 3. Results are below. Further, for all cases with $P_{doom}$ of 80% or less, the life expectancies of those making the sacrifice would remain generous, exceeding 140 years for $P_{doom}$ of 80%, exceeding 349 years for $P_{doom}$ of 50%, and exceeding 550 years for $P_{doom}$ of 20% or less.
Table A. ASI launch delay by P(doom) and safety progress, offering 50% of life expectancy. The table includes the scenarios from Bostrom’s Table 3.
Safety progress P(doom) 1% 5% 20% 50% 80% 95% 99% No progress
(0%/yr)*
29 y 29 y 29 y 30 y 35 y Never
launch
Never
launch
Glacial
(0.1%/yr)
29 y 29 y 30 y 32 y 42 y Never
launch
Never
launch
Very slow
(1%/yr)
29 y 30 y 32 y 43 y 80 y 102 y 108 y Moderate
(10%/yr)
29 y 31 y 38 y 47 y 52 y 54 y 54 y Brisk
(50%/yr)
29 y 31 y 33 y 34 y 35 y 35 y 35 y Very fast
(90%/yr)
29 y 30 y 31 y 31 y 31 y 31 y 31 y Ultra fast
(99%/yr)
29 y 30 y 30 y 30 y 30 y 30 y 30 y * Note that the sacrifice is fruitless in this case, because there is no safety progress during the delay. Similarly, the sacrifice may not have reasonable justification in the ultra-fast case.
[^1]: To confirm my code’s correctness, I also recreated Bostrom’s Table 3. This revealed a typo in Table 3: For $P_{doom} = 0.95$ and safety progress of 1%, the launch time is listed as 14.3 years but should be about 31.4.
Maybe it would be good to know how often the AI co-developed solutions can now be obtained by the latest AI models working on their own. (The checks would need to be limited to problems solved after the knowledge cutoff dates of the latest models, and probably restricted to Lean-verified only.)
That might provide an indicator for when we are starting to exit the centaur phase, at least for the problem solving part of math.
Also, how many of the VibeMath users / contributors do you think are agents? In their dataset, I see only one identifiably human name (JSON key: submittedBy).