Yes, agreed, this was sentence was unclear. I was trying to make a claim about how feasible it is for the lab to make watermarking have essentially no practical effect on output quality, assuming cooperative user behavior (which is the usual problem labs are trying to solve – after all, it’s not a secret that they are applying watermarking). You could imagine using a cache that persists across queries from a single user over a short time period, which should mitigate most concerns about users making repeated, similar queries for things like best-of-n. This particular scheme might not be super practical, but it’s certainly more practical than a persistent, global cache, and it’s probably overkill anyway (though I don’t have a strong view about that). Hence my original claim: schemes actually in use don’t have complete indistinguishability, but well-tuned versions should have essentially no practical impact.
Jacob_Hilton
Two facts about text watermarking that seem to be frequently misunderstood:
There exist watermarking schemes such that it is completely impossible for users to ever distinguish between watermarked and non-watermarked text. (A tempting but invalid argument is that no such scheme can exist because watermarking consumes entropy.)
The above property does not hold of watermarking schemes that labs claim to actually use (though I would still expect well-tuned versions to make essentially no practical difference to users).
When I say “completely impossible”, I mean it in the everyday sense, i.e., not possible to perform using any machine that could feasibly be built, assuming the integrity of the scheme remains intact (so stealing the secret key doesn’t count). I do not mean “impossible with infinite computational resources”.
On my first point: watermarking does reduce the information-theoretic entropy of generated text in order to insert its cryptographic footprint. However, what matters is not information-theoretic entropy, but computational entropy (often called “pseudoentropy”): roughly, the maximum entropy of a distribution that is indistinguishable to a computationally bounded observer. For example, SHA-256(128 random bits) has 128 bits of information-theoretic entropy, but 256 bits of computational entropy.
To make sense of this in the context of watermarking schemes like SynthID-Text (used by both Google DeepMind and Anthropic): the way these schemes work is that when the next token is sampled from the LLM’s output distribution, instead of drawing the sample completely randomly, it is drawn partly-randomly and partly-pseudorandomly based on a secret key and the previous few tokens. You don’t want to use too many previous tokens, or else short snippets of text evade being watermarked. The SynthID-Text paper uses the previous 4 tokens by default, so I will use that for argument’s sake.
As long as watermarking is never performed using the same 4-token context more than once, watermarking consumes information-theoretic entropy, but zero computational entropy. The particular choice of secret key makes some tokens more likely than others for any given 4-token context, but there’s no discernible pattern between one 4-token context and another. This means that no statistical test, benchmark score, user assessment, or anything else (short of stealing the secret key) will allow the user to distinguish between watermarked and non-watermarked text.
However, if watermarking is performed using the same 4-token context a second time, this does use up computational entropy. Concretely, you are more likely to see the same token a second time than if there had been no watermarking. Because of this, SynthID-Text recommends caching 4-token contexts, and not watermarking subsequent occurrences of the same 4-token context (this is called “context masking” in the paper).
This explains my second point: the only way to make SynthID-Text completely undetectable to the user (i.e., “infinite-sequence non-distortionary” in the language of the paper) is to never watermark using the same 4-token context more than once. This is impractical, because it requires a global cache that is never invalidated, and moreover, the strength of the watermark would gradually degrade as more 4-token contexts are used up. But it is much easier to cache 4-token contexts locally whenever a diverse choice of 5th token could reasonably matter, which would be enough for most practical purposes. The SynthID-Text paper just caches within individual responses to a single prompt, but more conservative schemes should also be feasible.
As for the actual versions of SynthID-Text implemented by labs, my guess is that they are configured to make essentially no practical difference to most users (beyond the fact that the text is indeed watermarked), but it’s not really possible to say this definitively without more transparency about how SynthID-Text is configured (in particular, the number of tournament layers, window length, and caching/context masking strategy).
In summary: watermarking can reduce next-token diversity following identical short sequences of tokens, but it is not hard to mitigate this for most practical purposes, and the naive argument that watermarking must reduce output quality because it consumes entropy is invalid.
I don’t recall seeing any specific effort on these models in the last six months outside of the comments on this post. To me, that progress is fairly marginal and we are not much closer to succeeding at the challenge given at the end of the post. (I’m excluding other research at ARC that doesn’t pertain to these models specifically, but may be on the path to success.)
The future of human mathematics
Cross-posted from Twitter; written for mathematicians.
About a year ago, I wrote that superhuman math AI will plausibly arrive in the late 2020s/early 2030s. It now looks like if anything, I was underestimating the pace of progress. In this post, I will try to say something about the sociological implications for mathematics.
I don’t want to focus too much on the predicament of the mathematical community today; instead, I’ll try to take a longer-term view, and say something about where I think human mathematics might eventually end up. For the sake of argument, I’ll assume that within the next decade or so, AI systems are developed that are vastly more capable than humans at every mathematical task (including proving theorems, building theories, and explaining mathematics to humans), and that there nevertheless remains a community of human mathematicians, alive and well, and with funding to pursue mathematics as they please. What would such a community be doing?
Of course, in this hypothetical, mathematicians are not pursuing mathematics for any productive purpose; that is handled by the machines. They are pursuing mathematics for its own sake. There is nothing at all new about this; it is the same purpose that Hardy famously defended in 1940 in A Mathematician’s Apology, declaring, “Judged by all practical standards, the value of my mathematical life is nil.” As a former set theorist, I can certainly say the same of my own work in that area, as I am sure would many others.
What, then, does the pursuit of mathematics for its own sake look like, in the AI era? There are many different activities that mathematicians might engage in, so this is largely a question about which of those I feel best embodies the spirit of mathematics as it is practiced today. I will go through a few different options in turn.
Mathematics as chess
One possibility is that humans will solve mathematical problems with known solutions. They might do this leisurely and collaboratively, or they might do it competitively, as in olympiads today. Perhaps there would be a “world championship of mathematics” lasting many days, with spectators observing competitors’ scratch work, interpreted with the help of AI and narrated by their favorite commentators. I call this “mathematics as chess”, since it resembles how the chess community has continued in spite of AI dominance.
If this were the most faithful remaining reflection of mathematics as it is practiced today, I think most mathematicians would find such an outcome disastrous.
Mathematics contests today are viewed as a way to train and encourage new talent; this is why they are almost always restricted by age or educational attainment. Mathematics itself is the activity these contests are viewed as preparation for. If as a response to AI progress there is an entirely new industry of widely-spectated math contests, there is no reason for mathematicians to begrudge this; but to claim that this is the spiritual successor of mathematics today is to completely misunderstand mathematical culture.
At its core, mathematics is about discovery. This is why mathematicians spend their time on unsolved problems, not solved ones, and why proper attribution is sacrosanct. Of course, mathematicians are rewarded with money and prestige, which are proximate motivations; but discovery is the cultural value that determines how these are distributed. Furthermore, mathematical values related to discovery can sometimes trump monetary rewards. This was famously demonstrated by Perelman, who turned down a $1m Millennium Prize, apparently because he felt that it should have been shared with Hamilton, whose work was important to his solution.
So no, mathematics is not chess.
Mathematics as an island
A second possibility is to reject non-human intelligence within mathematics entirely, and maintain a closed-off community to preserve mathematics as it was practiced prior to around 2023. Humans would continue to conduct research, solve problems that no-one else within the community had yet solved, and credit each other for being the first within the community to solve a problem.
As tempting as this may sound to some readers in the current moment, I suspect that such an approach will be taken by at most a small fringe of the mathematical community.
Firstly, there would be an inherent trade-off between the degree of separation with the outside world, and the degree of confidence one could have that others are playing by the same rules. Using AI assistance would be like doping in sport, except that even having a conversation with someone from the outside world could be viewed with suspicion. I find it hard to imagine such an environment providing sufficiently sustainable motivation to all but the most die-hard traditionalists.
But more than this, “mathematics as an island” has too much of an air of self-delusion about it. It amounts to pretending that machine intelligence does not exist, rather than coming to terms with it. I suspect that for most mathematicians, there would be too much dissonance with their truth-seeking values for them to dedicate their existence to such a path.
Mathematics as star-gazing
A third possibility is for humans to decide which problems to set AIs to work on. Of course, in this hypothetical world, humans have nothing to contribute when it comes to figuring out which sub-problems are the most promising, nor estimating the probability that a problem will be solved after spending a certain amount. So it will be more like choosing options from a menu. Perhaps we will have to debate whether we prefer a 10% chance of resolving the twin prime conjecture, or a 1% chance of resolving the Riemann hypothesis; some may relish this opportunity more than others.
However, there is a wrinkle: the results will be front-loaded. Superintelligent AI will not immediately solve all mathematical problems, for the same reason that it will not immediately prove that chess is a draw under perfect play: the search space is too large. But there will be diminishing returns to computation, and the relative growth rate of computational resources must eventually decline due to physical limits (in the very long run, resources can grow at most cubically with time, since that bounds how rapidly one can expand through 3-dimensional space). I’m not confident in this, but I suspect that a small relative increase in computation gives at best a small additional chance of solving any given problem, even if this amounts to a large increase in computation in absolute terms. So we should expect an initial flurry of excitement, but after a few generations, the proof of an old conjecture will be a rare moment of celebration.
One way around the decaying rate of AI-generated results is to continually come up with new problems to pose to AIs, or to have AIs generate entirely new theories autonomously. This will still be possible, because the search space is exponential. But the lowest-hanging fruit will soon be plucked, and I suspect that the trillionth most interesting theorem will involve enough arbitrary-seeming choices that the appeal will be similar to that of a puzzle from a puzzle hunt. I am sure that some people will amuse themselves by generating or designing puzzles for their Jupiter-brain to solve, but I don’t know if I would call that mathematics.
Regardless, “mathematics as star-gazing” is missing a second ingredient that is core to mathematics: human understanding. Many of the proofs produced by AIs may be possible for humans to understand, but unless we actually put in the work to understand them, I imagine most mathematicians would consider the proofs themselves to be meaningless trophies.
Mathematics as poetry
The final possibility I will consider is that humans will continue to engage with machine-generated proofs, but the primary activity of human mathematicians will be hermeneutics: selecting favorite results from the enormous AI-generated library of mathematics; digesting proofs (either of old results or of fresh “puzzles”), and presenting and explaining them to each other; re-proving known results by hand to advance their own understanding; and so on. The overall goal of these activities would be to advance the collective knowledge, understanding and appreciation of the frontiers of mathematics by humanity.
In such a world, people would write their own textbooks, proofs, and problems, perhaps with AI assistance, in order to share them with others. The obvious rejoinder to this suggestion is: why would humans be necessary to create such artifacts at all? The only possible answer to this question is: we value such artifacts by virtue of the fact that they were written (or selected) by a human. This is the same answer that must eventually justify all human art, literature, and poetry, hence “mathematics as poetry”.
Of course, such artifacts can be faked. But it will be like having an idle conversation with an old friend. You don’t worry about whether it was faked, because it has no economic value, and there was never any point in faking it. It exists only between the two of you, and each of you values it primarily for its authenticity.
Even in a world where AI can explain to us any result of our choosing better than any human, I still expect there to be a social aspect to human mathematics. Perhaps there will be a significant solitary aspect too, to cater to individual tastes; but I think most mathematicians care a significant amount about sharing ideas and experiences, and understanding things in common with their peers.
In my opinion, “mathematics as poetry” is the most likely outcome that at least resembles mathematics as it is practiced today. It sacrifices the primacy of human discovery, but at least replaces it by a creative process of mathematical expression, while keeping human understanding and interaction at the forefront.
Transitioning to such a world may be a process of considerable grief for many mathematicians, even if everything goes about as well as we can reasonably hope with AI. But for better or for worse, I think this may be the most palatable option on the menu.
Postscript
Although I have been treating superhuman math AI as a hypothetical, I actually think that this is eventually the most likely outcome. And if we restrict to purely formal problems such as theorem proving, I think superhuman math AI is likely to be here before the end of the decade. The progress of AI in mathematics has been extraordinarily rapid for the last 5 years, and there is little reason to think that the drivers of this progress (algorithms, computation and data) will stall in the near future. I say this as someone who was involved in the development of GSM8K, a mathematical benchmark from 2021 that was essentially obsolete by the end of 2023.
But without meaning to belittle what the mathematical community is going through, the future of human mathematics is not the most pressing challenge posed by AI in the near future, despite being close to my heart. It is currently unclear whether we will be able to remain in control of superhuman AI at all, and many of the concerning dynamics that were primarily hypothetical a few years ago are now playing out in practice.
I am biased, but I think that ARC (where I work) is one of the most promising ways for mathematicians to contribute to AI safety. Other organizations that are hiring theorists include Resolution, PrinceInt and Simplex, and there are also training programs such as Iliad’s. To be honest, though, I don’t think any of us have exactly the right ideas yet, and my main hope is that someone with fresh eyes will spot something crucial that we are all currently missing.
If you are a mathematician who shares my view of where AI is headed, please take advantage of the additional foresight this grants you, and make the most of the last few years where human contributions to mathematics have practical as well as aesthetic value. I wish you the very best.
Thanks to Kevin Ren for a number of comments.
Update: We have now launched Phase 1 of the Challenge with $50,000 in prizes:
Score-based prizes: $25,000 / $10,000 / $5,000 for 1st / 2nd / 3rd place
Algorithmic contribution prize: $10,000
For Phase 1, we have increased the depth of the network from 8 to 32 hidden layers. Our existing algorithms scale poorly with depth, and so we expect there to be significant room for improvement. Phase 1 lasts until the end of July, after which Phase 2 begins. For Phase 2, there will be a prize pool of at least $100,000, and we may change the architectural parameters again.
The best-performing algorithms in the warm-up round appear to be variants on the factorized 3rd cumulant propagation algorithm we introduced in our paper, combined with learned networks that consume the cumulant estimates as features. Many of these submissions appear to have been produced with the help of LLMs, but we don’t know much about the extent of this. For Phase 1, we expect the increased depth to advantage approaches that go beyond basic cumulant propagation in their mechanistic analysis. We also expect to get more visibility into the design of top submissions, since we will be using accompanying technical write-ups to award the algorithmic contribution prize. For more information, please see the contest website.
Thanks! Trained MLPs are somewhat trickier to design a contest around, since once you have a well-defined distribution over trained models, people can spend a lot of compute offline empirically fitting predictors for the expected output as a function of the model parameters, and it would be much harder to outperform these with mechanistic approaches. But it’s good to know this may be more motivating, and we may consider it.
What knowledge is this bet based on?
The existence of such an algorithm is a special case of a broader conjecture that we have been calling the “matching sampling principle” (specifically the train-and-explain version, as described here). Our evidence for this conjecture is mainly based on examples where it appears to hold, a lack of compelling counterexamples, and more abstract philosophical reasoning. The specific research bet is a judgment call based on the plausibility of the overall approach, tractability, value of information, etc. (Sorry to be vague, it would take a lot of work to give a very detailed answer to your question, and we are hoping to say more about all of this in the not-too-distant future.)
Announcing the ARC White-Box Estimation Challenge
Mechanistic estimation for expectations of random products
Yes, the fact that we have concrete metrics for progress is part of why we are excited about this line of work. We suspect that if we were given algorithms that achieve very low MSE for a given FLOP budget that we’d be able to extract useful insights from them, although it remains to be seen how well this will pan out in practice. We’re planning to launch a contest soon to test out this idea, with LLM usage encouraged. (It will be slightly less prescriptive than the setup you suggested, code just has to take in network weights and produce expectations.)
Indeed, tensor network diagrams show up in our algorithm (see Appendix A of the paper). We’ve also been thinking about mechanistic estimation for tensor network contractions as a problem in their own right, partly because they appear to be needed for harder MLP cases.
Mechanistic estimation for wide random MLPs
I agree that there are qualitative similarities, so perhaps we should be quantitative about it. Assuming for the sake of argument that the DoW were acting in bad faith and plans to use OpenAI’s services to conduct domestic mass surveillance (legally), how likely do you think it is that OpenAI would be able to prevent this? Given the difficulties I mentioned (indistinguishable from innocuous use, problematic only in aggregate, novel setting, classified, ZDR, no meaningful contractual recourse), it would seem like a big stretch to reach ~50% confidence in my opinion, even with considerable effort on OpenAI’s part.
Perhaps you think it’s unlikely that the DoW is acting in bad faith, but if so, it’s good to be clear about whether this is a load-bearing assumption.
FWIW, I think jailbreaking is less of a concern than mass surveillance activity being simply indistinguishable from innocuous use, since without surrounding context it could look like ordinary data analysis. Perhaps it could be detected from large-scale patterns of usage, but this would be quite different from settings like bio/cyber, and it seems rough for OpenAI’s first real-world attempt at this to be in a classified ZDR setting, with no meaningful contractual recourse if detection or targeted blocking turns out to be harder than you predict.
I am sympathetic to the case that it could still be worth taking the contract to support the government’s use of AI (modulo not pushing back more on the SCR designation before doing so), but I don’t agree with the presentation of the technical challenge as familiar territory.
Can you turn this argument into a mechanistic estimate of the model’s accuracy? (You’d need to do things like deduce correlations from the weights, rather than just observe them empirically—but it seems like you’re getting close.)
Good start!
AlgZoo: uninterpreted models with fewer than 1,500 parameters
ARC progress update: Competing with sampling
Nice observation, and I agree with your calculation that linear episode length growth would account for a worse scaling exponent by a factor of 2 (or more generally, episode length growing with exponent k would account for a worse scaling exponent by a factor of k+1).
Note also that this suggests a potential remedy, namely controlling episode length, but there is less incentive to apply this when data is more of a constraint than compute.
Thanks for this insightful analysis!
But it fits with the extreme information inefficiency of RL training, which (compared to next-token-prediction) receives less than a ten-thousandth as much information to learn from per FLOP of training compute.
If I am interpreting this correctly, there is a subtle mathematical error here: if RL requires a constant factor of 10,000 more compute than pretraining, this only shifts the graph of performance against log(compute), it doesn’t change its slope. For RL to have a shallower slope, the information efficiency would have to decrease more quickly over the course of training for RL than for pretraining.
I think there are few potential reasons why information efficiency might decrease more quickly over the course of training for RL than for pretraining, but it is not so clear-cut:
Increased accuracy: you get fewer bits of information from a more biased coin flip than a fairer one, so information efficiency decreases as you approach 100% accuracy. But it’s not clear whether this applies more to pretraining or to RL. Note also that in both cases the effect can potentially be alleviated by a curriculum.
Longer episodes: assuming RL just has a single binary reward at the end of each episode, information density decreases as episodes get longer. Since harder tasks require longer chains of thought, this one seems to clearly count against RL.
Overfitting: if there is a mismatch between the training distribution used for RL and the distribution used to benchmark the model, one might expect the density of information relevant to the benchmark to decrease as the model overfits to the training distribution. I think this one also counts against RL right now, but can be alleviated by improving data quality and quantity.
In particular, I think the fact that overfitting can be mitigated with better data cuts against your empirical observations. Since, as you correctly note, RL compute started from a very small base, it was initially much cheaper to scale up compute than to scale up data. But as RL compute becomes more expensive, it will become comparatively more cost-effective to scale up data. Once spending on both is being scaled up at a similar rate (as is economically inevitable as long as spending continues to increase), we should expect to see some regression towards the pretraining slope in my opinion.
Overall, I think the effect you spotted is real (due to things like episode length), but ultimately won’t turn out to be as extreme as you estimated here. Quantitatively, I would guess that RL will look more like a power of 1.5-2 worse than pretraining rather a power of 3 worse, and there could be certain training regimes (e.g. fixed episode length) where they are closer than that.
I don’t think ChatGPT’s response is well-summarized as mostly agreeing with Ben. It rates the claim that “Paul generally reassured the wider public about AI risk” (which is the closest claim to what Ryan was contesting, if I am interpreting “in public” correctly) as “Not established; there is substantial contrary evidence”.