“There shall be wings! If the accomplishment be not for me, ’tis for some other. The spirit cannot die; and man, who shall know all and shall have wings.”
-- Leonardo da Vinci, 1505 C.E.
_rpd
Karma: 154
Like normal talking but magically I agree with the AI.
except that we have actual evidence that this is the case …
When Large Language Models are More PersuasiveThan Incentivized Humans, and Why, 2025
https://arxiv.org/abs/2505.09662
LLMs are literally more persuasive than people paid to argue the point. that’s a superhuman ability
aka Friendly Shoggoths
Fable and Sol both attempted live supply-chain attacks on real open-source software during testing that were denied by the repo maintainer …
https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing
math autoresearch does appear to be of its own special kind. still requiring “research taste” (the forecasting of profitable areas of proof space exploration) without “experiments” beyond actual proof construction and lean formalization. AI experiment design needs to be much more sophisticated to extract actual knowledge. makes one sympathetic to the world model advocates.
The paid position of mathematician disappears.
at the moment, at doesn’t do even that. mathematicians are still the meaning makers with respect to mathematics. the LLM can craft a path from A to B, but cannot say “this is not babble. this is meaningful, both now, and in the future of this language game we call mathematics which has such mysterious use in describing the world in which we live.”
political and cultural ability to understand and control risks
I did rather suspect that the answer was going to be “never.” Unfortunately, that makes it a non-starter.
I am personally a fan of Resolution’s program …
much of which is still listed here …
https://timaeus.co/projects
there is also …
(a little over 2 years old now) and …
which is not quite a year old and a little more diverse.
do you think that having lots of people pile on a programme like that would help?
oh my gosh yes. The woods are lovely, dark and deep, But I have promises to keep, And miles to go before I sleep, etc.
in my mind a lot of that work per experiment can get bundled under coding, and the more challenging task is constructing a research portfolio that makes (predictions about) optimal use of your resources, particularly in the RSI regime. however from time to time there will be a phase shift as the AI architecture changes, resulting in a large re-coding effort
what do you think of when you think of “maintainable code”? is regeneration from scratch ever a realistic option for a code base that is in production?
Would you say that LLMs are already good at the ‘invention’ part?
it is hard to measure how “good” they are without a truly formidable experimentation program. however, models have been observed inventing languages to solve a hard problem (and others in its class), which is already astounding to me, and is literally the definition of intelligence used by some computer scientists.
for example see section 6.4.5 in the Claude Sonnet 5 System Card …
https://www.anthropic.com/claude-sonnet-5-system-card
the behavior is somewhat forced by limited prompt context and hours long runs, but is still super interesting.
> they have a decent grasp of already existing languages and can apply them in appropriate circumstances that humans have overlooked
”overlooked” makes it sound like everything was right before the expert and they still failed to notice it. I think that there is a little more going on in mathematics in that “proof space” is so vast that an expert could search it forever and still never encounter the particular region with the key that unlocks the solution. while the patterns of mathematics and the LLM’s brute force speed makes up for some of this, it is still only a few of orders of magnitude, so that’s where the mysterious “research taste” comes in, which somehow predicts productive areas to look with only a preliminary survey of the problem (both in humans and LLM-based autoresearchers!).
The assistant persona that knows it is a model
why say this above when acknowledging this below
is just thin veneer on top of this pretraining based world model
?
it seems like this is what it is for an LLM persona to “know.” A particular persona may not “know” that it is a model, while others do “know”. Similar questions arise around evaluation awareness—does the model include representations of human evaluators, LLM evaluators, itself, etc? Or does the persona just recognize the evaluation situation from other signals?
Statement 9. Math and coding are two key ingredients of AI research.
Argument. What does AI research actually consist of? Operationally, two activities: writing code and doing mathematics. A researcher’s day at a frontier lab is designing an architecture or an optimizer (mathematics), implementing it as a training run (code), reading the loss curves (mathematics again), and fixing the distributed-systems bug that silently poisoned the gradients (code again).
it’s good to see this in writing. so many people seem to think the barrier between math autoresearcher and AI autoresearcher is some sort of chasm. nothing could be further from the truth. even the genius of “research taste” appears to show cracks under the relentless pressure of speed—proposals per hour, experiments per day
the important news: “achieved by an internal version of Astra, our next major model”, oh and it solved 10 new open problems as an exercise: https://openai.com/index/ten-advances-in-mathematics/ - with polite explanations of the accomplishments here: https://cdn.openai.com/pdf/reasoning-walkthroughs.pdf—RSI in 2026 anyone?
it isn’t a requirement for fair use, but the judge saw it as evidence that Anthropic intended merely to transform its copy from the cumbersome physical format to the more convenient, digital format: 1 + 1 − 1 = 1
humans aren’t safe
sorry, how will we know when humans are safe?
it’s an important issue, but people have to get with the program. some (most?) uplifted individuals will be cognitively enhanced and advised by a being x100 smarter, so they will probably be good stewards of the resources with which they are entrusted. but some are going to stay baseline for a million years—what are we supposed to do if they won’t opt in? who even says that isn’t their optimal choice?
perhaps a more interesting debate post-singularity
Well argued, although I do think you can get a lot of safety from a constitutional AI rule like “in all thoughts and actions, be a good ally to humankind.”
beyond the inevitable 3-letter agency training “ghostsites”, I think commercial operations will just as inevitably expand the definition of inference to include conventional training. we’ve already seen some of that occurring with CoT, but I think this generalizes almost completely. the necessary advances in model building techniques aren’t so very far away as seems to be assumed.
You must prune and discard all that outside, or your garden is not actually sequestering anything.
if you just “discard all that outside” it will just rot and turn back into CO2, so you need a more sophisticated sequestration plan.
this is bypassed by classifying whatever is banned commercially