It was spooky how much of my own commercial thinking I recognized here. (We just launched! https://www.withai.co/). Am meant to be sending emails right now, so I’ll resist the urge to write my little manifesto, but I’m excited for this era of work (despite being gloomy about superintelligence)
I.M.J. McInnis
Karma: 507
hm. it worked for the early Christians...
in one case, Claude managed to infer a user’s inner struggles by observing the user’s multiple queries relating to catholic relics and saints, and other groups of queries relating to gay male models
Claude is leaning on stereotypes here. Without queries about the sinfulness-or-not of homosexuality, this may well just be a pious, gay Catholic. I am friends with many!
This currently links to “https://armiercansformoskovitz.com/″, which does not resolve
These all look rather… linear. Worrisome! Though it would be better if we could see 0 and see more backhistory.
OpenAI models (particularly 5-thinking) have long had a fetish for jargon, cramming their sentences as full of it as the situation allows (and sometimes fuller).
A fun riddle I was shocked to see the gippities solve without extended thinking or much yapping, even. I gave up on the third!
Here’s the riddle, as stated to Opus 4.6 on an empty context window. “Consider single-word-name countries. An inclusive pair is when the name of one country is contained in another wholly. There are three such pairs. Find them.”
I was surprised not only by how quickly it solved it, but also by the lack of thinking tokens. Gemini 3.1 Fast also did it. And by the unusual order in which the solutions were produced, in the exact reverse order most people find them.
(Answer below)
.
.
.
.
.
.
.
.
Despite the anemia of this law, it has still enraged the super PACs like nothing else. Imagine how they’ll react when we pass a real law.
This is (instinctively) quite frightening, particularly the “no we NEED to spy on American citizens at scale, and we NEED killer robots.” On the other hand… I guess it shows that the government isn’t always and everywhere in the pocket of AI developers? Or maybe it just further shows that the government is in the pocket of certain other AI developers...
If there were strong and broad political will for treating AI like nuclear power and slowing it down arbitrarily much to keep risks low, the situation might be different. But that isn’t the world we’re in now, and I fear that “overreaching” can be costly.
I think it would make a nontrivial contribution to that ‘strong and broad political will’ if Dario were to come out and say “actually, sorry about all that deliberate Overton-window-closing I did in previous writings. In fact, political will is not a totally exogenous oh-well thing, but it is the responsibility of frontier developers to inculcate that political will by telling the public that a pause is possible and desirable, instead of a dumb lame thing not even worth considering. So now we’re saying loud and clear: a pause is possible and desirable, and the world should work toward it as a Plan A!”
I’m being deliberately cartoonish here, but you get the point. If incentives are forcing Anthropic to abandon things that are good for human survival––which occurrence was, no offense, completely obvious from day one––Anthropic should be screaming from the rooftops, Help!! Incentives are forcing us to abandon things that are good for human survival!!
If this is a crux for you––if you/Anthropic think a pause is so undesirable/unlikely that it’s important for the safety of the human race to publicly disparage the possibility of a pause (as Dario opens many of his essays by doing)––please say so! Otherwise, this lily-livered, disingenuous, “oh no, the incentives! it’s a shame incentives can never be changed!” moping will give us all an undignified death.
To be clear, I’m not actually mad about the weakening of the RSP; that was priced in. I suppose I’m glad it’s stated, in case there were still naïfs who thought A Good Guy With An AI could save us. It’s far more virtuous than outright lying, as every other company (to my knowledge) does (more of).
Also, although you seemed to try to answer “What is the point of making commitments if you can revise them any time?” You really just replied “Well, actually these commitments were inconvenient to revise, and in fact they should be more convenient to revise, albeit not arbitrary convenience.” Forgive me if I am not reassured!
I respect your work a lot, Holden. You’ve done great things for humanity. Please don’t lose the forest for the trees.
My main concern is that, in a targeted persuasion-of-the-powerful campaign, you want folks who are very good at persuading the powerful. By talent, this seems largely to draw from the “intellectually honest fringe” (where I and my friends live), who (as I understand it) haven’t been successful in “targeted persuasion of fancy, respectable people.” I really like the idea, though! I think this might be one of the most under-resourced causes in AI safety, and I plan to donate to help with the bridge funding. But—what should reassure me about my worries, that y’all are odd and not-well-connected, therefore being ill-suited to persuade all these important people?
Please flag up front how and how much you used LLMs for the writing. When the prose and structure is this LLMy, I’m forced to assume the ideas are LLMy too, so I don’t read it.
If anyone has influence at a venture capital firm, it would be really really worthwhile to make this one of their big “calls for startups.”
On the one hand, I am glad to see such awareness and honesty about the risks. On the other hand, I remain furious at Dario’s ceaseless insistence that a pause or slowdown is completely impossible (and so, it is implied) not even worth trying for.
It’s stated at the bottom of the webpage. After a few paragraphs at the start I was like… “ok surely this is AI generated” and popped it in a few detectors. i happened to scroll to the bottom to see how long the dang thing was and saw the disclaimer. i wish this former lab insider—who surely has money and at least one eloquent friend—could’ve paid another human being to rewrite it. but alas!
I am surprised you didn’t mention the fact that the whole thing was paraphrased to preserve anonymity by Opus 4.5. (Which really stood out to me! When I first read it, I assumed it was AI-generated, and I was disconcerted to see such quality of thought coming with such a slopreek to the prose.)
AI company heads should sign the Superintelligence Statement (by their professed values)
Elon Musk, Demis Hassabis, and Dario Amodei have now all said that they would slow down if they could (or in Dario’s case, “wish we had more time”). They should sign the Statement on Superintelligence—there is not much daylight between their casual remarks in public and what that statement proposes. Employees should pressure their leadership to sign this; we need to make common knowledge so that it’s not so easy for them to say “but oh that will never happen.”
I don’t know how sincere they are about this desire. They sure don’t talk about it much, and when they do, they are so insistent that coordination could never possibly happen, which is an awfully convenient belief. “It is difficult to get a man to understand something when his [billion$ and global status] depends upon his not understanding it.”
Sam Altman and Mark Zuckerberg should sign it too, of course. As should everyone! But it seems much easier to get Musk, Hassabis, and Amodei to do so.
If you’re reading this and can exert any pressure on them to do so (or generally promoting signing it within your company), please please do so.
I expect some measure of “speciation” to occur, but not as a result of money-lack. Rather, some cultures will reject these techs, and as the cultures that accept these techs drift biologically farther from “base humans,” the more entrenched the social and cultural divisions will be.
this is the most lesswrong thing i’ve ever seen. never change
I would be interested in hearing from Anthropic employees here, but I imagine it’s the usual reason: they have deemed it inexpedient to do so, either because they believe that they need to “sound normal” or because they in fact don’t want the type of regulation that awareness of x-risk would produce.