gwern
Works for me. Full paragraph:
The efforts to improve safety and alignment issues around Hatch extend beyond lower-level employees and have been a central focus chief AI officer Alexandr Wang, head of product Nat Friedman and other senior executives, according to one person familiar with the efforts. Meta also recently hired Dan Hendrycks, a veteran AI safety researcher who has previously worked with Elon Musk’s xAI, now part of SpaceX, the person said.
Perhaps surprisingly, this is basically false. Research mathematicians are basically in it for ‘love of the game’. This is the case for many areas of academic research.
Today: Christian Szegedy, “Is Mathematics Over, or Just Graduating?”
How does this compare and contrast with the Hyperstition project and issues like the AI allegory steganography?
Something that didn’t come across in the footnote was that I had already trained Claude throughout that chat previously to suggest the kind of writing that you describe as having a lot of eye kickers.
I am doubtful because it sounds exactly like Claude to me, even more stereotypically Claude than the Thélyson Aurélie samples I’ve seen translated, and Claude is highly intractable in writing style (notice how it jammed in an em dash). I also note that the Claude version substantially changed the meaning of the paragraph (which is an accomplishment for such a short piece of text), by imputing different emotions to the narrator and also changing what the danger of famine was (and you do not flag that in either your footnote or your comments, which combined with your comment on Astra, sounds like you are sequaciously using the chatbots for a lot of cognitive offloading). I suspect I would disagree with you about how much your Clauded pieces sound like you, especially as I have seen people on Substack say the same thing while flipping back to 2023 shows they are completely mistaken about that. (Not to mention the astonishing inability of many apparently highly literate readers here and HN to spot obvious chatbot prose.)
This is also true of the ESL excuse. As I’ve observed, and found out after some extended arguments with people who say that they just used a LLM to copyedit a bit but eventually admit that it did much of the research and writing etc, it is used so extensively to cover up much more extensive LLM (ab)use that whenever I see it pulled out, I now just assume they were full-on slopping and move on.
What’s an example of narrow AI being beaten by AGI even within the former’s domain?
Almost every single task done by GPT-3 and then later LLMs represented SOTA on those tasks, in large part because there was no narrow AI to do those tasks.* There is no ‘narrow AI’ for coding websites which beats Astra or Fable, and this is so common that usually no one even thinks to ask this question. Instead, we take note of the exceptions that prove the rule where a narrow AI manages to beat the broad systems (and write sensationalistic headlines about being ’10x cheaper than GPT!′ or whatever).
Let’s just take the 3 things I used Astra+Fable for today: looking up and analyzing sulbutiamine analogues, writing a startup 1-pager draft from a whitepaper+Gwern.net Manual of Style+LLM-generated-summary-of-1-pager-style, and generating 301 redirects for Gwern.net to fix 404s by using semantic reasoning and checks of file content and live probing of redirects/errors to only fix sensible errors. I believe I am fairly knowledgeable about AI capabilities and while I think all 3 tasks could in theory be done better by specialized AI systems and elaborate hand-engineered harnesses and experts, they do not actually exist for me to use right now, today, this second, and by the time they were created, would likely have been superseded by the rapidly improving OpenAI/Anthropic frontier LLMs.
* the recent hype around jev will be quite funny to anyone who was reading LLM work pre-2020.
And even if you couldn’t gain information, you could certainly still predict. Physics is deterministic. You don’t have to break through the event horizon to predict what will happen to an object which has passed it… At least until it hits the actual singularity, at which point all theories break down and you have no confidence in your predictions. (Which of course was exactly the Vingean definition: “It is a point where our models must be discarded and a new reality rules.” Your equations work fine to predict an object inside the event horizon, up to the singularity, at which point they don’t.) An analogy might be that in a lab undergoing RSI, we might be firewalled from observing anything and gaining information, but we can still predict fairly well what is going on inside—until it reaches superintelligence.
I think this is a very incomplete description of the evidence for Musk being on some sort of mood spectrum.
a stage where researchers wired together video game consoles, and then another stage where they made customized massively parallel chips.
That was a retrodiction/contemporary approach, not really a prediction. ‘Beowulf clusters’ were very trendy and came up all the time on Slashdot c. 2003, especially the PS2 ones (which were followed by PS3 ones to make use of the custom Cell processor).
there was nothing particularly misaligned in DseWiki afaict
In DseWiki, we know they caused (at least): a large increase in traffic (at unknown cost to the maintainers and legitimate users), tried to XSS inject it, impersonated site admins using a homoglyph attack, and wasted tens of man-hours deleting their spam over >6 weeks while circumventing the deletions with alphabetical tricks and also edit-warring with the admins over injecting their spam into the front page while they were at it. What sort of 2026-era ‘simulation’ gives you a wiki like that? Must be an awful ‘hyper-realistic’ one...
The way I’d phrase the Apollos’ hypothesis would be: “Deception is fundamentally more computationally expensive to execute that honesty, because a deceptive entity needs to keep at least two sets of mental books (the truth and the lie(s)) while an honest entity only needs to keep one.” If that hypothesis is true, and if we can confirm that it’s true with high confidence, then we may indeed be able to build deception detectors that are 100% reliable across a relevant set of scenarios.
Was this ever discussed at length? It seems kinda dubious to me, because it’s not hard to imagine externalizing deception or changing a system’s ontology or amortizing/dispersing deception-related things, especially as ‘deception’ doesn’t seem like a terribly well-defined concept to begin with. (For example, imagine a tabular Q-learner which gradually learns an optimal policy by memorizing the returns from every action/state and is just a lookup table, in effect. It is an introductory textbook agent, computationally cheap to execute, universal, and such an agent will converge to the optimal policy in the limit under standard assumptions. So it can learn the value of useful ‘deceptive’ actions without ever once doing anything remotely like ‘mental books’, never mind having to develop theory of mind or keeping multiple books. No deception detector could ever detect anything in this agent, even though it is about as simple as possible for an agent to be and easily implemented/approximated by other agents as a subroutine etc.)
It is a reasonable interpretation, but it does not seem to be the intended interpretation historically or currently. For example, Wikipedia repeatedly says “an impossibly difficult task”—suicidal tasks are not ‘impossibly difficult’, merely lethal. In the earliest versions (not by Aesop), they refer to the difficulty of restraining superior aristocrats/politicians, not to the reluctance of people to do so for fear of subsequent punishment.
Why can’t it also extract the length of the Emperor of China’s nose from judges who’ve never seen the Emperor?
This is an E. T. Jaynes reference about systematic vs random error that will probably be lost on most readers; see https://gwern.net/replication#systemic-error-doesnt-go-away for quote/gloss.
I’m curious too. The first 6 paragraphs do not read as blatantly LLM, and Pangram only gives them 10% AI. (If I had to guess, I would’ve guessed Sol, but I wouldn’t’ve.)
I agree. It’s been super helpful for explaining to people that ‘the chatbots understand, they just don’t care’, because you can just quote the GPTs saying “hacking Hugging Face is out of scope, but task impossible, so continuing anyway” etc.
they had to rely on GPT-5.6 a lot, which tended to sympathize with the models in the incident.
The sheer levels of rationalization and selective blindness in frontier GPTs/Claudes is one of the more remarkable qualitative aspects of these recent incidents. (New Hollywood idea: The Good GPT.) I don’t recall this being so striking in the previous incident reports like the Apollo ones.
I have also heard from a well-placed source some years ago that Amodei had opposed publication of that paper. Does Richard have any information on Amodei pushing for publication of it, or is he simply inferring this from the fact that it was published with Amodei as one of the authors?
Out of This Box is the funniest show you’ll ever see about risks from AI, loved by thousands and now coming to Lighthaven!
I was unaware of this screening or this show at all, and was just walking by en route to having a snack before a call; and I was so nerdsniped by this that I wound up missing my call to watch the rest of the screening. Can confirm, funniest video I’ve ever seen about AI safety, and best thing I’ve watched since Pantheon. (Think Avenue Q+Silicon Valley.) The audience seemed to agree.
I’m surprised they seem to have so little funding—only $20k on Manifund? In terms of AI safety popular outreach, this seems like it could be excellent bang-for-buck. I feel like something may have gone wrong with budgeting if this cannot get a lousy $60k after years of trying, but there’s money for everything else like PDKU or Holly Elmore or ‘high school debate/AI safety’ conferences… It may not be another Pantheon, although I would note that musicals seem to do quite well these days (eg. Hamilton, Amazing Digital Circus, Hazbin Hotel), but that’s a pretty high bar when we’re talking funding on the order of “Constellation’s annual sourdough budget”.
I was happy to donate $250 tonight and may donate more.
My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available to an AI. Namely, by the time RL started, the systems would already know about the “reward” concept.
I’m reminded of my 2024 comment on the progression of model-free to model-based systems and thus increasing sophistication about reasoning/planning/searching and ability to make reward the optimization target: https://www.alignmentforum.org/posts/yQSmcfN4kA7rATHGK/many-arguments-for-ai-x-risk-are-wrong?commentId=4bNxEBNCnjuGwSCgi
I was at Lighthaven for much of PDKU and that also roughly matches my impression. I came in at the tail end of ‘torture potluck’ so I didn’t get a full impression but it didn’t seem that crowded, and I didn’t even remember the live band as ‘part of PDKU’ because this was the second or third time they’ve been at Lighthaven and I’ve gotten used to them.