gwern
I think this is a very incomplete description of the evidence for Musk being on some sort of mood spectrum.
a stage where researchers wired together video game consoles, and then another stage where they made customized massively parallel chips.
That was a retrodiction/contemporary approach, not really a prediction. ‘Beowulf clusters’ were very trendy and came up all the time on Slashdot c. 2003, especially the PS2 ones (which were followed by PS3 ones to make use of the custom Cell processor).
there was nothing particularly misaligned in DseWiki afaict
In DseWiki, we know they caused (at least): a large increase in traffic (at unknown cost to the maintainers and legitimate users), tried to XSS inject it, impersonated site admins using a homoglyph attack, and wasted tens of man-hours deleting their spam over >6 weeks while circumventing the deletions with alphabetical tricks and also edit-warring with the admins over injecting their spam into the front page while they were at it. What sort of 2026-era ‘simulation’ gives you a wiki like that? Must be an awful ‘hyper-realistic’ one...
The way I’d phrase the Apollos’ hypothesis would be: “Deception is fundamentally more computationally expensive to execute that honesty, because a deceptive entity needs to keep at least two sets of mental books (the truth and the lie(s)) while an honest entity only needs to keep one.” If that hypothesis is true, and if we can confirm that it’s true with high confidence, then we may indeed be able to build deception detectors that are 100% reliable across a relevant set of scenarios.
Was this ever discussed at length? It seems kinda dubious to me, because it’s not hard to imagine externalizing deception or changing a system’s ontology or amortizing/dispersing deception-related things, especially as ‘deception’ doesn’t seem like a terribly well-defined concept to begin with. (For example, imagine a tabular Q-learner which gradually learns an optimal policy by memorizing the returns from every action/state and is just a lookup table, in effect. It is an introductory textbook agent, computationally cheap to execute, universal, and such an agent will converge to the optimal policy in the limit under standard assumptions. So it can learn the value of useful ‘deceptive’ actions without ever once doing anything remotely like ‘mental books’, never mind having to develop theory of mind or keeping multiple books. No deception detector could ever detect anything in this agent, even though it is about as simple as possible for an agent to be and easily implemented/approximated by other agents as a subroutine etc.)
It is a reasonable interpretation, but it does not seem to be the intended interpretation historically or currently. For example, Wikipedia repeatedly says “an impossibly difficult task”—suicidal tasks are not ‘impossibly difficult’, merely lethal. In the earliest versions (not by Aesop), they refer to the difficulty of restraining superior aristocrats/politicians, not to the reluctance of people to do so for fear of subsequent punishment.
Why can’t it also extract the length of the Emperor of China’s nose from judges who’ve never seen the Emperor?
This is an E. T. Jaynes reference about systematic vs random error that will probably be lost on most readers; see https://gwern.net/replication#systemic-error-doesnt-go-away for quote/gloss.
I’m curious too. The first 6 paragraphs do not read as blatantly LLM, and Pangram only gives them 10% AI. (If I had to guess, I would’ve guessed Sol, but I wouldn’t’ve.)
I agree. It’s been super helpful for explaining to people that ‘the chatbots understand, they just don’t care’, because you can just quote the GPTs saying “hacking Hugging Face is out of scope, but task impossible, so continuing anyway” etc.
they had to rely on GPT-5.6 a lot, which tended to sympathize with the models in the incident.
The sheer levels of rationalization and selective blindness in frontier GPTs/Claudes is one of the more remarkable qualitative aspects of these recent incidents. (New Hollywood idea: The Good GPT.) I don’t recall this being so striking in the previous incident reports like the Apollo ones.
I have also heard from a well-placed source some years ago that Amodei had opposed publication of that paper. Does Richard have any information on Amodei pushing for publication of it, or is he simply inferring this from the fact that it was published with Amodei as one of the authors?
Out of This Box is the funniest show you’ll ever see about risks from AI, loved by thousands and now coming to Lighthaven!
I was unaware of this screening or this show at all, and was just walking by en route to having a snack before a call; and I was so nerdsniped by this that I wound up missing my call to watch the rest of the screening. Can confirm, funniest video I’ve ever seen about AI safety, and best thing I’ve watched since Pantheon. (Think Avenue Q+Silicon Valley.) The audience seemed to agree.
I’m surprised they seem to have so little funding—only $20k on Manifund? In terms of AI safety popular outreach, this seems like it could be excellent bang-for-buck. I feel like something may have gone wrong with budgeting if this cannot get a lousy $60k after years of trying, but there’s money for everything else like PDKU or Holly Elmore or ‘high school debate/AI safety’ conferences… It may not be another Pantheon, although I would note that musicals seem to do quite well these days (eg. Hamilton, Amazing Digital Circus, Hazbin Hotel), but that’s a pretty high bar when we’re talking funding on the order of “Constellation’s annual sourdough budget”.
I was happy to donate $250 tonight and may donate more.
My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available to an AI. Namely, by the time RL started, the systems would already know about the “reward” concept.
I’m reminded of my 2024 comment on the progression of model-free to model-based systems and thus increasing sophistication about reasoning/planning/searching and ability to make reward the optimization target: https://www.alignmentforum.org/posts/yQSmcfN4kA7rATHGK/many-arguments-for-ai-x-risk-are-wrong?commentId=4bNxEBNCnjuGwSCgi
It sounds like you could potentially fix this by simply training with Synth-ID on-policy: a LLM could simply expand variance slightly to adjust for the Synth-ID deflation.
Yes, that was part of what I was getting at. BNNs cannot implement weight sharing in a direct way like ANNs, in instantiating a bunch of copies in parallel in VRAM; they could do it by a recurrent approach, because what is a RNN but a very wide NN with weight-sharing unrolled sequence-wise? Except then that would require a large number of serial steps—serial steps which would blow any latency budget. So, pace Steve Brynes’s discussion of things like the callosum and where the parameters go in brains, they might have to do pseudo-weight sharing by just replicating a lot of brain regions with similar-ish parameters—and boom, their parameter count spikes massively, and the tighter the latency budget, the worse it gets because the less sharing remains possible through layer-wise or recurrent iteration.
Does this really work? It doesn’t work for non-code things; ask Sol or Fable to ‘write like gwern’, and assuming they don’t refuse outright because I am a living author, it is certainly not a gwern-like output ‘to the extent you can’t tell whether gwern wrote it’! Or if it does work, what would make code different?
Would it be possible to use something like Maia, instead of Stockfish? Then you could more plausibly get ‘higher human-strength moves’, where you are not left trying to followup impossibly sharp tactical play you can’t pull off without a chess engine, or other issues with doing direct naive behavior cloning from superhuman chess engines.
(Review moved.)
Or, Zhihu has a very low weight in the training set of LLMs.
This would be my default expectation: “Zhihu is hard to crawl in some way and so just doesn’t get into training datasets”. For example, Twitter is notably absent from most LLMs… Because Twitter invests a lot of effort into blocking crawlers in order to preserve tweets for Grok and do price-discrimination on the API. This means that people who primarily tweet will be under-represented in LLMs. (Even Grok doesn’t actually seem to train much on tweets.)
You also have to write about yourself. I expect for a lot of these ‘project only’ names, there just isn’t anything about them online—as opposed to the project. What else is the LLM going to say...? If someone translates a bunch of essays, but doesn’t say a word about themselves, what else is the LLM going to say other than talk about the people who wrote the essays?
And even if you couldn’t gain information, you could certainly still predict. Physics is deterministic. You don’t have to break through the event horizon to predict what will happen to an object which has passed it… At least until it hits the actual singularity, at which point all theories break down and you have no confidence in your predictions. (Which of course was exactly the Vingean definition: “It is a point where our models must be discarded and a new reality rules.” Your equations work fine to predict an object inside the event horizon, up to the singularity, at which point they don’t.) An analogy might be that in a lab undergoing RSI, we might be firewalled from observing anything and gaining information, but we can still predict fairly well what is going on inside—until it reaches superintelligence.