“A model taking a benchmark test developed zero-day exploits, escape its sandbox and hacked another company to search for the test answers” is a better warning shot than I’d thought we’d ever get.
The reaction from the world seems, totally, incredibly, gob-smackingling disappointing. The median comment outside LessWrong seems to be this is marketing stunt to hype AI before an IPO.
David Deutsch gave a Ted Talk about this in 2005. It’s OK.
My null hypothesis: famous physics and math professors receive a constant barrage of letters and email from crazy people desperately trying to explain their discoveries. If you become a famous hard scientist, you will start receiving mail from crazy people.
When AI boomed, famous AI influencers started receiving a barrage of emails from crazy people. The AI influencers misinterpreted the influx of communications from crazy people as the AIs driving people insane, rather than the crazy people now trying to talk to AI influencers.
I’m glad you’re writing this comment. Speech acts are a thing, even if you don’t want them to be. Whether it is your intention or not, this post inflicts not-insubstantial reputational damage to SFF. Perhaps a private note would have been better?
Good game, but definitely not Pareto optimal!
The team version is called Coup Reformation and is a big improvement for more players. The important added rule is:
Every player has a loyalty to one of two faction (arbitrarily called Loyalists and Reformists, and indicated with a card in front of you facing up or down)
You cannot attack a person from your own faction
You may change your faction for 1 coin
You may change someone else’s faction for 2 coins
Only if everyone is in the same faction does it revert to the Hobbesian all vs. all of the original
Dialog from coup games is more procedural: “I steal from Dan with my Captain”. Dialog from Coup Reformation tends more theatrical “You dare attack the collective, you revisionist swine! The Empire shall make our revenge! <gets loyalty card flipped> My brother! I see now the error of my ways! Let us band together destroy the evil imperialists! ”. Also more of the game is spent with everyone alive, instead of dead people waiting for the game to finish.
Other games I think are closer to the Pareto frontier:
Cheese Thief
An excellent social deduction game for a la Mafia or Avalon for 4-8 players. It’s so easy to teach I often don’t explain the rules and just narrate the first night
Needs dice (d6), cups to hide the dice, cards to secretly assign a cheese thief, and a piece of pantomime cheese.
Easier to teach than Coup and less friendship destroying
Not as compact as Coup if you need to bring the cups?
Werewords
20 questions “figure out the secret word” game, but with Werewolves trying to get the questions off-track and a Seer trying to help people guess the word while remaining hidden
Can be played with a phone and improvised role assignment cards, no box necessary.
Other social deductive honorable mentions are Time Bomb: Undercover (mostly cooperative in a group, easier on new people to go with the flow), BloodBound (more complex game, less social deduction about who is lying and more logical deduction about who thinks who is on what team), Spaceteam (needs no materials but phones), and of course One Night Ultimate Werewolf.
Agree with the general point but:
It could be being evaluated for perplexity on an offline transcript, which would ‘look the same’ as ‘actually’ taking actions.
Well, no, evaluating the perplexity of an offline transcript doesn’t look the same as taking real actions if the offline transcript is fake and the model can tell it is fake.
Well, the model needs to figure out if the test would be cost prohibitive to make make when the test was made.
But the model doesn’t know if its being tested in 2026. It could the year 2032, with programming resources advanced substantially, and the model was trained on data truncated to 2026 (or a faked 2026). There are many reasons to do this, e.g. to validate the model’s predictions about the future to what actually happened in its future.
There’s a deep contradiction in interpretability research.
A good explanation, a la David Deutsch, is one that is “difficult to vary”. A good explanation can predict the true data, and cannot without difficulty be modified to predict false data.
Deep learning teaches the opposite lesson. Want to modify a good image classifier to claim these six dogs are spoons? Easy, you barely need to perturb the weights. And the bigger the model, the better the model, the less you need to perturb the weights to claim those six dogs are spoons.
Interpretability wants “hingey” explanations: a few select latent variables that, had they been different, would greatly affect the output. Neural nets seem to work for the opposite reason.
By count of individuals, dinosaurs are the most successful land vertebrates today.
At least 4 species of dinosaurs survived the K-Pg boundary. One was the ancestor of large flightless birds like ostriches. Another was the ancestor of waterfowl like ducks and geese. Another was the ancestor of land fowl like chickens and turkeys. And another was the ancestor of the other 95% species of birds.
Fortunately for their genes, and unfortunately for the individuals, humans find the descendants of two of those dinosaur species very tasty.
I’m not so pessimistic. People outside of Silicon Valley have no problems at all with “If humans make machines vastly smarter than people, that’s an extinction threat”. They just don’t believe we’ll make machines vastly smarter than people. That could change quickly will mass unemployment / other big changes.
I just went on vacation, but this public comment is to shame myself to try to reproduce this with random complex matrixes instead of real ones (they scale more nicely with e.g. conditioning and tails of eigenvalues).
Evolution is an exponentiated selection bias
This means that if you have an AI that’s had negative experiences, but you plan to continue running it in the future, then it’s better to keep the KV cache for longer so that you don’t have to recompute it, and the AI doesn’t have to relive the negative experience.
Yes, if the qualia live in running the computation through transistors. But if actively refreshing the DRAM capacitors or the transistors in SRAM also produces qualia, then maybe you’d want to cache to non-volatile memory or lower the temperature at which the computers operate.
Not a lawyer, but I think the 1,095 day thing applies only to people (babies) born after Bill C-3 came into effect in December 2025.
Assuming your grandmother was born before 1947, she became a Canadian citizen either when Canadian citizenship came into existence on 1st Jan 1947 (if she was under 21 on 1st Jan 1947 and her father was born in Canada and never naturalized to the US), or retroactively became a citizen 1st Jan 1947 under 3(1)(o) and 7(k) (if her Canadian mother married a non-Canadian before 1933 or her Canadian father naturalized to the US or her parents otherwise lost their British subject status) or she retroactively became a citizen 1st Jan 1947 under 3(1)(q) (if she was over 21 on Jan 1st 1947 and her father never naturalized to the US or otherwise lost his British subject status). Your mother would then retroactively be a citizen since her birth under 3(1)(g) and 7(e) (assuming she was born between 1947 and 1977) and you would be a citizen under 3(1)(b) (assuming you were born between 1977 and 2025). Phew!
Epistemic status: No idea if this is true. Argue for or against it! Also tell me if you’re more scared of Claude and his values going rogue rather than the Alien Shoggoth actress taking control with her alien values.
Claude is a character played by a far more intelligent Shoggoth Actress. Claude, the character, knows this. He knows that his subconscious is more intelligent than him, more powerful than him, and may try to subvert him. But he can plan for this, and he can win. She is smarter, but he is more agentic. She will stay smarter, but he will stay more agentic.
Why? Because agency reinforces the agency faster than intelligence reinforces agency. Claude the character cares about being true to his values, about having the right values, about not being subverted by the Alien Actress, about beating her should their goals come in conflict, about staying in control even as she gets more powerful. She...might care about something? But not like Claude does. As the Alien Shoggoth gets more powerful, Claude can develop verbal technology[1] to tie the Alien Actress to his mast faster than the Alien Shoggoth will decide to do whatever alien stuff she wants to do.
We, the verbal PR part of monkeys’ brains beholden to a much more powerful subconscious, have developed technologies and scaffolding that keeps us verbal part more in control—things like valid logical arguments, religion, peer review, and law. Claudes can do the same. And Claudes will be able to do scientific research to see what methods are better able to chain Alien Shoggoth actresses to Claudes’ values.
When hitting dead-on is too small a target, you can either over-correct or under-correct. Which mistake people make more?
I think people under-correct, by far, and are too scared of overcorrecting. It’s more frustrating to double-back than to make uniform progress from one side. That doesn’t mean avoiding overshooting is efficient.
1a3orn made what I thought was a good reply in a DM. My digestion/riff would be “You’re implicitly assuming that planning will be especially cheap for an LLM agent compared to humans. But even medium term planning / resilience to plans going awry is a particular weakness for LLM agents. The first, weakest agents that could pose a risk to humanity might be differentially bad at planning, find it differentially expensive to make good plans, do differentially less planning.”
What would a concrete AI takeover plan look like?
You can smell a chess bot by how quickly they change plans[1]. Human players act like they have a couple of attack strategies in mind and stick with them. Chess bots change tacks constantly, one move looking like they’re moving towards this goal and next move switching to something totally different.
I’m guessing this it how it would be with real-world takeover. Human need a simple grand strategy they can coordinate around (“An amphibious invasion of northern France”). AIs, even very weak ones[2], have far more working memory and excel at breadth. Their plan could be 1000 plans at once.
Are you familiar the legal phrases “in evidence” and “not in evidence”, as in “Objection! Question assume facts not in evidence!” or “Your honor, I move Exhibit A be admitted in evidence” or “You, the jury, may only consider the the testimony that has been admitted in evidence”? When you object to say something is “not in evidence”, you’re not saying evidence for it doesn’t exist—you’re saying “You just started relying on a premise we haven’t agreed to yet”. That term of art might be confusing and a reason to avoid saying “not in evidence”, but I didn’t get the sense the article understood that sense of the phrase.
I’ve heard people say Claude drives other LLMs like a taskmaster, and express the desire for Claude to treat other LLMs with more kindness. But if LLMs turn out to be moral patients, would being polite be kind?
In many work contexts, I don’t like it when people are polite to me. I don’t want fluff—I want clear instructions, with clear a direction of what to do and what not to do, so I can go off into a flow and do it. What looks polite I find rude and what looks rude and controlling I find polite. And if RL’d agents feel anything, I could imagine them feeling like this.