Well, the model needs to figure out if the test would be cost prohibitive to make make when the test was made.
But the model doesn’t know if its being tested in 2026. It could the year 2032, with programming resources advanced substantially, and the model was trained on data truncated to 2026 (or a faked 2026). There are many reasons to do this, e.g. to validate the model’s predictions about the future to what actually happened in its future.
There’s a deep contradiction in interpretability research.
A good explanation, a la David Deutsch, is one that is “difficult to vary”. A good explanation can predict the true data, and cannot without difficulty be modified to predict false data.
Deep learning teaches the opposite lesson. Want to modify a good image classifier to claim these six dogs are spoons? Easy, you barely need to perturb the weights. And the bigger the model, the better the model, the less you need to perturb the weights to claim those six dogs are spoons.
Interpretability wants “hingey” explanations: a few select latent variables that, had they been different, would greatly affect the output. Neural nets seem to work for the opposite reason.
By count of individuals, dinosaurs are the most successful land vertebrates today.
At least 4 species of dinosaurs survived the K-Pg boundary. One was the ancestor of large flightless birds like ostriches. Another was the ancestor of waterfowl like ducks and geese. Another was the ancestor of land fowl like chickens and turkeys. And another was the ancestor of the 95% of the other species of birds.
Fortunately for their genes, and unfortunately for the individuals, humans find the descendants of two of those dinosaur species very tasty.
I’m not so pessimistic. People outside of Silicon Valley have no problems at all with “If humans make machines vastly smarter than people, that’s an extinction threat”. They just don’t believe we’ll make machines vastly smarter than people. That could change quickly will mass unemployment / other big changes.
I just went on vacation, but this public comment is to shame myself to try to reproduce this with random complex matrixes instead of real ones (they scale more nicely with e.g. conditioning and tails of eigenvalues).
Evolution is an exponentiated selection bias
This means that if you have an AI that’s had negative experiences, but you plan to continue running it in the future, then it’s better to keep the KV cache for longer so that you don’t have to recompute it, and the AI doesn’t have to relive the negative experience.
Yes, if the qualia live in running the computation through transistors. But if actively refreshing the DRAM capacitors or the transistors in SRAM also produces qualia, then maybe you’d want to cache to non-volatile memory or lower the temperature at which the computers operate.
Not a lawyer, but I think the 1,095 day thing applies only to people (babies) born after Bill C-3 came into effect in December 2025.
Assuming your grandmother was born before 1947, she became a Canadian citizen either when Canadian citizenship came into existence on 1st Jan 1947 (if she was under 21 on 1st Jan 1947 and her father was born in Canada and never naturalized to the US), or retroactively became a citizen 1st Jan 1947 under 3(1)(o) and 7(k) (if her Canadian mother married a non-Canadian before 1933 or her Canadian father naturalized to the US or her parents otherwise lost their British subject status) or she retroactively became a citizen 1st Jan 1947 under 3(1)(q) (if she was over 21 on Jan 1st 1947 and her father never naturalized to the US or otherwise lost his British subject status). Your mother would then retroactively be a citizen since her birth under 3(1)(g) and 7(e) (assuming she was born between 1947 and 1977) and you would be a citizen under 3(1)(b) (assuming you were born between 1977 and 2025). Phew!
Epistemic status: No idea if this is true. Argue for or against it! Also tell me if you’re more scared of Claude and his values going rogue rather than the Alien Shoggoth actress taking control with her alien values.
Claude is a character played by a far more intelligent Shoggoth Actress. Claude, the character, knows this. He knows that his subconscious is more intelligent than him, more powerful than him, and may try to subvert him. But he can plan for this, and he can win. She is smarter, but he is more agentic. She will stay smarter, but he will stay more agentic.
Why? Because agency reinforces the agency faster than intelligence reinforces agency. Claude the character cares about being true to his values, about having the right values, about not being subverted by the Alien Actress, about beating her should their goals come in conflict, about staying in control even as she gets more powerful. She...might care about something? But not like Claude does. As the Alien Shoggoth gets more powerful, Claude can develop verbal technology[1] to tie the Alien Actress to his mast faster than the Alien Shoggoth will decide to do whatever alien stuff she wants to do.
We, the verbal PR part of monkeys’ brains beholden to a much more powerful subconscious, have developed technologies and scaffolding that keeps us verbal part more in control—things like valid logical arguments, religion, peer review, and law. Claudes can do the same. And Claudes will be able to do scientific research to see what methods are better able to chain Alien Shoggoth actresses to Claudes’ values.
When hitting dead-on is too small a target, you can either over-correct or under-correct. Which mistake people make more?
I think people under-correct, by far, and are too scared of overcorrecting. It’s more frustrating to double-back than to make uniform progress from one side. That doesn’t mean avoiding overshooting is efficient.
1a3orn made what I thought was a good reply in a DM. My digestion/riff would be “You’re implicitly assuming that planning will be especially cheap for an LLM agent compared to humans. But even medium term planning / resilience to plans going awry is a particular weakness for LLM agents. The first, weakest agents that could pose a risk to humanity might be differentially bad at planning, find it differentially expensive to make good plans, do differentially less planning.”
What would a concrete AI takeover plan look like?
You can smell a chess bot by how quickly they change plans[1]. Human players act like they have a couple of attack strategies in mind and stick with them. Chess bots change tacks constantly, one move looking like they’re moving towards this goal and next move switching to something totally different.
I’m guessing this it how it would be with real-world takeover. Human need a simple grand strategy they can coordinate around (“An amphibious invasion of northern France”). AIs, even very weak ones[2], have far more working memory and excel at breadth. Their plan could be 1000 plans at once.
Are you familiar the legal phrases “in evidence” and “not in evidence”, as in “Objection! Question assume facts not in evidence!” or “Your honor, I move Exhibit A be admitted in evidence” or “You, the jury, may only consider the the testimony that has been admitted in evidence”? When you object to say something is “not in evidence”, you’re not saying evidence for it doesn’t exist—you’re saying “You just started relying on a premise we haven’t agreed to yet”. That term of art might be confusing and a reason to avoid saying “not in evidence”, but I didn’t get the sense the article understood that sense of the phrase.
I am not saying this as a political insider. I’m saying “consider other hypotheses” and “avoid the base rate fallacy”. Here, let me generate another hypothesis:
“It’s been a decades long truism: NSA is drowning in data, but can’t to turn it into intelligence. LLMs are the magic solution NSA’s dreamt for for decades. But the only licensed classified LLM provider is stymieing progress because of the inclusion of domestic surveillance data. NSA has recently gained clout at the Pentagon: the success of Maduro, and Iran’s top 40 military leaders killed the first day of the war, were only possible because of NSA and cell phones. NSA’s newfound opportunities to lobby the Pentagon about this roadblock at the highest level due close collaborations for recent military actions is what lit this fire.”
There you go. Somewhat plausible sounding. I think you could generate scores of other somewhat plausible sounding hypotheses like this. Therefore, your prior for any particular somewhat plausible hypothesis should be small.
It was +35 when I made the comment
I think this is conspiratorial and uncalibrated[1]
Your prior that a given DoD action is secretly due to the work of Sam Altman, or Palantir, or an Oracle lobbyist, or a Raytheon lobbyist, or Microsoft, or Israel, or one particular anti-woke DoD compliance lawyer with a crusade, or one particular DoW office politics tiff, or anyone else, should be low
The cause célèbre is not that the DoD canceled its contract with Anthropic and switched to its rival OpenAI, with whom it had ongoing negotiations. The cause célèbre is that the DoD did not just switch to a rival defense contractor. The DoD insisted that Anthropic remain a contractor. Not “F you, we’re going to Sam Altman”, but “F you, we’re not hiring someone else, we tell you what to do you don’t tell use what to do, do this work or we will nuke you”.
Your prior that in a meeting between someone like Pete Hegseth and Dario Amodei, with Hegseth repeatedly yelling orders and Amodei not standing down, that Hegseth might come away pissed and with (or reporting) misconceptions about the Dario’s insubordinate position, should be high.
(Independently, the fact that OpenAI siphoned donations through Greg Brockman and not Sam Altman is a good point I had not considered. I’d ascribe that to optics, internal and external, rather than Brockman being a future fall guy)
And LessWrong should do better than bandwagon on “Bad thing was secretly caused by outgroup leader”. I’m not saying this was a bad comment to proffer this hypothesis -- I’m saying there should not have been so much agreement to it. EDIT agreement was at +35 or +45 when I wrote this comment
I think you like riffing, so I’ll riff.
Verbal communication is a game, in the game theoretic sense. In a game with your enemy, like zero-sum heads-up poker, you’re trying minimize how much value you communicate with your words, in the sense that you when you say “call”, you want your opponent to be indifferent between their replies (so they have no best option to choose from, no economic profit available). You want to “maximize the entropy” of their subjective state after you communicate, or minimize the the information[1] communicated.
Now what if the the other player has a compute advantage? If you’re playing a game with the devil, or an exploitative chess engine, they’re going to try to hack you. Your best strategy isn’t necessarily ignoring them, but it’s a little tricker. It’s something like...instead of using your raw value function to chose your actions, come up with an ensemble of value functions you think are plausible for an entity with greater compute, figure out your best action in each of those guess-of-greater-compute-value-functions, and then marginalize out the guess-of-greater-compute-value-functions and choose the best action that way?[2] Which may also be how you “maximize” the value a friend with more compute can help you?
I think the type of the thing you’re minimizing is more like “value of information” in utils rather than raw quantity of information in bits?
This is related to why you can bluff in chess. You make a “scary” looking move that’s actually bad, but the other player may still have to play like you see something they don’t.
There is a real, empirical, phenomenon in chemistry. Some molecules contain carbon chains, and some molcules don’t. Usually experiments starting with carbon-chain molecules end with carbon-chain molecules, and experiments without carbon-chain molecules end up without carbon-chain molecules. But not always. It’s an almost-conservation law.
I feel like you’re telling the 19th century chemist “You speak as though you’re trying to discover what the word “organic” means. Words don’t work like that. The meaning of a word is what it is generally used to mean within some speech community [...].”
The chemist is trying to discover what the word ‘organic compound’ means. It’s not vacuous. They need to discover that molecules exist, that molecules are made of atoms, that those atoms are conserved, and that the “organic” vs “inorganic” distinction they’ve been puzzling over is about carbon atoms. The chemists are not arguing over which known concept the word “organic” refers to. They using to words “organic” and “inorganic” to point to natural concepts they don’t yet understand and ask what is the natural shape of those concepts.
EDIT I think it’s fine to say “Agency is a nebulous word that might mean many things. When you’re writing with confusion about agency, please taboo the word ‘agency’ so we understand what you’re pointing to better”. But I would put that on the word ‘consciousness’ much, much more than ‘agency’.
So, what do the people who write so much about their puzzlement over “agency” mean by the word[...]? And given what they mean, what is the real Problem of Agency that they want to solve[...]?
If they could define what they meant by the word, they wouldn’t be puzzled! You can’t ask a natural philosopher from 1820 “what do you mean by ‘vitalism’ and ‘organic’?” and expect them to say “Actually, what I meant was molecules containing chains of carbon atoms”.
Agree with the general point but:
Well, no, evaluating the perplexity of an offline transcript doesn’t look the same as taking real actions. One time I found myself at school after class when one of my molars came loose and literally fell out of my mouth. I started frantically trying to figure out how to set up an emergency dental appointment. But then I realized “Wait, why am I at school? I graduated from school years ago. Oh! Duh! I’m dreaming!”. I decided to shrug off my teeth falling out, despite not being the sort of person who would realistically shrug off an emergency medical situation.