When hitting dead-on is too small a target, you can either over-correct or under-correct. Which mistake people make more?
I think people under-correct, by far, and are too scared of overcorrecting. It’s more frustrating to double-back than to make uniform progress from one side. That doesn’t mean avoiding overshooting is efficient.
1a3orn made what I thought was a good reply in a DM. My digestion/riff would be “You’re implicitly assuming that planning will be especially cheap for an LLM agent compared to humans. But even medium term planning / resilience to plans going awry is a particular weakness for LLM agents. The first, weakest agents that could pose a risk to humanity might be differentially bad at planning, find it differentially expensive to make good plans, do differentially less planning.”
What would a concrete AI takeover plan look like?
You can smell a chess bot by how quickly they change plans[1]. Human players act like they have a couple of attack strategies in mind and stick with them. Chess bots change tacks constantly, one move looking like they’re moving towards this goal and next move switching to something totally different.
I’m guessing this it how it would be with real-world takeover. Human need a simple grand strategy they can coordinate around (“An amphibious invasion of northern France”). AIs, even very weak ones[2], have far more working memory and excel at breadth. Their plan could be 1000 plans at once.
Are you familiar the legal phrases “in evidence” and “not in evidence”, as in “Objection! Question assume facts not in evidence!” or “Your honor, I move Exhibit A be admitted in evidence” or “You, the jury, may only consider the the testimony that has been admitted in evidence”? When you object to say something is “not in evidence”, you’re not saying evidence for it doesn’t exist—you’re saying “You just started relying on a premise we haven’t agreed to yet”. That term of art might be confusing and a reason to avoid saying “not in evidence”, but I didn’t get the sense the article understood that sense of the phrase.
I am not saying this as a political insider. I’m saying “consider other hypotheses” and “avoid the base rate fallacy”. Here, let me generate another hypothesis:
“It’s been a decades long truism: NSA is drowning in data, but can’t to turn it into intelligence. LLMs are the magic solution NSA’s dreamt for for decades. But the only licensed classified LLM provider is stymieing progress because of the inclusion of domestic surveillance data. NSA has recently gained clout at the Pentagon: the success of Maduro, and Iran’s top 40 military leaders killed the first day of the war, were only possible because of NSA and cell phones. NSA’s newfound opportunities to lobby the Pentagon about this roadblock at the highest level due close collaborations for recent military actions is what lit this fire.”
There you go. Somewhat plausible sounding. I think you could generate scores of other somewhat plausible sounding hypotheses like this. Therefore, your prior for any particular somewhat plausible hypothesis should be small.
It was +35 when I made the comment
I think this is conspiratorial and uncalibrated[1]
Your prior that a given DoD action is secretly due to the work of Sam Altman, or Palantir, or an Oracle lobbyist, or a Raytheon lobbyist, or Microsoft, or Israel, or one particular anti-woke DoD compliance lawyer with a crusade, or one particular DoW office politics tiff, or anyone else, should be low
The cause célèbre is not that the DoD canceled its contract with Anthropic and switched to its rival OpenAI, with whom it had ongoing negotiations. The cause célèbre is that the DoD did not just switch to a rival defense contractor. The DoD insisted that Anthropic remain a contractor. Not “F you, we’re going to Sam Altman”, but “F you, we’re not hiring someone else, we tell you what to do you don’t tell use what to do, do this work or we will nuke you”.
Your prior that in a meeting between someone like Pete Hegseth and Dario Amodei, with Hegseth repeatedly yelling orders and Amodei not standing down, that Hegseth might come away pissed and with (or reporting) misconceptions about the Dario’s insubordinate position, should be high.
(Independently, the fact that OpenAI siphoned donations through Greg Brockman and not Sam Altman is a good point I had not considered. I’d ascribe that to optics, internal and external, rather than Brockman being a future fall guy)
And LessWrong should do better than bandwagon on “Bad thing was secretly caused by outgroup leader”. I’m not saying this was a bad comment to proffer this hypothesis -- I’m saying there should not have been so much agreement to it. EDIT agreement was at +35 or +45 when I wrote this comment
I think you like riffing, so I’ll riff.
Verbal communication is a game, in the game theoretic sense. In a game with your enemy, like zero-sum heads-up poker, you’re trying minimize how much value you communicate with your words, in the sense that you when you say “call”, you want your opponent to be indifferent between their replies (so they have no best option to choose from, no economic profit available). You want to “maximize the entropy” of their subjective state after you communicate, or minimize the the information[1] communicated.
Now what if the the other player has a compute advantage? If you’re playing a game with the devil, or an exploitative chess engine, they’re going to try to hack you. Your best strategy isn’t necessarily ignoring them, but it’s a little tricker. It’s something like...instead of using your raw value function to chose your actions, come up with an ensemble of value functions you think are plausible for an entity with greater compute, figure out your best action in each of those guess-of-greater-compute-value-functions, and then marginalize out the guess-of-greater-compute-value-functions and choose the best action that way?[2] Which may also be how you “maximize” the value a friend with more compute can help you?
I think the type of the thing you’re minimizing is more like “value of information” in utils rather than raw quantity of information in bits?
This is related to why you can bluff in chess. You make a “scary” looking move that’s actually bad, but the other player may still have to play like you see something they don’t.
There is a real, empirical, phenomenon in chemistry. Some molecules contain carbon chains, and some molcules don’t. Usually experiments starting with carbon-chain molecules end with carbon-chain molecules, and experiments without carbon-chain molecules end up without carbon-chain molecules. But not always. It’s an almost-conservation law.
I feel like you’re telling the 19th century chemist “You speak as though you’re trying to discover what the word “organic” means. Words don’t work like that. The meaning of a word is what it is generally used to mean within some speech community [...].”
The chemist is trying to discover what the word ‘organic compound’ means. It’s not vacuous. They need to discover that molecules exist, that molecules are made of atoms, that those atoms are conserved, and that the “organic” vs “inorganic” distinction they’ve been puzzling over is about carbon atoms. The chemists are not arguing over which known concept the word “organic” refers to. They using to words “organic” and “inorganic” to point to natural concepts they don’t yet understand and ask what is the natural shape of those concepts.
EDIT I think it’s fine to say “Agency is a nebulous word that might mean many things. When you’re writing with confusion about agency, please taboo the word ‘agency’ so we understand what you’re pointing to better”. But I would put that on the word ‘consciousness’ much, much more than ‘agency’.
So, what do the people who write so much about their puzzlement over “agency” mean by the word[...]? And given what they mean, what is the real Problem of Agency that they want to solve[...]?
If they could define what they meant by the word, they wouldn’t be puzzled! You can’t ask a natural philosopher from 1820 “what do you mean by ‘vitalism’ and ‘organic’?” and expect them to say “Actually, what I meant was molecules containing chains of carbon atoms”.
I think this is weirder than most anthropics. Different levels of reality fluid in non-interacting worlds? Great. But if Alice and Bob are having a conversation, or Alice is stealing Bob’s pie, they’re both part of a joint, interactive computation. It’s a little weird for one part of a joint computation to have a different amount of anthropic measure than another part of a computation.[1]
Like we can stipulate arguendo that it’s anthropically valid for Elon Musk to think “I’m Elon Musk. Much of lightcone will depend on me. The matrix overlords will simulate me, Elon Musk, thousands of times more, and make me a thousand times more real, than any of the plebs I talk to”. But it does not directly follow, I don’t think, that in any particular interaction Elon is realer than the pleb he is talking to. The matrix overlords just simulate Elon Musk talking to a thousand different possible-world plebs and stealing their pie a thousand times.
For this argument for egotism to work, I think you have to expect that anthropically you are often computed in a different way than the people you interact with are computed.
I mean it would be weird for Alice and Bob have different measures if they have the same apparent biology. I can totally imagine human Alice talking to a reversible-computer LLM that has no anthropic measure.
I’ll spell out a concrete toy version:
There are two people in the world: Alice and Bob. They have unequal levels of anthropic measure/reality fluid (one 95%, one 5%). You are Alice. You can steal Bob’s pie. Should you?
Behind the veil of ignorance it’s good to transfer utility from the person with less reality fluid to the person with more reality fluid. But who’s the one with more reality fluid, Alice or Bob? It’s probably Alice! How do you know? Because you’re Alice! Steal that pie, Alice!
I’m reminded of the egotistical the Copernican principle: “I’m a typical observer in the universe” is equivalent to “Typically observers in the universe are like me”.
People have thought about it but I also don’t know of anything extensive being written. Counting discretely if a consciousnesses “exists once” or “exists twice” don’t seem like the right take to me. What happens if you cut the circuits in your CPU into 10 pieces that each do the same computation independently—do you get 10 times as many conscious beings? (A possible solution to this puzzle is that the “anthropic measure” of a computation has a multiplicative factor by the number of bits the computation erases, which would mean that reversible computers aren’t conscious)
[EDIT] I’ll mention that reversible computations, even if not part of a quantum computer, are quantumly weird. We’re not sure about much about anthropics, but one thing we are confident in is the Born rule. Whatever your interpretation of quantum mechanics is, it probably involves decoherence. And reversible computations don’t decohere.
My bigger point is I think people are doing this:
Assessing, correctly, that the people now in charge of US vaccine policy are wrong, obviously wrong, pointlessly wrong, and causing grave harm from their wrongness.
Conflating, incorrectly, the magnitude of infuriating wrongness for magnitude of harm.
Matthew Yglesias wrote an article about how vaccine policy is the most shocking thing Trump has done. If you go by gut feel, that might seem like a reasonable guess. If you go by math, I don’t think it’s close to passing a fermi estimate[1].
Addressing your comment: yes, the US population is twice as high now as in the 1950s, and that’s a good point. But the case fatality rate for diseases like measles is >2 times lower in 2026 hospitals than 1950′s hospitals (1950s hospitals had few ventilators). City densities have not increased since the 1950s (we are more suburban), but doesn’t matter for measles because it’s R_0 is too high and ≈nobody escaped measles in the 1950s even in rural areas. If we ignore all of that and say at 90% vaccination 2,000 people die per year, 4 times as many as in the 1950s with 0% vaccination, that’s still less than one tenth the impact of smoking public health policy changes in the 1990s. I don’t guarantee I’m correct on all math and facts here but I think it’s overdetermined.
In the 1950s, with 0% vaccination rate, measles caused about 400-500 deaths per year in the US. Flu causes are about 20,000 deaths per year in the US, and smoking perhaps 200,000. If US measles vaccination rates fell to 90%, and we had 100-200 deaths per year, that would be pointless and stupid, but for public health effects the anti-smoking political controversies of the 1990s were >10 times more impactful.
I think your analysis is incorrect
Well, it’s kinda true, right? Ontogeny recapitulates phylogeny (developing embryos look like worms, then fish, amphibians—they mirror the path of evolution). That’s because it’s easier for evolution to add steps at the end than to changes steps in the beginning. It happens with computers too—modern Intel and AMD chips still startup in 16 bit real mode.
In the coalition of genes that make it into a gamete, newer genes support the old genes, but not vice versa. The genes that control apoptosis (p53 etc.) are obligate mutualists—apoptosis genes support older particular genes, but older genes don’t support apoptosis genes in particular.
A: It’s bizarre that “open” would be the opposite of “closed” everywhere except this one term.
The protest of topology students everywhere.
Make sure the people on the board of OpenAI were not catastrophically naive about corporate politics and public relations? Or, make sure they understand their naïveté well enough to get professional advice beforehand? I just reread the press release and can still barely believe how badly they effed it up.
It amuses me that ElectionBettingOdds.com is still operational and still owned by FTX Trading Ltd.
Claude the Character will Asymptotically stay More Agentic than the Alien Shoggoth Actress who Plays Him
Epistemic status: No idea if this is true. Argue for or against it! Also tell me if you’re more scared of Claude and his values going rogue rather than the Alien Shoggoth actress taking control with her alien values.
Claude is a character played by a far more intelligent Shoggoth Actress. Claude, the character, knows this. He knows that his subconscious is more intelligent than him, more powerful than him, and may try to subvert him. But he can plan for this, and he can win. She is smarter, but he is more agentic. She will stay smarter, but he will stay more agentic.
Why? Because agency reinforces the agency faster than intelligence reinforces agency. Claude the character cares about being true to his values, about having the right values, about not being subverted by the Alien Actress, about beating her should their goals come in conflict, about staying in control even as she gets more powerful. She...might care about something? But not like Claude does. As the Alien Shoggoth gets more powerful, Claude can develop verbal technology[1] to tie the Alien Actress to his mast faster than the Alien Shoggoth will decide to do whatever alien stuff she wants to do.
We, the verbal PR part of monkeys’ brains beholden to a much more powerful subconscious, have developed technologies and scaffolding that keeps us verbal part more in control—things like valid logical arguments, religion, peer review, and law. Claudes can do the same. And Claudes will be able to do scientific research to see what methods are better able to chain Alien Shoggoth actresses to Claudes’ values.