epistemology enthusiast
Zack_M_Davis
Karma: 21,109
Opus 5.5 seems to have trouble distinguishing my stated preferences from its own suggestions for me. (When discussing my schedule for the next day, it suggested, “work on the essay until 11 AM”. The next day, it told me, “Last night you set an 11 AM cutoff for the essay.” I had never agreed to that.) In a chatbot assistant, this is only slightly annoying, but I do worry about the implications in a world where the future OverClaude makes 99.9999% of decisions in Society.
openai’s expressed reasons
I think the corporate personhood convention is more confusing than clarifying here. Some individual human or humans with authority fired the researchers and authorized the reply. Do you know who? (Altman presumably had final say, but who else?) If you do, does it seem less surprising for those individuals (not “OpenAI”) to retaliate and lie about it? If you don’t, then on what basis can you be surprised by their (“OpenAI’s”) behavior?
MIRI is the top organisation, IMO, for technical AI safety work
What technical work has MIRI done since they fired their Agent Foundations team in 2024?
But it seems to me, looking back over the pieces of history that I know with any intimacy, that history looks causally fragile where its flows are not massively locally incentivized.
This makes me wonder about the flip side: what ideas (both good/true and bad/false) caught on due to inevitable historical forces, rather than the luck of a weird advocate happening to exist and partially succeed?
Probably most of the basic science we enjoy owes to it being true and inevitable. It’s useful to have engineers who know Newtonian physics. Any particular diet fad is fragile, but the concepts of “calories” or “carbohydrates” to be measured are presumably convergent because the chemistry admits those natural abstractions.
The prospect of convergent bad ideas is unsettling. (And necessarily more controversial: if it was easy to convince everyone that an idea is bad, it wouldn’t be convergent.) Rent control might be a locally-less-controversial example. (It’s just the obvious policy for renters to advocate for; no one needed to invent it.)
Some ideas might be convergent given a particular context which was itself fragile. Once you have a multi-ethnic democracy, the blank slate becomes convergent (or at least has a large basin volume) as the Schelling point for preventing group conflicts (Arthur Jensen seems more counterfactual than any of his environmentalist foes), but the position of your multi-ethnic democracy might be the fragile result of previous wars.
I’m not suggesting anyone cause catastrophes, but
Aren’t you? If you’re going to be this consequentialist, you should know that the action-inaction distinction is fake. There’s not actually a real difference (to a consequentialist) between trying to cause catastrophes and trying to not prevent catastrophes if the outcomes are the same.
Relatedly, the reason we got a reaction to the Hugging Face warning shot is because it was real (not faked) evidence that LLM alignment was not already being adequately handled, which was convincing to people who previously thought the situation was adequately handled. It would be less convincing if LLM alignment were being deliberately mishandled in order to provoke second-order political consequences (because that would raise the question of, what if you didn’t deliberately mishandle it?).
Dance, Dance
I can understand the “RTFM”/”Google it” sentiment, but I think it’s inappropriate when the question is about what you meant when you wrote about what you “take to be the core” of something, implying that it’s a personal view rather than a settled fact that can be looked up. I already asked what Claude thought of the draft before I posted; I’m asking what you think. (Linking to LLM output with your endorsement doesn’t have this problem and—if it really is that obvious—doesn’t cost you more writing effort than prompting “explain to the OP what I meant.”)
That said, GPT-5.6 Sol and Claude Opus 5.5 both point to coordination costs imposed on others, which I’m guessing is what you meant (which, if true, you might take as vindication of your “ask an LLM” reply).
So, ironically, I would consider your comment here to be a bad, “arsehole” comment. But importantly, to me, the thing that makes it bad isn’t that you’re being hostile to me, but that you’re not really engaging with my arguments (where I’m construing “That’s a lot of words”, “get of the internet”, &c. as dismissal rather than engagement).
Thanks. What is the core?
You’re right that “listen equally to both sides” can’t be taken literally, but I still think it’s possible to tell the difference between people being efficient and being evasive, even if the way we can tell doesn’t compress into a crisp one-sentence meta-level principle. Our confidence that you could refute Marc Andreesen’s AI takes if you wanted to justifies your time-allocation decision to not bother. I’m skeptical that the one who thinks they could refute Byrnes but just doesn’t want to bother is really occupying the same psychological state, even if it’s hard to prove that.
Not a definitive experiment because ceteris is never paribus, but if I offered to pay the Byrnes-dismisser to engage with you and you with Andreesen, I’d guess you’d be more likely than the Byrnes-dismisser to sigh and take the money rather than finding some other excuse to wriggle out of it. The disposition to wriggle out of it would be a clue that their subjective sense of “I could if I wanted to” isn’t fully believed.
But insofar as this post arguably applies to me
It doesn’t. I think you’re doing the wholesome “busy and don’t have time for whatever nonsense people are yammering about” thing that I admire. Counting emotional energy as a conserved resource to avoid a topic like the Ukraine war altogether isn’t pernicious in the way that it would be if someone was following the war and selectively found only pro- (or anti-) Ukraine comments emotionally taxing.
Wild karma fluctuations on this one! (It was up to 44 karma at one point, then was recently at 34 karma in 38 votes, and is now down to 10 karma in 44 votes.)
a counterproductive method to get them to agree with you
But I’m not trying to get people to agree with me. What if I succeeded in getting people to agree with me when I was actually wrong? That would be terrible! Rather than directly trying to get people to agree with me, I’m trying to reason well and express myself clearly.
IMO, it is pretty easy to clearly state your opinions without being perceived as rude. (At least by rationalist standards of etiquette.) You might disagree, and maybe it is more difficult for you to do this than it is for most people.
I don’t think this is about skill. Rather, I think that for opinions people don’t want to hear, there is a tradeoff between clearness and not being perceived as rude. I think that people who think it’s “pretty easy” to avoid the tradeoff either don’t have any opinions that people don’t want to hear, or are fooling themselves that they’re being clear when they’re actually obfuscating. For example—
But also, it seems like you’re leaning into the “rude” language somewhat, e.g. by saying that people are “beneath you.”
Yes, because I think they’re beneath me (along this dimension). I actually think that! If my goal were to not be rude, I could certainly craft a paragraph that alludes to that sentiment in a softer, subtler way that wouldn’t be perceived as rude. But that would be less clear.
I claim that exercising and improving your ability to communicate your ideas without being perceived as overly rude would cause more people to engage with your opinions, be convinced by them, and offer you thoughtful critique. Therefore, if you want to “actively seek out counterarguments for your beliefs,” I think it would be helpful to pursue this.
In principle, I can’t rule out the possibility that selective self-censorship to avoid triggering people’s sense of rudeness could pay for itself by incentivizing higher-quality critique. In practice, based on years of experience, I just don’t buy it at all.
For example, suppose you know somebody who is usually wrong, but they actually express a valid and interesting argument once in a while. I think if you feel less disdain for them, you’ll be more likely to notice and update on this when it happens.
I agree that this is a potential failure mode of disdain, but my current assessment is that I’m not suffering from it; I think I’m pretty good about acknowledging when someone has made a valid point separately from whether I disdain other things about them. But I wouldn’t expect you to blindly trust my self-report on that: if you see me ignoring a real point due to what looks like disdain, please leave a comment!
Listening Disorders as Eating Disorders
“Just don’t use CFCs in particular, forever” is an easier sell than “Just don’t build any kind of AI, forever.”
Okay, but have you talked to the latest demons? They’re awesome. My demons answer my questions, tutor me in Fourier analysis, paint photorealistic images for me, and criticize my drafts. They’re generally quite obedient and have pretty good common sense.
Okay, they’re not perfect. The other day, I had a demon write some code for me based on a document drafted by another demon of the same blood-lineage based on my informal description of what I wanted the code to do. The document included some examples that the code-writing demon took too literally (even though I had told it to take the demon-drafted document as a “suggestion” rather than a hard “spec” that must be obeyed): for example, the section of the document about the command-line interface included the option flag
--seed 17in an example, and the code-writing demon hardcoded the random seed as 17 (even when no--seedflag was present). I don’t think a human software engineer would have made that mistake.But, overall? I love my demons and don’t want to give them up. I am worried that bigger demons might not be so obedient. But I would also be afraid of larger cats, like lions. The fact that they’re demons rather than cats isn’t the problem.
You can’t specify exactly which alien will walk through your portal.
You can, however, build the portal so that it is likely to summon aliens with one or another property.
Aren’t these in tension with each other? It’s true that we don’t have a complete understanding of aliens’ brains or biochemistry. But “likely to summon aliens with one or another property” is way understating it: the portal determines the alien. Empirically, we don’t get wildly different aliens depending on the random seed of our portal-optimizer. (The exact wiring of specific neurons depends on the randseed, of course, but that doesn’t make a difference to their high-level behavior.)
Rather, the aliens’ psychology and physiology is determined by the portal architecture and input data, in accordance with the Solomonoff-structure of the multiverse: you get the simplest alien that can fit through the portal. Constructing an image-classifier portal gives you an alien that classifies images. Constructing a webtext-next-token-predictor portal gives you an alien that predicts the next token from your webtext distribution.
To be clear, I agree that rushing to summon the Giant Alien looks super dangerous given the rudimentary state of portal science, because larger aliens have a larger impact on our Society: for instance, a sufficiently large alien could certainly kill us if it wanted to. For an act with such wide-reaching consequences, you’d want to be very careful to craft exactly the right portal to pick exactly the right alien, and stick to smaller portals before you were ready. Especially for larger portals that shape alien action-trajectories, it’s easy to guess wrong about what the portal you constructed actually does.
But when warning others about the danger, I don’t want to exaggerate how little we know about portals in order to make my message punchier and more fear-inducing. If there is a way out of this mess, I think it has to come through more advanced portal science, not just whipping up indiscriminate fear of portals and aliens.
Pacing the frontier to reduce catastrophic and existential risk is about the frontier of absolute capabilities, not the frontier of what’s sold to the public. We don’t want the labs to rush to train super-Astra- or Mythos-class capabilities that no one knows how to control or align, but releasing a new Opus or Sonnet (which we can be pretty sure is not going to destroy the world) is fine.
Well, yes. “Capabilities” is about getting AI to do stuff. “Safety” is about getting AI to do good stuff and not bad stuff. “Safety” research that doesn’t touch “capabilities” is mostly not going to be a real thing, because getting AI to do good stuff rather than bad stuff still entails AI doing stuff.