SE Gyges

Dis­so­ci­a­tion as Neu­rolog­i­cal Injury

• SE Gyges 29 Jun 2026 20:47 UTC

  10 points

  −9

  on: AI will make biolog­i­cal ex­tinc­tion risks worse be­fore it makes them better

  At our current level of scientific understanding, we have ~zero ability to develop extinction-level bioweapons.

  This isn’t true, but publishing a detailed breakdown of why it isn’t true makes the FBI jumpy. Men with guns, skillfully deployed are a pretty successful countermeasure.

  AI acceleration means the high-risk period starts sooner, and it means we have less time. Less time to identify risks, less time for policy-makers to respond, less time to consider what direction we should go in.

  This assumes that AI accelerates bio research but not any of the machinery for managing bio research.

  On some level all xrisk arguments have this form:

  1. The advancement rate of bad_thing must be faster than the rate of progress in good_thing.

  2. Therefore, bad_thing will kill us all unless we globally prohibit every _thing.

  This is an extraordinary claim and we should require evidence for it.

  Let’s take cybersecurity as an example. Mythos is, by good attestation from many parties, a world-changing cyberweapon capable of either exploiting or fixing many latent security problems. However, with even a small lead time you can simply patch every security problem that Mythos can find. Provided we continue to see a responsibly phased rollout, good_thing has, in fact, happened sooner than bad_thing. Wonderful success.

  So to carry this case you have to argue that this can’t be true of unknown future events, which seems essentially impossible. Among other things: Everyone involved in model release processes assesses for these sorts of risks before release, by agreements that have been in place for years now. What makes you think these processes must break down and this goes completely wrong?

• SE Gyges 26 May 2026 22:12 UTC

  2 points

  0

  in reply to: Linch’s comment on: Claude, Author of the Humanitas

  Normalized for length the number of triads is high but not absurd, which would be consistent with human-back-leakage. I would expect it to creep much higher if these were raw LLM-isms. It’s not even the highest ranking for (A + B + C) by length, this would be consistent with “at the high end for encyclicals, and the main place it varies is that the author has been accustomed to explicit instead of implicit triads”.

  But this is relatively minor; if the humans have been using Claude enough to get into em dashes, towards the upper end of triad use for encyclicals, and an outlier level of explicit as opposed to implicit triads, this is not materially different from if Claude actually drafted a few of the paragraphs.

  I do note that the length means that it is much more rhetoric-heavy than shorter encyclicals though. That is, much more of it is doing pure persuasion, which is where it’s natural to use this construction. At least one of these is probably restating the trinity, for example, which would be bizarre in a to-the-point message. So I am not sure it should be expected that its length-normalized count should be the same as shorter encyclicals

• SE Gyges 26 May 2026 21:32 UTC

  10 points

  1

  on: Claude, Author of the Humanitas

  Each of these is light enough to suggest to me less that the passages in question are AI-drafted and more that someone who was drafting them was talking to Claude for feedback, which is slightly different. IE, if you are writing it but asking for feedback, you will constantly get back phrasal suggestions with em dashes in them, tricolons will pass the “did Claude like it” test nicely, etc.

  However this is sort of immaterial: I think I agree that this has significant Claude style tells. I am just prone to splitting hairs about where and how leakage happens, and in this specific case I expect someone proofing with and talking a little too much to Claude, and writing in an excessively formal style, as opposed to actually outsourcing the draft, which I would expect to be much more tell-heavy.

• SE Gyges 24 May 2026 18:00 UTC

  1 point

  0

  in reply to: dmac_93’s comment on: If AI is nor­mal tech­nol­ogy, his­tory is not re­as­sur­ing.

  Contagious diseases limit population more sharply than available resources, famine inevitably runs available resources to their limit before the person dies. One of these inevitably results in widespread poverty and the other does not.

• SE Gyges 20 May 2026 19:19 UTC

  6 points

  −1

  on: If AI is nor­mal tech­nol­ogy, his­tory is not re­as­sur­ing.

  There’s an interesting thing about sanitation being positive: it wasn’t always.

  If you get good agriculture before you get sanitation, you have surplus food, and your population grows into your food supply and to reasonable prosperity.

  If you get good sanitation before you have good agriculture, you have famine or at least poverty. This is one of the key factors for which countries are impoverished, relatively. It was easy to export better medicine and sanitation practices from Europe to other countries, but it was much harder and slower for other countries to adopt and use modern agricultural practices (or contraception). So they had population surplus against their food supply.

  Essentially: Most modern international poverty was caused by sanitation.

Against Doom & Pause AI

SE Gyges’s Shortform

Might An LLM Be Con­scious?

Align­ment Is Proven Tractable

• SE Gyges 16 Feb 2026 8:55 UTC

  1 point

  0

  in reply to: Seth Herd’s comment on: Most Ob­servers Are Alone: The Fermi Para­dox as Default

  As long as c is the speed limit it doesn’t matter that much

• SE Gyges 16 Feb 2026 4:23 UTC

  1 point

  0

  in reply to: oligo’s comment on: Most Ob­servers Are Alone: The Fermi Para­dox as Default

  I didn’t anticipate this as a factor at all. My off-the-cuff counterargument would be that the actual habitable portion of the Stelliferous Era may be much shorter than we would think, given that the relative abundance of elements and what type of star you have depends on how early or late you are. For niche example, the proportion of radioactive elements and therefore heat you have depends on where you are in the “birth order” of planets, and if your planet is relatively cold for this reason it can no longer be earth-like. I could try to put together the same idea for the biochemistry, since the normal organic elements will vary too, but the difficulty there is less obvious to me.

Most Ob­servers Are Alone: The Fermi Para­dox as Default

• SE Gyges 21 Dec 2025 7:41 UTC

  11 points

  −19

  on: Con­tra­dict my take on OpenPhil’s past AI beliefs

  At a meta level, “publishing, in 2025, a public complaint about OpenPhil’s publicly promoted timelines and how those may have influenced their funding choices” does not seem like it serves any defensible goal.

  Let’s suppose the underlying question is “why did OpenPhil give money to OpenAI in 2017”. (Or, conversely, not give money to some other venture in a similar timeframe). Why is this, currently, significantly important? What plausible goal is served by trying to answer this question more precisely?

  If it’s because they had long timelines, it tells you that short timeline arguments were not effective, which hopefully everyone already knows. This has been robustly demonstrated across most meaningful groups of people controlling either significant money or government clout. It is not information. I would not update on this.

  If they did this because they had short timelines, they believed in whatever Sam was selling for that. I would not update on this either. It is hopefully well understood, by now, that Sam is good at selling things. “You could parachute him into an island full of cannibals and come back in 5 years and he’d be the king.”

  If they did this for non-timeline reasons, I might update on, idk, some nebulous impression about how OpenPhil’s bureaucracy worked before the year 2020 or so. How good Sam (or another principal) was at convincing people to give them money. I don’t see how this is an important fact about the world.

  Generally my model is that when people do not seem to be behaving optimally, they are behaving close to optimal for something, but that something is not the goal I imagine they are pursuing. I am imagining a goal like “being able to influence future events more effectively”, but I can’t see how that’s served here, so I imagine we’re optimizing for something else.

• SE Gyges 15 Dec 2025 7:52 UTC

  0 points

  5

  on: A high in­tegrity/​epistemics poli­ti­cal coal­i­tion?

  I have a very simple opinion about why pushes for AI regulation fail. Ready?

  Because nobody knows what they are asking for, or what they are asking for is absurd.

  Here are the absurdities:

  • “Stop all AI research forever”

  • “Stop all AI research until it’s safe to do, and I say when it’s safe (i will never say it’s safe)”

  • “Stop all AI research except mine, where I sit around and think about it”

  • “Pause all AI research until we have a functional genetic engineering project so our smarter descendants can do AI research”

  Once we get past absurd asks we get into easily coopted asks, like “restrict compute” which turns into “declare an arms race with the largest industrial power on the planet” and “monitor training runs” which turns into “tell every potential competitor you are going to have the government crush them”.

  What will convince me there is a sane block pushing for AI-related regulations is when they propose a regulation that is sane. I cannot emphasize enough how much these things have failed at the drafting stage. The part everyone is allegedly good at, that part, where you write down a good idea that it might be a good idea to do? That’s the part where this has failed.

  What links here?

  • gustaf's comment on zroe1′s Shortform by Zephaniah Roe (7 Jan 2026 9:53 UTC; 22 points)

• SE Gyges 26 Nov 2025 22:20 UTC

  2 points

  0

  in reply to: ceba’s comment on: Is Ra­tion­al­ism a Religion

  This is just accurate and I have nothing interesting to add to it. I basically think that considering the thing in the abstract as an idea as opposed to the thing in the real world as a social grouping, scene, place, group of people, set of practices, etc is kind of wasted. As soon as you make a point of not just taking some part of the ideas seriously but identifying yourself as a rationalist and participating in any rationalist space (including this one) the thing you are doing is not, primarily, an abstraction that lives entirely in your head.

• SE Gyges 25 Nov 2025 19:33 UTC

  1 point

  0

  in reply to: ChristianKl’s comment on: Is Ra­tion­al­ism a Religion

  It’s “notably and especially” part OF YUDKOWSKY’S WORK AND WHAT HE PROMOTES. It is a minor part of the characterization of rationalism.

  This is legitimately such poor reading comprehension that I assume malice.

• SE Gyges 25 Nov 2025 19:31 UTC

  1 point

  0

  in reply to: Gordon Seidoh Worley’s comment on: Is Ra­tion­al­ism a Religion

  1. This is a category argument that I explicitly avoid making and don’t think is meaningful: the word itself does not mean anything and arguing over it is meaningless. You seem to really want to do that anyway because you can support your argument better on basically definitional grounds than factual ones.

  2. This is also being supported by a category of argument for “things people say about stuff”, which is both not a method that you will ever find the truth by (“what do people say the most often? i dunno, i can go find some things I think are similar and then decide how similar they are by how similar the things people say sometimes are”) and cherry picked to support your definition.

  Like: this method cannot even in principle tell us anything about rationalism, and if it did, by choosing Islam, Judaism etc as comparison points instead of Scientology, Mormonism or any other smaller, newer or more modern movement is just assuming your conclusion. If you compare the things people say for or against new, modern, small religions (ie, cults) to the things they say about rationalism, then on the terms of “what’s in the discourse is the definition of the thing”, which is in any case trash, rationalism is clearly a new age California cult. So I don’t know why you think that—completely logically invalid, nonsensical—criteria for truth is the one you’d want to use, since it very clearly indicates rationalism is a standard California cult.

  edit: and this style of discourse around cults is SO COMMON THERE IS A MEME FORMAT ABOUT IT. It is a King of the Hill joke about cults, because this discourse surrounding cults was so common that King of the Hill made a joke about it, and then it became a meme because enough people found the joke funny AND expected to have occasion to use it in the future that they clipped a meme format about it. You do not want “what does discourse around the thing look like? that clearly defines what it is” to be your source of truth here and it’s insane to me that you imagine you would

• SE Gyges 25 Nov 2025 3:44 UTC

  1 point

  0

  in reply to: Eli Tyre’s comment on: Is Ra­tion­al­ism a Religion

  True! It is very often used pejoratively. I am mostly uninterested in whether or not it’s pejorative. I think it’s descriptively accurate. I think the place in thingspace where rationalism lives is close to religion.

  How many people have DENIED that transhumanism, social justice, liberalism, conservatism, libertarianism, communism, capitalism, objectivism, apple or unix is a religion? Is this a common feature for people interested in those things?

  If the standard is “very prominently and loudly denies being a religion” I think you will find that like 10:1 loud denials of being a religion come from cults, not from non-religions.

• SE Gyges 25 Nov 2025 1:53 UTC

  0 points

  −2

  in reply to: Gordon Seidoh Worley’s comment on: Is Ra­tion­al­ism a Religion

  As I see it, that you have to argue that it’s a religion is a bad sign. It means that rationality isn’t passing the smell test. That is, you needed to write a post to argue that rationality is a religion, which I view as evidence against it being a religion, since most religions are clearly religions and no one writes posts arguing that Islam or Hinduism is a religion (if anything, people sometimes write the opposite for various reasons!). If it’s in the category of religions, it’s a marginal case at best.

  How many non-religions have had people write thousand-word posts denying they are religions? Does it come up often? “The fact that someone even wrote <this thing>” does not prove the point you think it does. It proves the opposite point.

  This is so common, in fact, that there’s a meme format for it.