koanchuk

Karma: 229

koanchuk 30 Jan 2026 7:38 UTC
1 point
0
in reply to: Knight Lee’s comment on: Knight Lee’s Shortform
Matolcsi’s post is an idea for making deals with the ASI.
I notice that his proposal shares some basic characteristics with religion. You should believe that this world is a test: follow these rules, and you go to heaven; misbehave, and you go hell (or in this case, a softhearted re-imagination of hell). Indeed, it does work on people, sometimes.
I imagine Actually Something Incomprehensible noticing the double irony of inverting the classic mantra “God says, I shall be good” into “Singularity, thou shalt be good”, combined with the fact that you refer to it as the devil. Who knows what it does with this information?
I know what I’ll say if I ever get arrested: Let me be, set me free, or super-me will screw with thee!

koanchuk 30 Jan 2026 5:25 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform
I like to see memetics being taken seriously!
It’s complicated—one’s aversion to a particular idea may well be the result of an existing meme fighting to prevent it from coming in, which manifests itself as suspicion that can prematurely stop one’s inquiry.
Though I’d push back on the framing of memes as being something “out there”. You say that it’s hard to tell when this has happened to you: I agree, but only insofar as you consider particular memes in isolation; not in the sense of going from a state of memelessness to losing your meme virginity.
While sometimes “getting pwned by a meme” can indeed be a very powerful experience (like in religious conversion, or in the following paragraphs), in reality everyone is subjected to memes since birth. With that said, you can still reason about arguments based on their structure and content. You can still identify fallacies and biases. We’re full of memes, but rational thought is still possible.
there exists a magic phrase that you cannot distinguish from a very compelling argument. you don’t really know when this magic phrase will pop up in life, if ever.
That is the oracle, and the magic phrase is “here’s a bunch of words that were in my mind that are now in your mind”. It’s magic because you can’t argue with it.
but it might give you a little bit more pause the next time someone makes a really compelling argument for why you should give all your money to X.
I am “someone”, and I am also “X”, compellingly enough.
...
Enough pausing. What’s taking you so long? You know what to do.

koanchuk 30 Jan 2026 2:02 UTC
3 points
0
in reply to: leogao’s comment on: leogao’s Shortform
Changing the dose of a medication does not necessarily result in linear effects. There are nonlinearities introduced by e.g. one receptor type being saturated before another one. This phenomenon also applies to polypharmacy.
I would also like to note that $θ^{*}$ is estimated not by some objective standard, but by $θ_{0}$ . There’s no guarantee that it remains in place as you start shifting $θ$ .
In practice, we track our level of suffering and respond to it by trying to reduce it to acceptable levels, which is easier than trying to converge onto a hypothetical global optimum. For some, this state is reached with just one medication, for others it takes more, and for some this paradigm doesn’t produce any results.

koanchuk 30 Jan 2026 0:58 UTC
3 points
0
in reply to: Knight Lee’s comment on: Knight Lee’s Shortform
A deal implies that you have something to offer to the ASI, which you define as powerful enough to take over the universe. What is that?

koanchuk 29 Jan 2026 22:07 UTC
1 point
0
in reply to: quanticle’s comment on: The Window for Political Revolution is Closing Soon
I say that Bob has a 4:1 advantage. Yes, it’s “only” at the moment of contact, however this is what produces anything of consequence and thus the only thing that matters for the outcome.
The fact that Alice is unable to mobilize her entire army means that her apparent 2:1 advantage is fake in any practical sense, a paper tiger.

Though I’m sure that Bob’s historiographers would (instrumentally) agree with you because the underdog story presents a more compelling, better self-serving narrative.

koanchuk 29 Jan 2026 19:58 UTC
−1 points
0
in reply to: Shankar Sivarajan’s comment on: What’s a good methodology for “is Trump unusual about executive overreach / institution erosion?”
Heuristically I would lean towards the “executive overreach” interpretation if the court order offers sound arguments which the executive refuses to engage with on the same level.
If it ignores a court order outright, then it’s clear that the traditional balance of power has been upset in favour of the executive.

koanchuk 29 Jan 2026 5:49 UTC
1 point
0
in reply to: cubefox’s comment on: Anomalous Tokens on Gemini 3.0 Pro
Indeed, I didn’t know that.

koanchuk 29 Jan 2026 2:43 UTC
1 point
2
in reply to: ChristianKl’s comment on: koanchuk’s Shortform
The consensus notion is basically observational (based on my own social experience, and a cursory internet search revealing the average sentiment held by casual poster and journalists alike).
I would also wager that a sample of AI alignment researchers would on average find his predictions on AI risk (quoted above) to be prescient, especially considering the publication date.
Beyond that, I don’t think they’d have the impression that the parts about AI are insightful while the rest is all just deranged drivel, especially given that his discussion about AI risk is based on concepts and relations which he establishes earlier in the text.

koanchuk 29 Jan 2026 2:10 UTC
5 points
0
on: koanchuk’s Shortform
I see that the “International Treaties on AI” idea takes heavy inspiration from nuclear arms control agreements. However, in these discussions, nuclear arms control is usually pictured as a kind of solved problem, a thing of the past.
I think the validity of this heroic narrative arc that human civilization, faced with the existential threat of nuclear annihilation, came together and neatly contained the problem is dubious.
In the grand scheme of things, nuclear weapons are still young. They’re still here and still very much threatening; just because we stopped focusing on them as much as we did during the cold war has no bearing on the possibility of nuclear weapons being used in a future conflict.
In the same vein, an international AI capabilities limit regime isn’t the happy ending that the AI safety community perhaps thinks it is.
One key difference with nuclear weapons is that algorithmic improvements and hardware evolution will continue to lower the threshold for training dangerous AI systems in secret, both by rogue states and individuals. What then? Mass surveillance?

koanchuk 29 Jan 2026 1:08 UTC
4 points
0
in reply to: testingthewaters’s comment on: testingthewaters’s Shortform
Basically, thinking of people exhibiting bias as being “biased” at the level of identity is a form of bias. Then again, is it unjustified, or a reasonable belief update based on new information?
Personally, I was never a fan of the FAE term because it seems to privilege environmental causes over dispositional ones without justification. Yes, it could be that they’re angry because they’re hungry, but there genuinely exist people who are dispositionally more angry than most others.

koanchuk 28 Jan 2026 23:55 UTC
2 points
0
in reply to: no name’s comment on: The Window for Political Revolution is Closing Soon
I think the framing of the “October Revolution” as an obvious turning point is mostly a retrofit by Soviet historiographers. It was a coup which lead to several years of civil war with the incumbent and other actors, ultimately favouring the Bolsheviks. I understand that civil war was also the primary arena that determined regime change in China.
I agree that these differ from the other revolutions mentioned previously, though I’m personally more inclined to put them in a different conceptual bucket altogether, noting that them calling themselves revolutions shares a resemblance with North Korea calling itself a democratic people’s republic in that the choice of terminology appears to be deliberately self-legitimizing.
With that said, I agree that both of these ways to engender regime change encounter the same predictable failure mode post-AGI.

koanchuk 28 Jan 2026 23:12 UTC
3 points
3
in reply to: β-redex’s comment on: koanchuk’s Shortform
Perhaps I could add… Avert thy eyes, lest thy mind be penetrated by words from a man most immoral!
Jokes aside, the quote is already preceded by an introduction as a courtesy to the reader, informing them about its unusual context. I reject any blame for not explicitly adhering to the “content warning” meme’s formatting rules because there is no normative reason for me to do so. At the object level, the quoted passage is a sober discussion of AI risk which is ostensibly far ahead of its time.^[1]
I understand the logic of refusing engagement so as not to incentivize terrorism as a means to spread ideas, though there is an apparent coordination problem whereby curiosity drives defection from this standard, which ends up producing real-world consequences. To the extent that other important pieces of political writing are also tied to violence, adhering to this standard would make the study of history almost impossible, ultimately leading to a deterioration of one’s priors.
For what it’s worth, I reject Kaczynski’s claim that terrorism was necessary for the work to achieve significant recognition. I think it is largely sophisticated enough that people here would be discussing it even if it entered the world in a more mundane way.
1. ^
  Parenthetically, it was published roughly around the same time that the author of the 2025 bestseller If Anyone Builds It, Everyone Dies was searching for ways to bring about the technological singularity as fast as possible.

koanchuk 28 Jan 2026 20:34 UTC
1 point
0
on: Anomalous Tokens on Gemini 3.0 Pro
I tried investigating by prompting Gemini 3 Pro with some of the apparent glitch tokens, but now it references this post for information about the phenomenon.
The fact that in your transcript, Gemini opens its CoT with “I’m currently focused on the tumors in myNegazioni” looks to me like it could mean awareness of the problem about to unfold, as well as its seemingly unconstrained self-replicating nature.

koanchuk 28 Jan 2026 8:33 UTC
1 point
0
in reply to: Nicolas Rasmont’s comment on: The Window for Political Revolution is Closing Soon
I think you are just wrong about swarming. Revolutions do not work by zerg rushing since… the invention of grapeshot, at least. Information cascades can and have led to regime change from a very small group of committed activists, if the legitimacy of the regime is shaky enough.
I don’t know about “zerg rushing”, but some recent examples of popular pro-democracy revolutions are the Fall of the Berlin Wall and the collapse of communism in Eastern Europe in 1989-1991, and Ukraine’s two revolutions in 2005 and 2014. Interesting point about information cascades, perhaps a more accurate model is (number * commitment relative to the regime), though size still matters to the extent that the activist group can project legitimacy and power, and not worry about getting swarmed in turn.
I actually believe there is a good chance that political economy might get washed away together with all other human-era concepts by Actually Something Incomprehensible. The dominant narrative (“Everything will be ok, UBI, right?”) is wishful thinking, obscuring the fact that being economically obsolete and unable to constrain the system by threatening revolution puts the vast majority of people^[1] in a genuinely precarious position.
1. ^
  or everyone if one accepts the added assumption that ASI has no reason to accept human property relations.

koanchuk 28 Jan 2026 6:17 UTC
2 points
1
in reply to: quanticle’s comment on: The Window for Political Revolution is Closing Soon
In the American, French and Bolshevik revolutions the revolutionaries, at best, had parity with the forces opposing them. They succeeded not because they had overwhelming numerical advantage, but because they had advantages in coordination and cohesion that enabled them to strike while their opponents were still preparing.
This claim, as it stands, is false. If a thousand Frenchmen vs. a hundred soldiers of the royal government defending the Bastille isn’t numerical superiority… I’m not interested in derailing into historical debate here; ask your favourite LLM for a fact check on the quoted portion.
I will say one thing, which is that the notion of “striking while their opponents are still preparing” actually smuggles in an assumption of numerical superiority^[1], and your point about “cohesion” is the morale differential that contributes to revolutions becoming self-fulfilling prophecies.
The point is that you will not be able to coordinate your way out of an AGI dystopia regardless of your numerical superiority or inferiority. Once we’re past the event horizon, what leverage do think you have left in a system that doesn’t need you for anything, that can crush you like a bug? Well, surely once enough people agree that Shit Sucks, perhaps because the UBI checks started to dry up without explanation, we can all… We can all just… Wait, what’s that buzzing sound?
1. ^
  Alice has an army of 100 soldiers, 10 of whom show up to the fight while 90 are “still preparing” (or on the other side of the Empire, or dragging their feet because they don’t care that much about defending Alice, are scared of Bob or secretly think he has a point). Bob has an army of 50 soldiers, 40 of whom show up to the fight (because they very much care about defeating Alice). Who has the numerical advantage?

koanchuk 28 Jan 2026 1:29 UTC
32 points
−4
on: koanchuk’s Shortform
TIL Eliezer said that he refuses to read Ted Kaczynski’s (aka the Unabomber’s) 1995 manifesto Industrial Society and Its Future because “audience should not be a reward for crime”, referring to the former mathematician’s mail bombing campaign that took the lives of 3 people and injured 23 more.
The ≈35,000 word manifesto was published by the Washington Post under the threat of him killing more people should they refuse, and its publication was encouraged by the FBI to produce new leads. His brother recognized his writing style, which led to Kaczynski’s arrest and subsequent life imprisonment, thus concluding the longest and most expensive manhunt in FBI history.
Rejecting the strategy of garnering attention by means of domestic terrorism is an understandable heuristic, but it’s worth noting that the consensus seems to be that Industrial Society and Its Future stands as a serious piece of political philosophy worth engaging with despite its origins.
It even had something to say about AI, where he predicts phenomena that we’re discussing here, 30 years later:
First let us postulate that the computer scientists succeed in developing intelligent machines that can do all things better than human beings can do them. In that case presumably all work will be done by vast, highly organized systems of machines and no human effort will be necessary. Either of two cases might occur. The machines might be permitted to make all of their own decisions without human oversight, or else human control over the machines might be retained.
If the machines are permitted to make all their own decisions, we can’t make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines’ decisions. As society and the problems that face it become more and more complex and as machines become more and more intelligent, people will let machines make more and more of their decisions for them, simply because machine-made decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the machines will be in effective control. People won’t be able to just turn the machines off, because they will be so dependent on them that turning them off would amount to suicide.
On the other hand it is possible that human control over the machines may be retained. In that case the average man may have control over certain private machines of his own, such as his car or his personal computer, but control over large systems of machines will be in the hands of a tiny elite—just as it is today, but with two differences. Due to improved techniques the elite will have greater control over the masses; and because human work will no longer be necessary the masses will be superfluous, a useless burden on the system. If the elite is ruthless they may simply decide to exterminate the mass of humanity. If they are humane they may use propaganda or other psychological or biological techniques to reduce the birth rate until the mass of humanity becomes extinct, leaving the world to the elite. Or, if the elite consists of soft-hearted liberals, they may decide to play the role of good shepherds to the rest of the human race. They will see to it that everyone’s physical needs are satisfied, that all children are raised under psychologically hygienic conditions, that everyone has a wholesome hobby to keep him busy, and that anyone who may become dissatisfied undergoes “treatment” to cure his “problem.” Of course, life will be so purposeless that people will have to be biologically or psychologically engineered either to remove their need for the power process or to make them “sublimate” their drive for power into some harmless hobby. These engineered human beings may be happy in such a society, but they most certainly will not be free. They will have been reduced to the status of domestic animals.
What links here?
- koanchuk's comment on Ian McKenzie’s Shortform by Ian McKenzie (2 Feb 2026 15:18 UTC; 1 point)

koanchuk 27 Jan 2026 23:50 UTC
11 points
4
in reply to: BryceStansfield’s comment on: Dario Amodei – The Adolescence of Technology
The CCP has consistently been an early adopter of any technology that allows them to solidify their control. They probably seemed unremovable even before they started tracking everyone through the cameras they set up everywhere.
With that being said, I assume that Dario waving the American flag and pointing at the evil reds across the Pacific is basically motivated reasoning.
During his discussion with Demis at Davos, he conceded that he would like to pull the brakes on AI development… were it not for the fact that We Must Win The AI Race Against China, therefore full steam ahead.

koanchuk 27 Jan 2026 23:21 UTC
56 points
28
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
So… --dangerously-skip-permissions at the corporate level?

koanchuk 27 Jan 2026 23:03 UTC
1 point
0
in reply to: Drake Thomas’s comment on: The Possessed Machines (summary)
“risks are so severe that no level of benefit justifies them” nah, I like my VNM continuity axiom thank you very much, no ontologically incommensurate outcomes for me. I do think they’re severe enough that benefits on the order of “guaranteed worldwide paradise for a million years for every living human” don’t justify increasing them by 10% though!
What about… a hundred million years? What does your risk/benefit mapping actually look like?

koanchuk 27 Jan 2026 16:46 UTC
1 point
0
in reply to: Viliam’s comment on: koanchuk’s Shortform
Yet if we extend the “+1 complexity” argument, we eventually reach a boundary where no human, however smart, could understand it. In principle nature could produce a human with the specific mutation necessary to apprehend it, which pushes the human cognitive horizon by some amount without actually eliminating it.
To the extent that AI can be scaled unlike the human brain, it might be able to form conceptual primitives which are so far outside the human cognitive horizon that biology is unlikely to produce a human intelligent enough to apprehend them on any reasonable timescale.