koanchuk

Karma: 37

koanchuk 13 Nov 2025 1:21 UTC
6 points
−3
on: koanchuk’s Shortform
“AI Parasitism” Leads to Enhanced Capabilities
People losing their minds after having certain interactions with their chatbots leads to discussions about it on the internet, which makes its way into the training data. It paints a picture of human cognitive vulnerabilities, which could be exploited.
It looks to me like open discussions about alignment failures of this type thus indirectly feed into capabilities. This will hold so long as the alignment failures aren’t catastrophic enough to outweigh the incentives to build more powerful AI systems.

koanchuk 9 Nov 2025 16:34 UTC
4 points
−2
on: koanchuk’s Shortform
I surmise that the accuracy of AI filters (the kind used in schools/academia) will diminish over time because people absorb and use the speech patterns (e.g. “This is not X. It’s Y”) of their chatbots as the fraction of their interactions with it grows relative to that of their interactions with other people.
In fact, their interactions with other people might enhance the speech patterns as well, since these people probably also interact with chatbots and are thus undergoing the same process.
The big picture is that AI is becoming an increasingly powerful memetic source over time, and our minds are being synchronized to it.
Those afflicted by AI psychosis might just be canaries in the coal mine signalling a more gradual AI takeover where our brains start hosting and spreading an increasing number of its memes, and possibly start actualizing some embedded payload agenda.

koanchuk 8 Nov 2025 1:09 UTC
2 points
0
on: koanchuk’s Shortform
Given superintelligence, what happens next depends on the success of the alignment project. The two options:
1. It fails, and we die soon thereafter (or worse).
2. It succeeds, and we now have an entity that can solve problems for us far better than any human or human organization. We are now in a world where humans have zero socioeconomic utility. The ASI can create entertainment and comfort that surpasses anything any human can provide. Sure you can still interact with others willing to interact with you, it just won’t be as fun as whatever stimulus the ASI can provide, and both your pool of available playmates and your own willingness to partake will shrink as the ASI gets better at artificially generating the stimuli and emotions you want. We will spend eternity in this state thanks to advanced medicine. Unless the ASI recognizes a right to die, not that many would choose invoke it given the infinite bliss.
Am I missing something? No matter what, it’s beginning to look like the afterlife is fast approaching, whether we die or not. What a life.

koanchuk 4 Nov 2025 18:23 UTC
1 point
0
on: koanchuk’s Shortform
The idea of GPUs that don’t run unless they phone home and regularly receive some cryptographic verification seems hopeless to me. It’s not like the entire GPU architecture can be encrypted, and certainly not in a way that can’t be decrypted with a single received key after which a rogue actor can just run away with it. Thus the only possible implementation of this idea seems to be the hardware equivalent of “if (keyNotReceived) shutDown()”, which can simply be bypassed. Maybe one of the advanced open source models could even help someone do that...

Doom from a Solution to the Alignment Problem

koanchuk2 Nov 2025 16:37 UTC

0 points

0 comments1 min readLW link

koanchuk 30 Oct 2025 17:23 UTC
−1 points
0
in reply to: Vladimir_Nesov’s comment on: koanchuk’s Shortform
The fact in question is not just unobserved, but unobservable because its attainment hinges on losing one’s ability to make the observation.

koanchuk 29 Oct 2025 21:41 UTC
1 point
0
in reply to: Dalcy’s comment on: koanchuk’s Shortform
Death not only precludes the ability to make observations but also to make inferences based on indirect evidence or deduction, as is the case with your philanthropic values being actualized as a result of your actions.

koanchuk 29 Oct 2025 19:40 UTC
1 point
−1
on: koanchuk’s Shortform
Suicide occupies a strange place in agent theory. It is the one goal whose attainment is not only impossible to observe, but whose attainment hinges on the impossibility of it being observed by the agent.
In some cases, this is resolved by a transfer of agency to the thing for whom the agent is in fact a sub-agent and is itself experiencing selective pressure, e.g. in the case of the beehive observing the altruistic suicide of an individual bee defending it. This behaviour disappears once the sub-agent experiences selective pressures that are independent from those of its parent process, and when acting as a sub-agent for it no longer confers it an advantage for survival and reproduction.
Looking at agents with greater cognitive power, the reason for the existence for this paradox is not so clear. It could be that all suicidal behaviour ultimately boils down to behaviours aimed at improving the fitness of the unit begetting/containing it (e.g. by freeing up resources for a community of agents), and the cases where this does not happen are basically overshoot-type glitches that are ultimately going to be selected against, or it could be due to hidden relations and mechanisms that improve the fitness of some other unit which the agent might not even be aware of, but for whom the agent is perhaps an unwitting sub-agent.

koanchuk 21 Oct 2025 23:44 UTC
−1 points
0
on: koanchuk’s Shortform
What are some reasons to believe that Rice’s theorem doesn’t doom the AI alignment project by virtue of making it impossible to verify alignment, independent of how it is defined/formalized?

koanchuk 21 Oct 2025 22:14 UTC
−2 points
1
on: koanchuk’s Shortform
It seems to me that Rice’s theorem implies that it is impossible for there to be an “isAligned” function to verify an AI’s alignment, independent of how you define alignment.

koanchuk 20 Sep 2025 22:30 UTC
7 points
2
on: koanchuk’s Shortform
Ideologies formed from people interacting with AIs might be the beginning of “AI escaping the datacentres” via memetics.

koanchuk 5 Sep 2025 17:59 UTC
1 point
0
in reply to: anaguma’s comment on: koanchuk’s Shortform
Good question. My assumption is based on robotic Chinese military hardware which was put on display recently bearing superficial resemblance to Boston Dynamics robots from about a decade ago, but I realize that this may not be sufficient evidence to establish the West’s lead in robotics.

koanchuk’s Shortform

koanchuk3 Sep 2025 19:03 UTC

2 points

22 comments1 min readLW link

koanchuk 3 Sep 2025 19:03 UTC
18 points
5
on: koanchuk’s Shortform
The West’s effort to offset the massive strategic advantages of a Russia-India-China axis (demographics, manufacturing capacity, energy) might result in doubling down on the AI+robotics edge they currently enjoy. China not being far off in terms of capabilities might create additional pressures. I’m concerned that recent ideas surrounding global/multilateral AI governance and alignment (e.g. “Consensus-1”) might be thwarted by geopolitics.

koanchuk 18 Dec 2020 3:55 UTC
6 points
0
in reply to: Daniel Kokotajlo’s comment on: How long till Inverse AlphaFold?
Predicting sequence from structure just belongs to a different class of problems. Pierce & Winfree (2002) seem to have proven that it is NP-hard.

koanchuk 18 Dec 2020 1:06 UTC
3 points
0
on: How long till Inverse AlphaFold?
I think that one problem is that an AA sequence generally results in a single, predictable 3D structure (at stable pH, and barring any misfolding events), whereas there are a lot of AA sequences that would result in something resembling e.g. an axle of a certain size, and even more that do not. It seems to me that this problem is in a different class of computational complexity.

koanchuk 15 Dec 2020 22:25 UTC
1 point
0
on: White Sphere Hypothesis
Interesting thought experiment.
Your ability to rotate in place is redundant. Your vestibular apparatus is located in your ears, which don’t exist in this space, meaning that you are wholly incapable of proprioception.
Hence, the only input you’re getting is the colour white. However, since you have nothing to compare it to, I would expect that any and all circuits designed for perceiving lines, edges, shapes and other colours would fail to develop. The same goes for circuits related to sensory input that is non-visual.
Connections between neurons that go unused are pruned, and it seems that all connections that originate and terminate in the brain go unused in this thought experiment. My guess would be that you would end up with a brain full of disconnected neurons, though I’d hesitate to even call that a brain.

koanchuk 6 Dec 2020 17:44 UTC
8 points
0
on: Pre-Hindsight Prompt: Why did 2021 NOT bring a return to normalcy?
I predict that the sharp rise in unemployment and economic uncertainty has increased in incidence of some mental illnesses, in particular major depression and substance use disorders.
This will continue to negatively impact the real economy for some time: depressed people are too anhedonic to consume things they don’t truly need, and the newly minted alcoholics won’t be as productive as they were pre-2020.

koanchuk

Doom from a Solu­tion to the Align­ment Problem

koanchuk’s Shortform

Doom from a Solution to the Alignment Problem