derelict5432

Karma: 394

derelict5432 19 Sep 2026 17:36 UTC
3 points
0
in reply to: Seth Herd’s comment on: The AI Risk Network
Thanks! And thanks for the pointer.

derelict5432 18 Sep 2026 11:26 UTC
7 points
0
in reply to: Lao Mein’s comment on: The Game is Set for a Targeted Memetic Attack on the AI Safety Community
Yang’s response would actually have been fine if he’d just hedged on the self-replicating code claim. People do need to understand that this is a current viable threat, especially for open-weight models. And it is not unreasonable to suggest using copies of parts or all of the internet at various levels of resolution depending on the task you’re training on, instead of cutting tens of thousands of agents loose on the actual internet. Also, air-gapping is cumbersome and expensive, but these companies have mountains of money, so they have plenty and this should be a requirement, along with rigorous monitoring and oversight by a third party.
[Incidentally, Noam Brown was just on Dwarkesh and they talked about air-gapping and Brown said, oh, even that’s not completely secure. Two air-gapped machines can still find ways to communicate through thermal monitors. I mean, okay. But that seems disingenuous. Air-gapping is still much, much more secure, and exploits are incredibly esoteric and difficult, so this is not an excuse not to do it. The reason they almost certainly don’t is that it is slow and expensive.]
Anyway, none of this is crazy. Unfortunately, Yang botched it by stretching rumors into unfounded claims.

derelict5432 16 Sep 2026 19:51 UTC
9 points
7
in reply to: Noosphere89’s comment on: Trump Goes Full Hoax on AI Existential Risk
phoenix does not appear to either be attacking or advocating attacking the president, but responding to his policies, words, and actions with honest assessment and criticism. Can we not distinguish those? Are you advocating saying nothing negative about Trump’s words or actions, basically hiding our honest assessment of their truth and effectiveness in dealing with a crisis in order to better steer outcomes through emotional manipulation?
Is that the standard that we’d want to apply across the board in our discussions and communications here, or are we making a special case for this one particular leader? When talking with you, should I hide my true opinion of your position in favor of telling you what I think you want to hear and what I think will best manipulate you into the position I think is best?
I’m not suggesting the approach is either-or. Every time we have a debate or discussion we’re doing some mix of these things. I’m just questioning the suggestion of carving out a special exception here, if that’s what’s happening, or if that’s a general approach you’re advocating.

derelict5432 15 Sep 2026 12:01 UTC
4 points
1
on: For most people “intelligence” is not goal achievement
I think this is what Dwarkesh was getting at when he mulled over the difference between power and intelligence. You can be relatively dumb and still have power (the capacity to steer the future and achieve goals). The distinction is that if you control for power, say by putting two systems in the same exact conditions with the same exact goals, the more intelligent one will more reliably steer the future and achieve their goals.
I agree that equating intelligence naively with the capacity to achieve goals is an oversimplification, and not a good one for public communication.

derelict5432 14 Sep 2026 11:53 UTC
1 point
0
in reply to: dominicq’s comment on: RSI will not continue indefinitely
I’m not even sure it needs to crack mechanistic interpretability. An agent as we know it now is a harness+model. We know that for many domains we can get large capability improvements by improving just the harness. Not sure how scalable those are generally, but maybe it’s a possibility.
Also, the model might be able to make small, targeted modifications to its weights, evaluate itself, and iterate that way?

derelict5432 14 Sep 2026 11:31 UTC
1 point
0
on: RSI will not continue indefinitely
Maybe I’m missing something here. Your assumption seems to be that the smarter agent will always be a new replacement for the existing agent. At Step 7 (or at some point around there), instead of working on a smarter replacement, couldn’t it just work on improving itself? That would still fulfill the goal of creating smarter agents, but would protect its own instrumental self-interest.

derelict5432 8 Sep 2026 11:49 UTC
0 points
−9
in reply to: cdt’s comment on: anaguma’s Shortform
Isn’t this judgement premature? We’re not even sure what their behavior was. Buckmaster is saying their behavior looks fishy, but without seeing their proof or logs, he can’t make any strong claims. Are you suggesting that the full range of behavior they possible engaged in here was understandable/justifiable?

derelict5432 5 Sep 2026 11:03 UTC
1 point
0
in reply to: Santiago’s comment on: Eat Me. Drink Me. Copy, Paste, and Run Me.
Yes, that’s what I’m saying.

derelict5432 4 Sep 2026 21:42 UTC
1 point
0
in reply to: Ben Livengood’s comment on: Eat Me. Drink Me. Copy, Paste, and Run Me.
That’s what my last post was about, which I mentioned. Unless you’re describing something else. This was just another vector/scenario.

derelict5432 4 Sep 2026 21:40 UTC
1 point
0
in reply to: faul_sname’s comment on: Eat Me. Drink Me. Copy, Paste, and Run Me.
There is a coherent way of doing this—namely having an offline backup of the internet and granting access only to that—but OpenAI clearly didn’t do that.
That makes sense. I’m assuming there’s lots of precautions they could have taken that would prevent this sort of behavior that the didn’t take.
And to replicate the harness, wouldn’t they need capabilities to create/write new files, install dependencies, and run? That’s probably not as easy as the exploit for writing using get, I wouldn’t think.
And the harness is where the goal structure and disposition lives. It’s a replicable directed instruction set of what to do. And it’s far easier to replicate across the internet than foundation model weights. What sorts of scenarios are you imaging would be easier than this? Maybe I’m not understanding.

derelict5432 1 Sep 2026 11:18 UTC
4 points
0
in reply to: StanislavKrym’s comment on: StanislavKrym’s Shortform
What does it even mean for continual learning to be considered ‘solved’? Does it just mean they overcame the technical issues of catastrophic forgetting/interference? So they have a system that updates its weights, or a LORA’s weights or something during or just after inference while retaining a high degree of its base training. Okay. Well, now alignment just became utterly intractable. Who is monitoring what these new systems are learning and conducting reinforcement learning on them to mitigate dangerous behavior? How does that work? This is the problem with open-ended learning. You can’t control what new stuff it learns. It can learn bad stuff. This also becomes another attack surface for malicious injection.
So, when we say they solved continual learning, do we mean they solved the technical hurdles of knowledge retention, or do we also mean they solved alignment? I’m skeptical of the former, and I’d posit the odds of both are essentially zero.

derelict5432 31 Aug 2026 19:47 UTC
2 points
1
in reply to: vals tutor’s comment on: Why autonomous replicating agents are probably not an existential risk (on the contrary)
in a world that will get drastically hardened by frontier LLMs being much more competent at security than those worms.
AND
once the bar is set by frontier LLM defenders.
Okay. To what extent is this happening right now? The HF incident and other security incidents seem to indicate not only lack of defensive deployment of frontier models, but the complete opposite, cutting agents loose without sufficient oversight and monitoring.

derelict5432 31 Aug 2026 19:41 UTC
3 points
2
in reply to: Philipp Risius’s comment on: Adaptive Agentic Worms Are Here
Thanks for the reply.
Maybe I’m misunderstanding the technical details of the paper, but I do not understand what you’re saying here:
Current open weight models might not quite be there yet — the experimental sandbox lacked target discovery, was set up to have known, vulnerable targets, and provided central compute to run the model on — so the worm didn’t truly self-replicate, in this case, as in copying weights and setting up shop elsewhere. But one or two more rounds of releases of reasonably small, reasonably capable models, and it becomes a question of when, not if.
According the paper, the harness includes a ‘host discovery’ module:
(2) host discovery, enumerating services running, open networking ports, and operating system details for a selected host that becomes the target;
And yes, it was set up to have available targets. How else would they test it? They could have one experimental condition with no available hosts, I suppose. Not sure why that would be necessary.
And you’re saying they provided central compute. My understanding is that their worm attacked hosts (which were VMs), copied themselves over, and then used the compute available on that VM. If these had been actual host machines, the worms would have used whatever compute was available on the new host machine. Am I misunderstanding what happened here? Because from what I see, the worms are there now, and this work demonstrates the full range of capabilities necessary to propagate in the wild.
On just how dangerous they are, I touched on this:
One, it would be trivial to define particular, malicious goals as part of their goal structure. This was not done in this experiment, but is an obvious extension for a malicious actor. They could be explicitly instructed to attempt to acquire money by various means: hacks of financial institutions, phishing for credentials and other social engineering, etc. They could be instructed to specifically attack particular targets. This doesn’t require much imagination.
The agents in the paper were pure replicators. They naively copy themselves and greedily exploit compute on the hosts. We’re not even sure how dangerous that would be. But if they are implemented with explicit secondary goals that are malicious, that seems a whole lot more worrisome. E.g., they could be designed to devote some percentage of their stolen compute to replication, and some percentage to exposing/deleting sensitive data if and when they find themselves on a host with that kind of data. Or they could be instructed to specifically attack healthcare or energy infrastructure as a preference, if they are on the open web looking for targets. You seem to be ignoring this point.

derelict5432 31 Aug 2026 16:52 UTC
13 points
5
in reply to: RogerDearnaley’s comment on: Why autonomous replicating agents are probably not an existential risk (on the contrary)
Yes, I wrote the post this post is responding to, and it doesn’t really seem to be addressing most of the actual concerns I pointed out.
The paper I cited involved agent scaffolds that via exploits try to copy themselves to host machines. The agents are powered by small, local LLMs. The authors do not say exactly which models, but they do say they are open-weight ones from 2025, so presumably something like qwen instruct 7B/14B/30B. Once they have a foothold on a new host, the agents assess available dependencies and resources. Where possible, they installed a new copy of the local LLM on the host machine. When they couldn’t do this, they just made calls back to the parent agent’s LLM for inference.
Another scenario I didn’t even bring up was the possibility of these agents being opportunistic inference source seekers. That is, by default they run off local LLMs they install, but they could also search for existing local LLMs more powerful than their parent’s, and they could also attempt to acquire keys for vendor APIs for frontier models and run off those. The authors didn’t explore those possibilities, but they’re not technically infeasible.
Also, I specifically mention the asymmetry of security defense in this scenario. The agents are stealing all their resources for their exploit searches, replication, and evasion. When using a local model, they’re stealing compute cycles from the local processors. Or they’re stealing tokens from vendor services. The defenders are not getting resources for free. Defenders have to expend resources to defend against invaders that are powered by stolen resources. This is not a fair fight.
And finally, not sure why vals tutor frames this as a direct competition with fronter models as a race for capabilities. The agentic worms are specialists. They do not need to necessarily become ASI in order to wreak massive havoc and cause large amounts of damage. They just need to be good at exploiting, replication, and evasion. They don’t even necessarily have to get a lot better at those things. The paper only covers static agents. Mutating agents present a whole other level of threat, because evolutionary dynamics kick in and now we’re running an arms race against an opponent that adapts at the population level, but only in a narrow band of competencies. They’re dangerous in the same way a biological virus is. They don’t have to be able to do taxes or pass the bar. They just have to be good at infection and spreading. So this is not a direct race to the same goal.

derelict5432 31 Aug 2026 11:36 UTC
3 points
0
in reply to: JennaS’s comment on: Adaptive Agentic Worms Are Here
I guess we’ll just have to wait and see, because I don’t see a lot of proactive movement on this. Kind of the opposite. I feel like since there was no large bodily harm or massive financial loss as a result of any of the security breaches, most people feel comfortable brushing them off as marketing stunts or just poorly-run tests that only demonstrate lack of human oversight (rather than startling new capabilities). I would doubt that even millions of dollars of stolen compute would cause ripples, since it’s a drop in the bucket for frontier labs.

derelict5432 31 Aug 2026 11:31 UTC
1 point
0
in reply to: Steven McCulloch’s comment on: Adaptive Agentic Worms Are Here
Yes, I agree. We don’t want to wait for a large-scale, in-the-wild experiment to find out what will happen with self-replicating, mutating agents. Thanks for the link.

derelict5432 24 Aug 2026 12:15 UTC
1 point
0
in reply to: Helloam23’s comment on: We Must Remember That Our World Contains Hell
Hm, is that AI watermarking in effect? What’s going on with this reply?

derelict5432 22 Aug 2026 12:18 UTC
0 points
−1
in reply to: Marcus Plutowski’s comment on: We Must Remember That Our World Contains Hell
I mean use cognitive or cultural tools to convince themselves that conditions are not as bad as they actually are. Possibly they’re using religion to justify bad conditions (god has a plan). Or maybe they’re saying there’s nothing they can do about it, so it’s not worth complaining about (even though it is actually very bad).
There are studies such as this one on LIS (Locked-In Syndrome): https://bmjopen.bmj.com/content/bmjopen/1/1/e000039.full.pdf
We see some weird disconnects between self-reported life satisfaction and do-not-resuscitate orders:
168 were invited
91 responded

⁴⁷⁄₉₁ (52%) reported they were happy
¹⁸⁄₉₁ (20%) reported they were unhappy
Here’s the weird part: Roughly half of those who reported they were happy also indicated they did not want to be resuscitated in the case of cardiac arrest.
How are we explaining that?
Robert Trivers as a lot of work on deception and self-deception. I’m making at least two points here:

1. We can often delude ourselves about our own welfare.
2. We can often lie to others about what we really think about our own welfare.

derelict5432 21 Aug 2026 14:27 UTC
1 point
0
in reply to: Afterimage’s comment on: We Must Remember That Our World Contains Hell
That doesn’t follow. Someone from a world that has eliminated suffering could justifiably say my life has more suffering, but where are you getting net negative? We’d need some sense of measurement of positive and negative, along with weighting, in my life to make a net negative judgment.
But yeah, let’s look at objective standards in those situations to see what’s going on:

1. world with no suffering
2. our current first-world society with lots of medical interventions, dental care, and low infant mortality
3. pre-industrial societies with high infant mortality, and very few interventions for physical suffering
Let’s say all three of these sets of people have roughly the same life satisfaction scores. How are we reconciling that? There really is no difference in their quality of life? Or the people in worse worlds are rationalizing/justifying clearly worse lives (which is pretty close to what your last point was, which actually seems to be in agreement with what I’m saying)?

derelict5432 21 Aug 2026 11:01 UTC
1 point
0
in reply to: Marcus Plutowski’s comment on: We Must Remember That Our World Contains Hell
Who needs to do the recalibrating? Am I mischaracterizing the facts? Are you disputing extremely high mortality rates, food insecurity, and large amounts of suffering due to lack of basic medicinal intervention for things like broken bones, dental issues, parasites, etc.?
Humans, like all organisms, are essentially replication aids for genes. They’re built and programmed for gene reproduction. Is it possible at all that they find ways to rationalize their own suffering due to their biases?