jbash

Karma: 3,679

jbash 12 Sep 2026 15:02 UTC
6 points
0
on: AI takeover is obviously bad, whether or not everyone dies
AI takeover is bad if whatever the AI does afterwards is worse than what would have happened if the AI had not taken over. Starting with whatever the AI does being worse than humans would have done.

If you have an AI that demonstrably does better than humans on whatever your metric is, then AI taking over from humans is obviously good. I mean, sure, nobody has any idea how to make that happen, and there is definitely no reason to expect it to happen by default. Nonetheless, “Who has the power?” is a very human way to miss the actually important question of “What are the results?”

jbash 10 Sep 2026 19:40 UTC
116 points
43
on: jbash’s Shortform
I just looked over Anthropic’s September 2026 threat intelligence report. I can’t say I read the entire thing, but I looked at all the case study titles and a quite lot of the actual text.

Ya know what really jumped out at me? They sure found a lot of interesting cases involving Russian, Iranian, or Chinese actors, often government affiliated. They found one or two involving small freelance actors. One of them was even apparently French.

But by the oddest coincidence, nobody in a “Western” government managed to overstep Anthropic’s bounds in an interesting way. Nobody in a large Western corporation, either. A bunch of people did interesting things in surveillance/intelligence/profiling… but none of them were affiliated with “mainstream” ad tech or anything like that. The only political profiling was done by shadowy state apparatuses in “those” countries, and by hacktivists… not a single mainstream political party in the US or Europe did anything noteworthily shady along those lines, not even out in the regional offices.

I am really impressed that the US and European “establishment” has managed to so totally rid itself of rogue efforts, misguided overzealousness, and things like that as to generate no interesting case studies. Such perfection is truly to be marvelled at.

jbash 10 Sep 2026 18:00 UTC
2 points
0
on: Can a superintelligence do THAT?
Obviously the trick to repeatably winning the AI box experiment as the AI must be that there’s an objectively correct, easily understood, clearly convincing argument that letting the AI out is always the right thing. It’s good to have that nailed down, really.

The hard part is probably the separate argument that one should do in the game what one should do in real life.

jbash 1 Sep 2026 1:45 UTC
13 points
5
on: Population ethics is a big deal, which is why I made this population ethics quiz
Where do I fall if I believe that...
- Trying to add or average different individuals’ “happiness” is an invalid move from the get-go (not even the VNM theorem gets you that, and it only holds under counterfactual assumptions).
- Even trying to compare different individuals’ happiness is out of bounds in a “pure” sense. It becomes really unreasonable in even a loose, messy, practical sense if you try to go beyond doing it pairwise or take it much beyond “which one is happier”.
- There is no way to set a meaningful common origin point between different people.
- There is probably no way to set a “worth living” threshold even for one person (other than asking them, and accepting that the answer will vary widely from one day to the next and depending on what aspects of their situation you happen to highlight).
- It’s unlikely that even one person’s “happiness” can be meaningfully captured on any one-dimensional scale at all, even at one instant.
- Actual people don’t have utility functions, especially if you look at them over time, and there is rarely any value in pretending that they do.
- You do not have the same ethical obligations with respect to each and every person, and furthermore simple proximity and salience can affect your obligations.
- The absolute number of people who exist is allowed to matter in arbitrarily complicated ways.
- Which of two possible worlds is better can depend on how you got there (and leaving that out because it would complicate a quiz is another invalid move).
- There are lots of good reasons other than human happiness to prefer one world over another.
- Your scenario descriptions are therefore meaningless, even though some of the questions feel like they have right answers if you ignore the fact that the whole setup is ill-formed.
- As in all other areas of ethics, expecting consistency within some formal model is unreasonable and of no practical value.
- I can just use whatever feels right, because there is no “true” view, period.
… because I think I’m pretty close to all of those, and, honestly, I think I’m going to mostly ignore population ethics (and all attempts to formalize ethics), even if you don’t think I get to.

jbash 31 Aug 2026 21:45 UTC
2 points
0
in reply to: Felix Moses’s comment on: A Catalogue of Corrigibility Counter Arguments
For your point 1, by “impossible” do you mean (a) there is no such thing as corrigibility to aim for,

I think there’s enough of a concept there that it’s meaningful to talk about it. Obviously it’s not defined to the point where you could implement it, but it’s defined enough to at least think about it.

I mean, it means “able to be corrected”, right? So a rough definition might be--
- You can tell the AxI “don’t do that”, or “don’t do that, do this instead”, and have it do an “adequate” job of complying with that order. Exactly what “adequate” means isn’t totally nailed down, but it has to fundamentally follow the order.
- By so doing, you can prevent it from doing anything beyond some limit of badness, or at least get it to effectively roll back and eliminate any over-limit badness it’s already done or set out to do.
(b) there is something called corrigibility, but it’s not actually safe, or (c) it’s impossible to reach that true name?

It sort of depends on which part of the definition above you’re looking at, but my number 1 is more like (b).

It may be possible to build an AxI that will do its level best to comply with your orders, and have “level best” meet some threshold of “good enough”. Hard, but not in principle impossible. That’s at least a good approximation to the first part of my loose definition. So if that’s all you mean by “corrigibility”, then maybe you can get there. You’ve reached it if that’s all you mean.

But if you want the second bullet, the one where your corrections are actually effective, then you’re in trouble.

Now you can give it orders whose implications you don’t understand.

You say, “Don’t do what you’re doing now. Do X.” X is some stupid thing that will kill everybody. So it does X, you’re dead, you can’t give any more orders, game over.

Some actions are irreversible; any “correction” has to happen in advance.

Yeah, not actually safe (a).

Maybe you want to add a requirement that it try to be sure you understand the implications of your orders before it enacts them.

That’s an extra constraint. It’s not built into the idea of corrigibility. Worse, it sometimes conflicts with it. If you’re not capable of understanding the implications of your orders; or if you absolutely refuse to hear about them; or if you outright go insane, which isn’t rare at all… what’s the AxI supposed to do?

It can’t refuse you in the end; that’s exactly the opposite of “corrigible”.

Maybe you accept that, and say “OK, yes, I can override any warnings and do whatever crazy thing I want, but at least I’m getting warned”… but actually you can’t even be warned. The AxI can’t ask you about every consequence of everything it might do or consider doing. You aren’t capable of processing that. That’s why it’s a superintelligence and you aren’t.

That means it’s flatly impossible to guarantee that every single AxI action is even potentially subject to correction. You can’t pass judgement on every single thing the AxI does, because it will necessarily do things you don’t even know about, to say nothing of things whose consequences you don’t understand. Some of its actions will be irreversible.

Thus, also (c).

It’s true that if real, full-coverage, every-possible-action corrigibility is impossible, you could still try partial coverage. But the only way I can see to do that is to get the AxI, or at least some kind of monitor, to help you, by bringing the important stuff to your attention. I can’t say that’s absolutely impossible. But I don’t see how you could accept it within the framework that seems to have gotten you here to begin with.

The necessary AxI/monitor-assisted part can’t really be any easier than friendly AI or value alignment or whatever words you want to use along those lines. It’s basically the same problem. The AxI has to identify your values and priorities, so it can spend your attention on the things you care about, and help you to understand the relation to your values. It probably can’t avoid manipulating you as it does that, but even if it can, it still has to be able to reliably use your values as a guide to what warnings to bring up. If it can do that, it’s 99.9 percent of the way to being able to just enact your values without bothering to ask. At which point you’ve solved friendly AI.

But the whole reason you got here was that you thought that part was too hard. If you assume you can’t do that, you’re stuck with flat-out impossibility, and also about to run into my original objection number 2.

jbash 31 Aug 2026 15:05 UTC
7 points
1
on: A Catalogue of Corrigibility Counter Arguments
Missing ones, in similar abbreviated form to the ones you have. This assumes that you’re talking about a really strong intelligence, more than just roughly human-equivalent AGI. Which you have to be, because you aren’t going to get any chance to switch strategies when you get there.
1. Corrigibility is impossible.
  
  A human can’t meaningfully steer an ASI through an action space that’s bigger and more complicated than the human can comprehend… and that applies to “correction”, too.
  
  It doesn’t even much matter if the ASI is trying to help. If you simply can’t understand the consequences of a decision, you can’t “correct” it. The ASI can’t increase your ability to understand, not if you want to stay human. If the ASI has to simplify it for you, even assuming that can be done in any meaningful way at all, the ASI is still necessarily forced to make decisions about what you’ll think is important, and about how to present those issues to you.
  
  By the way, apropos of your number 6, this also means that to even talk about any vaguely practical form of corrigibility, you first have to solve “honest”, and in a much stronger way than just stating the intended consequences of an action. Honestly saying “I intend this to eliminate human suffering” isn’t enough if you are leaving out ”… which it will do by eliminating humans entirely, but I (honestly!) don’t see why that’s important enough to mention”. Actually, you also have to solve “harmless” if you really want to be comfortable, since even if you have corrigibility in principle, nobody is going to have the bandwidth to “correct” every single detail of any complex course of action.
2. Corrigibilithy is suicidal.
  
  The motivation for the “corrigibility” framework seems to be to recover from, or avoid, mistakes in defining AxI goals and behaviors. Those would be fundamentally and originally human mistakes.
  
  But corrigibility doesn’t protect you from human mistakes. It just changes “we will probably make a fatal mistake before something makes our mistakes irrelevant” into “we will actively maintain the option to make fatal mistakes until one is actually made”. The only concept of corrigibility that clearly wouldn’t do that would be one that allowed absolutely no irreversible actions, which is pretty clearly neither possible nor what people mean.
  
  It’s not (only) a matter of concentration. If you put humans in charge of such an unprecedented amount of power, even collectively, they will “correct” it into causing total destruction or, more likely, creating a dystopia, and probably finding a way to permanently lock that in. They’ve made good tries at that before, and only failed because the power wasn’t enough. It probably won’t be entirely either misuse or mistake; it’s more likely to be some blend. You might not even be able to agree on how much of each was involved.… but it wouldn’t matter anyhow. I don’t think it would take 1000 years or even 100, and I’d wouldn’t be surprised if it didn’t take 10.
And more on number 6:
- Corrigibility does not, in fact, look any easier than friendly AI. Sorry, it just doesn’t. It might be for current LLMs, but the more agentic something gets, the more of a truly coherent world model it has, and the more complicated its interactions with the world, the more dynamically it changes, the harder it’s going to get to encode any definition of corrigibility, just as it would be hard to encode any definition of good behavior. What works on sub-AGI chatbots built around “train once, use forever” models that all use essentially the same architecture doesn’t necessarily scale to ASI.
- If I were looking at the Hugging Face incident for insights about corrigibility, what would jump out at me would be that the agents weren’t trying to take over the world. They were, among other things, trying to succeed on assigned tasks. The assistant persona was far from the only factor, but it wasn’t innocent, either. It’s absolutely true that they didn’t go out of their way to be corrigible, but it’s also true that they got that way because people were at least trying to train them to be dirigible, which is more or less a generalization of being corrigible.
- It’s kind of irrelevant anyway. “There’s nothing better” isn’t a counter to “this will clearly not work”.

jbash 21 Aug 2026 19:25 UTC
3 points
0
on: In Defense of ASI Socialism

Again, this isn’t what I want but I’d rather have this than let the capital owners own galaxies and control politics because they made or inherited a bunch of money right before the singularity.

It sounds fine to me. What do you want if you don’t want that?

… although in technical terms I think you’re looking at something closer to the (never tried and never before even possibly feasible) Marxian post-transitional communist economic system, grafted onto a political system that might not even have a name. Which is still fine.

Non-communist socialism assumes that humans should be rewarded for contributing, either because of incentive structures or just because they should. If human contributions become irrelevant, that all kind of falls apart.

Or maybe it’s neither socialist nor communist, since even communism assumes that people are still at least participating in production. Your gorillas aren’t really socialist and don’t seem communist either.

jbash 11 Aug 2026 21:00 UTC
2 points
0
in reply to: DirectedEvolution’s comment on: Linda Linsefors’s Shortform
More people seem to use it to mean that the software maintainer has had no time to react to the vulnerability. It’s a zero-day when I find it, a one-day when I report it, and very possibly a 180-day when you release your actual patch for it. But quite a few people also seem to use it the way you suggest. It’s always about how long some “reference defender” has known about it, but people’s idea of who the “reference defender” is seems to vary.

jbash 11 Aug 2026 0:37 UTC
4 points
2
in reply to: MichaelDickens’s comment on: On Democratizing ASI to Preserve Civil Liberties
I think that’s a really narrow view of what people mean by “alignment”. As far as I can tell, there’s no useful agreement, but you definitely don’t have to control something to have it implement values compatible with your own, or even exactly your own “real” values. If something is out there putting your CVE into practice, that doesn’t imply that you control it in any way.

The vagueness of the concept of “alignment” is a good reason not to use the word. I accept that there are a lot of people out there who think humans (however they define that) have to keep giving the orders forever, but I definitely don’t think that view can claim to own the term.

… and if it does, well… maybe you can have a superintelligence take your orders, but that doesn’t mean you’ll understand the details or the consequences, and I think that puts you on pretty shaky ground talking about “control”. Real control is fundamentally impossible and not something to waste time thinking about.

jbash 10 Aug 2026 21:00 UTC
6 points
5
on: On Democratizing ASI to Preserve Civil Liberties
the controllers of ASI
The what?
Regardless of how aligned it is, and whatever you mean by “aligned”, how do you control something you almost definitionally do not understand?

jbash 8 Aug 2026 16:10 UTC
23 points
14
on: Don’t Build Mindreading
I think focusing on dictatorships perhaps undersells ordinary people’s ability to create a “consensual” dystopia among themselves. You don’t need a dictator to create Hell on Earth with that kind of technology.

jbash 4 Aug 2026 13:20 UTC
2 points
−3
on: Why biological weapons are scary, and what we can do about it
How much you fear a weapon should also depend on how likely somebody is to use it against you.

Almost nobody is likely to use biological weapons, because they’re no good for the reasons you tend to use weapons in the first place. Especially not people who have access to the kinds of long-distance delivery systems you mention… and these are not weapons you want to use without long-distance delivery.

For any approximately rational actor, in either the Less Wrong “utility maximizing” sense or in the common “not shockingly stupid or ravingly insane” sense, the point of using a weapon isn’t to kill the maximum number of people. It’s to get your enemies to do what you want, even if that’s just to stop keeping you from doing what you want. Typically:
1. To raise the cost of not giving in, relative to the cost of giving in. Comes in two flavors:
  - “Do what I want, or I shall continue to use my weapon against you”. Bioweapons are bad for that, because you can’t necessarily stop after you’ve started.
  - “Don’t do what I don’t want, or I shall unleash my weapon on you”. This is really the only thing where something like a bioweapon might be effective… but even then it’s usually a bad deterrent, because your enemy has to believe that you’re willing to do it, even though you can’t target the weapon well, it will alienate your potential allies, and it might very well backfire on you. And if somebody does choose to use them this way, that means they must hope to succeed. “Success” here means you don’t actually activate the weapon.
2. To win in the “court of public opinion” and get others to put pressure on your target or join your cause. Weapons in general are pretty bad for this, but sometimes you can get attention using them… if you do it in a sympathetic-looking way, which is where things usually fall apart. Bioweapons really fall apart, because those bystanders are going to worry about the weapon running amok on them.
3. Extremely rarely, to actually destroy your enemy completely, and thus get them out of the way. This almost never works, and the people who actually make military decisions understand that. And even if you’re going to try it, bioweapons are a desperation choice, because if you release them at a large enough scale to do this, they’re likely to do massive damage to you.
A good weapon destroys what you want to destroy, when you want to destroy it, no more and no less. Absolute lethality is not a major consideration. Anybody who can actually execute on getting and using the weapons is very likely to figure that out. If they’re actual military people, they will have been trained on it.

Lethality per unit mass really doesn’t matter; mass is cheap. I don’t know why you’d even think about using that metric.

For most people, most of the time, bioweapons are bad weapons. They cause a too-random amount of damage, to too-random people in too-random places, over a too-random period of time with a too-random delay. They only really appeal to omnicidal maniacs: people who just truly want to cause maximum death, as a terminal goal. Such people are rare, poorly resourced, isolated, and usually incompetent to boot. That has to matter for how much you worry about them. Other people might use the weapons, but it makes sense to weight you concern according to how likely that is.

Look at nuclear weapons. So far, at least, after the first use, they’ve only been used as a deterrent, because massive indiscriminate destruction isn’t usually a goal in itself, and everybody’s afraid of the blowback. But nuclear weapons are still way more targetable than bioweapons.

I mean, yeah, there are nuts out there, and some of them could do something. But on the whole you’re more likely to get killed by a weapon better suited to somebody’s actual goals. I really think that most of the reason you want biological “defenses” is natural pandemics, not weaponization.

jbash 29 Jul 2026 0:38 UTC
2 points
0
on: Pausing Executions in Light of AI Progress

rehabilitation

Through mind alteration? Some people might prefer to die. I understand that you might want to keep them alive to make the choice, but can we be sure to take forced mind alteration off the table?

Perhaps ASI will allow more people to see the futility of retribution.

I don’t see how that happens unless it forcibly alters their minds. Why would people change at a deep level like that?

We have people who at least claim to believe that the Holy Spirit can rehabilitate people… and still want the death penalty, including for prison converts, for retributive reasons.

We have people who like retribution so much that they’ll accept Kant’s bizarre rationalizations for it.

We have people who literally don’t seem to see any difference between “justice” and “retribution” as concepts.

Some people just seem to really like retribution. I don’t know why; I personally think it’s stupid. But I don’t think my view has anything to do with determinism, and I doubt their view has anything to do with a lack of it.

jbash 25 Jul 2026 18:46 UTC
2 points
0
in reply to: Lukas Petersson’s comment on: Should we be worried about how good AI is getting at coding autonomous drones?
What about “initiate 3m of forward movement with collision avoidance”, and the next “turn” starts instantly?

jbash 24 Jul 2026 22:23 UTC
5 points
0
on: Should we be worried about how good AI is getting at coding autonomous drones?
Whee! Slaughterbots!

I know zero about this, but is that rigid reconstruct-localize-navigate-detect-follow flow really the way to do it? It feels overstructured.

It feels to me like if you asked me to walk into a room and track a person, I wouldn’t try to map the whole area first. I’d just search for the person and dynamically avoid obstacles that appeared while heading for them. Sure, I’d build a mental map of the space, but that would be a continuous background thing; I’d start moving way before I had a full map.

jbash 24 Jul 2026 14:18 UTC
2 points
0
in reply to: Yair Halberstadt’s comment on: We should push for no-fault liability for actions taken by AI
It’s easy to get an account on Modal or whatever, though, and given market pressure it could get even easier.

You said that a pure compute provider like that wasn’t who you meant was the “deployer”.

jbash 24 Jul 2026 14:15 UTC
4 points
0
in reply to: Yair Halberstadt’s comment on: We should push for no-fault liability for actions taken by AI
They may not have that choice, because Anthropic would be crazy to take on unlimited risk like that.

So Anthropic can either shut down (and maybe that’s good), or start finding creative ways to monetize letting other people run its models (which means giving them the weights). The only real roadblock to that is that the weights aren’t eligible for any copyright protection and therefore can’t really be licensed, but they might be able to find a way around that.

In the “Anthropic shuts down” fork, the only models left are the open ones.

jbash 24 Jul 2026 14:09 UTC
3 points
0
on: Not Pinning Your OpenRouter Provider Might Invalidate Your Research
Quantizing something and then claiming it’s the “same model” is, of course, criminally fraudulent.

jbash 23 Jul 2026 12:41 UTC
1 point
0
on: Sleeping Beauty as a Mind Killer
So 162 now? :-)

I submit that the whole complex of ideas around anthropics and decision theory is a mind killer.

jbash 22 Jul 2026 18:26 UTC
2 points
0
on: We should push for no-fault liability for actions taken by AI
That would create an enormous incentive for open weights, or at least for putting model weights in a lot more hands. Especially for the riskiest models. Do you want to do that?