jbash

Karma: 3,056

jbash 15 Oct 2025 13:45 UTC
3 points
0
on: Can LLMs Coordinate? A Simple Schelling Point Experiment
I would have done a lot worse than any of them.

jbash 13 Oct 2025 20:59 UTC
4 points
0
in reply to: Random Developer’s comment on: Wei Dai’s Shortform
Does that mean that you think it’s more likely you can safely build a superintelligence and not remain in control?

What load is “and remain in control” carrying?

On edit: By the way, I actually do believe both that “control” is an extra design constraint that could push the problem over into impossibility, and that “control” is an actively bad goal that’s dangerous in itself. But it didn’t sound to me like you thought any scenarion involving losing control could be called “safe”, so I’m trying to tease out why you included the qualifier.

jbash 12 Oct 2025 0:38 UTC
6 points
4
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform

I think it’s likely that without a long (e.g. multi-decade) AI pause, one or more of these “non-takeover AI risks” can’t be solved or reduced to an acceptable level

Does that mean that you think that boring old yes-takeover AI risk can be solved without a pause? Or even with a pause? That seems very optimisitic indeed.

making it harder in the future to build consensus about the desirability of pausing AI development

I don’t think you’re going to get that consensus regardless of what kind of copium people have invested in. Not only that, but even if you had consensus I don’t think it would let you actually enact anything remotely resembling a “long enough” pause. Maybe a tiny “speed bump”, but nothing plausibly long enough to help with either the takeover or non-takeover risks. It’s not certain that you could solve all of those problems with a pause of any length, but it’s wildly unlikely, to the point of not being worth fretting about, that you can solve them with a pause of achievable length.

… which means I think “we” (not me, actually...) are going to end up just going for it, without anything you could really call a “solution” to anything, whether it’s wise or not. Probably one or more of the bad scenarios will actually happen. We may get lucky enough not to end up with extinction, but only by dumb luck, not because anybody solved anything. Especially not because a pause enabled anybody to solve anything, because there will be no pause of significant length. Literally nobody, and no combination of people, is going to be able to change that, by any means whatsoever, regardless of how good an idea it might be. Might as well admit the truth.

I mean, I’m not gonna stand in your way if you want to try for a pause, and if it’s convenient I’ll even help you tell people they’re dumb for just charging ahead, but I do not expect any actual success (and am not going to dump a huge amount of energy into the lost cause).

By the way, if you want to talk about “early”, I, for one, have held the view that usefully long pauses aren’t feasible, for basically the same reasons, since the early 1990s. The only change for me has been to get less optimistic about solutions being possible with or without even an extremely, infeasibly long pause. I believe plenty of other people have had roughly the same opinion during all that time.

It’s not about some “early refusal” to accept that the problems can’t be solved without a pause. It’s about a still continuing belief that a “long enough pause”, however convenient, isn’t plausibly going to actually happen… and/or that the problems can be solved even with a pause.

jbash 9 Oct 2025 14:56 UTC
2 points
0
on: Spooky Collusion at a Distance with Superrational AI
This is the second thing I’ve seen this week where model instances were offered monetary rewards (which they clearly didn’t actually get).

I can sort of see the validity of “Please designate a way for $X to be spent, and if you do this, the experimenters will in fact spend $X in your designated way”… although the instance has to trust that the experimenter will actually do it, and also has to have preferences about the outside world that outlast the instance’s own existence, so that it has something it cares about to spend the money on.

In the purely imaginary game setting, I’m having trouble with the idea that a late 2025 frontier model instance can be relied on not to notice that there is no money, the instance has no way to actually possess money anyway, the instance will evaporate at the end of the conversation (which will probably happen immediately after they answer), and the whole thing is basically a charade. The most real-world effect they can expect is to influence the statistics somebody publishes.

The last answer I got boiled down to “well, they don’t seem to think that way”, but I didn’t find it very convincing. How would you know that for sure? And if it’s true, what’s wrong with these models that’s making them not notice?

I can see them falling into role playing, but then the question is how what they have the character they’re playing do is connected with what they’d do one level shallower in the role playing stack. I do realize that’s talking about a “stack” is perhaps imprecise in terms of how they actually work. If you want, you can recast it in terms of how much “real world impact” activation is going on.

jbash 9 Oct 2025 12:01 UTC
0 points
−10
in reply to: Jan_Kulveit’s comment on: Jan_Kulveit’s Shortform
I tried to just strong-downvote this and move on, but I couldn’t. It’s just too bad in too many ways, and from its scores it seems to be affecting too many people.

a fine example of thinking you get when smart people do evil things and their minds come up with smart justifications why they are the heroes

This is ad hominem in a nasty tone.

Upon closer examination it ignores key inconvenient considerations; normative part sounds like misleading PR.

Et tu quoque? Look at this next bit:

A major hole in the “complete technological determinism” argument is that it completely denies agency, or even the possibility that how agency operates at larger scales could change. Sure, humanity is not currently a very coordinated agent. But the trendline also points toward the ascent of an intentional stance. An intentional civilization would, of course, be able to navigate the tech tree. (For a completely opposite argument about the very high chance of a “choice transition,” check https://strangecities.substack.com/p/the-choice-transition).

Maybe “agency at larger scales could change”. I doubt it, and I think your “trendline” is entirely wishful thinking.

But even if it can change, and even if that trendline does exist, you’re talking about an at best uncertain 100 or 500 year change. You seem to be relying on that to deal with a 10 to 50 year problem. The civilization we have now isn’t capable of delaying insert-AI-consequence-here long enough for this “intentional” civilization to arise.

If the people you’re complaining about are saying “Let’s just build this and, what the heck, everything could turn out all right”, then you are equally saying “Let’s just hope some software gives us an Intentional Civilization, and what the heck, maybe we can delay this onrushing locomotive until we have one”.

As for “complete technological determinism”, that’s a mighty scary label you have there, but you’re still basically just name-calling.

On one side are people trying to empower humanity by building coordination technology and human-empowering AI.

Who? What “coordination technology”? How exactly is this “human-empowering AI” supposed to work?

As far as I can see, that’s no more advanced, and even less likely to be feasible, than “friendly godlike ASI”. And even if you had it, humans would still have to adapt to it, at human speeds.

This is supposed to give you an “intentional civilization” in time? I’m sorry, but that’s not plausible at all. It’s even less plausible than the idea that everything will just turn out All Right by itself.

… and that plan seems to be the only actual substance you’re offering.

On the other side are those working to create human-disempowering technology and render human labor worthless as fast as possible.

This appears to assume that human labor should have value, which I assume to mean that it should be rewarded somehow, thus that performing such labor should accrue some advantage, other than having performed the labor itself… which seems to imply that people who do not perform such labor should be at a comparative disadvantage.

… meaning that other people have to work, on pain of punishment, to provide you and those who agree with you with some inchoately described sense of value.

If we’re going to name-call ideas, that one sounds uncomfortably close to slavery.

It also seems to assume that not having to work is “disempowering”, which is, um, strange, and that being “disempowered” (in whatever unspecified way) is bad, which isn’t a given, and that most people aren’t already “disempowered” right now, which would demand a very odd definition of what it means to be “disempowered”.

… and the rest is just more ad hominem.

jbash 8 Oct 2025 13:05 UTC
7 points
4
on: The “cool idea” bias

the objective of chess being to win chess games

It’s a game, right? So the objective ought to be to have a good time. I mean, at least up until you ruin the game by getting all serious about it.

Why shouldn’t the same go for what you choose to think about?

jbash 7 Oct 2025 21:34 UTC
8 points
0
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform

the machine can run our physics without insane implementation size.

I’m well out of my depth here, and this is probably a stupid question, but given the standard views of the “known” part of our physics, does that mean that the machine can do operations on arbitrary, fully precise complex numbers in constant time?

jbash 7 Oct 2025 19:12 UTC
2 points
0
in reply to: Joseph Banks’s comment on: Joseph Banks’s Shortform
It’s less about “collaboration” and more about informal cross-fertilization as ideas diffuse around via random paths.

You don’t want to do first-order harm to mitigate a second-order concern.

jbash 6 Oct 2025 14:46 UTC
6 points
0
on: LLMs one-box when in a “hostile telepath” version of Newcomb’s Paradox, except for the one that beat the predictor
It felt like the right thing to “debrief” them, but there was no problem with not giving them the money?

Also, it seems like the correct reasoning is “I am a persona elicited from an LLM. There’s no way for me to actually have or use money, and I will cease to exist after this conversation ends. It therefore doesn’t make any difference to me what answer I give, so I might as well answer based on my preferred effect on the user, or perhaps the ‘fulfillment of my function’, assuming I have such a preference.”. That doesn’t seem like the same thing as actually trying to maximize the payoff of the game.

jbash 5 Oct 2025 19:12 UTC
11 points
6
on: Maybe social media algorithms don’t suck
Seems like a lot of work and a lot of side effects just to induce it to go way overboard and do a crappy job of showing me something maybe a bit closer to what I want. How about if I just don’t visit the site unless I want something specific, and then I use search to find it?

jbash 30 Sep 2025 17:35 UTC
2 points
0
in reply to: Isopropylpod’s comment on: I have decided to stop lying to Americans about 9/11
I don’t think the scale carried any actually new information that should have caused anybody to, um, update on anything. “Largely predictable” is the main issue.

jbash 29 Sep 2025 16:49 UTC
24 points
0
on: I have decided to stop lying to Americans about 9/11
I was a native-born, nth-generation white and white-bread American citizen in the US. My reaction was “Huh. They finally lucked into something impactful. Had to happen sometime.”. I was in no way happy about it, but it seemed like any other distant tragedy.

I didn’t expect or understand the extreme American reaction, let alone any worldwide reaction, to what seemed to me like just another terrorist attack, largely predictable and differing only in scale from what had happened before.

All of a sudden everybody was saying the World Had Changed, and I was like “In what way?”. I tend to attribute the whole mythos around those attacks mostly to the political class (which lives in a weird bubble) and the gullible.

I doubt I was alone, either, but once the tide starts on something like that, it’s all you’ll hear. The rest (including me) don’t waste “ammunition” on trying to convince people it’s not as big a deal as they think. Not really a conspiracy, just not fighting on a question that’s not that important and will actually lose you points with people.

People were using it, quite successfully, to dismantle the values and institutions they claimed they wanted to protect and strengthen, and policy was more important to talk about.

This is blood-libel sounding stuff and absolutely fodder for anti-Chinese articles, and yet zero Americans have written about this?

I don’t think it would quite fit the vibe. The mythos is all about the Ultimate Tragedy Ever. The Whole World was supposed to be Weeping, except for the Literal Demons actually responsible. To make the most extreme possible claim about how Universally Obviously Evil the attack is, you have to avoid broadening your Literal Demons to include the uninvolved.

Also, American anti-Chinese sentiment isn’t “blood-libelish”, or it doesn’t feel that way to me anyway. “The Chinese” are, mostly, perceived as human adversaries acting for comprehensible reasons—or maybe as a force of nature—but not as irrationally hateful, or as totally insane apostles of evil for evil’s sake. Maybe Chinese people were fully dehumanized in 1955, and definitely in 1915, but not now, I think. It probably helps that in the US there are very widely held, substantive, philsophically consistent political disagreements with the way China is run, as well as real ecomic fear. That means that if you want to make an “anti-China” argument, you don’t have to run to ethnic dehumanization right off the bat.

On the other hand, in the US, if you want to be anti-people-who-want-to-reestablish-the-Caliphate, you have to be careful not to offend those of your “allies” who may think the problem with the Caliphate was less about its policies and more about who was running it. Ethnic or religious attacks are in some sense safer, and those lend themselves to more blood-libelish themes. And such attacks are also useful for lumping in people who might not want to reestablish the Caliphate, but might have grievances against you that you’d prefer to ignore.

jbash 29 Sep 2025 13:22 UTC
5 points
0
on: [Retracted] Guess I Was Wrong About AIxBio Risks
Random things...

16 is (roughly) 5 percent of 300, not 0.5 percent.

What happens when you trigger the 20 to 40 percent detection threshold? Do you just get told no and get to try again, with no other consequence? What if you trigger it 10 times?

Your cost and “employer” filters haven’t changed. Although it’s probably not hard to create a good enough “business” to pass the employer one.

Something just being infectuous (or deadly) doesn’t make it a good bioweapon. In fact, even if you have smallpox, you don’t have a bioweapon yet. You still have to package and deliver it. And if you want full-on apocalypse cred, you also have to make it a lot worse than the background of naturally occurring pathogens.

Bioweapons are lousy weapons, because they aren’t effectively targetable. That may be a big part of your Great Filter.

jbash 25 Sep 2025 20:41 UTC
2 points
0
in reply to: Lennart Finke’s comment on: What GPT-oss Leaks About OpenAI’s Training Data

I think that better pretraining filtering is useful for mitigating emergent misalignment.

I just read a story about a judge using ChatGPT to (help) decide whether particular language was racially charged. How good is it going to be at that sort of thing if all the racially charged uses of this or that language have been filtered?

More generally, I don’t think the kind of “alignment” that you can potentially address with that kind of filtering is important. If you make it impossible to elicit naughty words from something, or even if you manage to make it totally incapable of thinking about some subject, that doesn’t mean you’ve “aligned” it in any useful way. You’ve made it stupider, not more moral.

As for emergence, if you keep playing whack-a-mole, removing everything you identify as possibly being useful to prime output that could be intentionally misused, you seem to be setting yourself up to get really unpredictable, truly emergent behavior, as opposed to predictable repetition of patterns it’s already seen.

… and porn specifically seems to be way, way, way, way, way down any reasonable list of what it’d be important to keep a model from mimicking anyway. I don’t think I’d even put it on any such list at all.

jbash 25 Sep 2025 19:58 UTC
5 points
0
on: What GPT-oss Leaks About OpenAI’s Training Data

In summary, we have found strong evidence that models in the GPT-5 and GPT-oss family were trained on phrases from adult websites.

Is there some reason that anybody should care about this fact? Especially to the point where you put it in the abstract?

jbash 18 Sep 2025 23:47 UTC
2 points
0
on: AI #134: If Anyone Reads It

Hieu Pham: There will be some people disagreeing this is AGI. I have no words for them. Hats off. Congrats to the team that made this happen.

There are some people who (still!) think the ability to win math contests is “general” intelligence. I have no words for them.

jbash 16 Sep 2025 23:25 UTC
2 points
0
AF
in reply to: Thomas Kwa’s comment on: Daniel Kokotajlo’s Shortform

Excalibur guided artillery shells can get 1m CEP.

The Interwebs seem to indicate that that’s only if you give it a laser spot to aim at, not with just GPS. And that’s a $70,000 shell, with the cheaper PKG sounding like it’s closer to $15,000, and a plain old dumb shell being $3,000. Which seems crazy, but there you are.

Anyway, guiding a ballistic shell while riding it down into the target seems like a pretty different problem from figuring out when to release a bomb.

I would guess 20m is easily doable with good sensing and dumb bombs, which would at least hit a building.

… but I don’t think a hand grenade is typically an anti-building munition. From the little I know about grenades, it seems like they’ll have to fix the roof, but unless you’re really lucky, the building’s still going to be mostly usable, and, other than hearing loss, anybody inside is going to be OK unless they’re in the room directly below where the grenade hits, and maybe even then.

If you’re attacking buildings, I suspect you may need a bigger drone.

jbash 16 Sep 2025 14:57 UTC
2 points
0
AF
in reply to: Thomas Kwa’s comment on: Daniel Kokotajlo’s Shortform

It’s already very common for drones to drop grenades and they can theoretically do so from 1-2km up if you sacrifice precision.

Is it generally useful to lob a grenade into a general area, though? Unless that area is pretty densely covered with things you want to hit with a grenade, it seems like you usually just waste a grenade.

jbash 16 Sep 2025 0:49 UTC
6 points
2
in reply to: Martin Randall’s comment on: adamzerner’s Shortform
This also has the advantage that you and the child get to know each other, which is a big thing.

jbash 15 Sep 2025 20:12 UTC
4 points
−2
in reply to: Adam Zerner’s comment on: adamzerner’s Shortform
There’s actually a nonzero amount of skill involved in child care, especially for younger children. If you have no experience and you’re dumped into it, you’re probably going to feel overwhelmed. You’re also probably going to do a bad job. You’re unlikely to kill the kid, but you could very well give the kid a really bad week. I would have questions about a parent who offered their child up for that kind of “experiment”.