Ishual

Karma: 391

Ishual 24 Oct 2025 21:13 UTC
3 points
0
in reply to: Garrett Baker’s comment on: Reasons to sign a statement to ban superintelligence (+ FAQ for those on the fence)
I think your advice is a quite clear articulation of the strategy behind the CAIS statement. I think this is a great and difficult strategy to pursue. Indeed, I have been circulating https://superintelligence-statement.org/ for some time now (secretly), and I think this statement first ensured it would get some of the major signatories before even making its way to me.

I think the OP is both talking about object-level reasons people gave, and it also shows that there is another strategy one might pursue which is to go around and convince people to actually support an existing statement publicly once they support it privately.

Ishual 16 Oct 2025 13:24 UTC
3 points
0
on: The Mom Test for AI Extinction Scenarios
- A convincing scenario cannot involve any bioweapons. Normal people just don’t know how vulnerable the human machine is. They think pandemics are just something that happens every 5-20 years, and don’t think about it besides that. They don’t think about the human body as a nano factory that’s vulnerable to targetted nano-attacks.
- A scenario that passes the mom test will also not include any drones. Yes, even though drones are currently used in warfare. Drones are the future. Drones are toys. Futuristic toys don’t sound like a realistic threat.
- A mom test scenario also shouldn’t involve any hacking. Regular people have no idea how insecure computer systems are. It’s basically safe to do online banking on a computer, which gives people the intuition that computers are mostly secure. Any story involving hacking violates that intuition.
I think there is a model of “normies” employed here which is a decent first approximation, but isn’t precise enough. I think for instance that all three above things can be make real to normies. I feel like the “decent first approximation” model of normies here present an “impossible” (in video game sense) problem. But the real problem is merely hard.

Here are some diffs between my and your model of normies:

1) I think most normies can be convinced that things that actually happened happened, even if the things sound very sci-fi/weird. It is worth thinking how to do this well. It isn’t always easy, but this is a great sanity check, can you actually convince normies of things that already happened? (the answer is yes, you just have to become good at it) In particular, if you can’t convince most normies that drones are a major factor in the ukraine war, if you can’t convince most normies that stuxnet and various other hacking ops was super impressive and scary, if you can’t convince most normies that the black death killed a lot of people and disrupted society for years, then you are making mistakes that take like 1 month of deliberate effort to fix at most.

2) what presents naively as skepticism of sci-fi stuff is actually something else. Specifically, I think most people (including normies) have a default where everything they see “makes sense” to them, and if they reflect about it, they have some explanation for the evidence they see. If someone has a pretty bad explanation for some evidence, your argument will not make sense to them. If you identify these places where the explanation for the evidence is misleading, you can dig there and find a different example which will have a different explanation, or use some other method to make your argument flow more explicitly than if you were making it to someone who had shared models for the observation you present.

3) normies (and most non-normies, but that is a different post lol) don’t run on arguments. They have intuitions that activate when you present some idea and that determines whether they agree or not. People are bad at verbalizing what the intuition is that is triggering, and there isn’t a direct mapping between what the intuition is and what they say when it triggers. An example of this is someone being skeptical that AI would be lethal, asking “is elon gonna program it to kill all <insert ingroup>s?” and then I said “AI is not programmed line-by-line, we just tweak it through trial and error until it achieves some measurable capabilities” and then they were like “holy shit, now I get it”. It is not obvious (at all) a priori that “I am skeptical that AI could hurt us” was linked to the “AI is just programmed software” intuition. And normies differ immensely in which intuitions they have. My default guess is that if it appears to you like “you must never mention X to normies if you want to convince them”, then probably you are faced with some intuitions that you have misidentified.

4) normies are a social species and implement a social epistemology. It isn’t fully accurate to say that normies don’t change their minds in isolation, but it does capture some real thing, which is that there is huge variance in how their intuitions work and they mostly change their minds as a group. If you have some kind of convincing short story that they feel they could repeat and justify-themselves/convince others with, then that will do most of the work. If you give them that first, then fully convincing them through “logic and facts” may just go smoother. Don’t neglect the 5 minute version of what you are saying. the 5 minute version mostly needs to be plausible and defensible. Also, social proof is important, finding someone they respect to acknowledge a topic as valid does a lot of work.

Ishual 15 Oct 2025 21:59 UTC
3 points
0
in reply to: Taylor G. Lunt’s comment on: The Mom Test for AI Extinction Scenarios
idea to make a thing that happened believable:
- stuxnet has a wikipedia page that is easy to point to. Pointing to things as a reflex sometimes works.
- there are perhaps some videos/books that are aimed at the general public, are somewhat entertaining for the general public, and have a visible view count of >1M.
- Maybe it was covered by one of the news channels that the general public person acknowledges as existing and a valid source of real things.

I think for the “scary hacks” category, it is worth coming up with 3-5 very different illustrative cases, and looking for ways to connect them to other things the person (likely) thinks are real.

I think it is worth doing the same with pandemics. (for instance there was the black death, there was the spanish flu, there was covid, which (depending on the politics of the person) was acknowledged as having been engineered)

I think cases of “going hard” by human groups are worth knowing about.

I think if some such cases are very useful for making the case that ASI could win in a fight against humanity, it would be worth first getting really good at establishing and discussing many such cases in a fun and believable way, and then once you succeed at enough of them to establish their existence, you can talk about how an ASI could pull these human-pullable levers.

Ishual 29 Sep 2025 16:18 UTC
3 points
0
in reply to: Vika’s comment on: Safety researchers should take a public stance
I think that signing sufficiently clear open letters/similar things seems sufficient to count as “taking a public stance”.

[Here is a first rough attempt at expressing an idea:]
I don’t think pushing for an AI pause now is what most people have in mind (definitely it doesn’t match what I had in mind when writing the post, keeping in mind that the post isn’t about what kind of public stance it would be effective for experts (inside and outside the frontier labs) to take). Instead, what matters imo is to have a legible thing (eg open letter/statement) that says clearly that you think a coordination to stop the current race dynamic would be good if feasible (because the current race has serious risks of extinction/disempowerment). Let me try to make the distinction (between that and the pause letter) clearer:

The CAIS statement “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” clearly establishes that extinction from AI is a thing many experts worry about. It was signed in 2023. It didn’t cry wolf. It is still useful today (probably more useful today than when it came out). If you need to convince a policy maker or a member of the general public that extinction is not a fringe concern, there is a clear statement you can quote, a simple link you can share, and a list of names that speaks for itself. In practice, this info is evergreen I think.

When a policy maker considers whether they should pay attention or discuss a “pause”, they have their concerns, and they are capable of worrying about whether this is “crying wolf” without our help. But maybe they wonder “would experts actually oppose me if I spoke about this?” or “do experts actually think some international coordination is required?” especially if they have experience with international coordination, I think they will suspect it is very hard to pull off, so they might just think that pushing for alignment project X to get marginally more funding is a better use of their time.

> I signed the FLI pause letter in 2023, but looking back it seems a bit premature.

I basically agree. But expressing the fact that plan A would be (a lot) better than plan B (along with some clarity on why) is useful even if it doesn’t cause an immediate shift from plan B to plan A. There is a strong sense in which expressing “plan A >> plan B, because xrisks” is a lot more your job as an expert than timing a pause. In other words, I think experts should make future cooperation easier rather than try to unilaterally create a shelling point.

if we are fully ignoring the internal pressures to not say such things within frontier labs, and purely focus on the efficacy of taking a public stance, I think there is a way to take a public stance without incurring the cost you mention above.

If in the future there is some serious discussion toward an international agreement of some kind, my guess is that it will have been made possible/[much easier] by individual people clearly and credibly expressing that they’d be on board with implementing such an agreement, and I think experts could be helpful here (by clearly expressing something like “plan A >> plan B, because xrisks”).

Ishual 22 Sep 2025 10:40 UTC
1 point
0
in reply to: Neel Nanda’s comment on: Safety researchers should take a public stance
This sounds to me like there would actually be specific opportunities to express some of your true beliefs that you wouldn’t worry would cost you a lot (and some other opportunities where you would worry and not do them). Would you agree with that?

Ishual 22 Sep 2025 10:37 UTC
4 points
1
in reply to: evhub’s comment on: Safety researchers should take a public stance
I agree that people should clearly state that they think there’s a catastrophic risk, but I disagree that people should clearly state that they think we should pause.
If we premise (as this post does) on the fact that the person we are talking about actually believes that an international ban would be a great improvement over the current mad AI race, then the above quote seems wrong to me.

I agree that experts should not pretend like they have more authority than they do in judging whether we should pause. But they could still say 1) that the race is insane, 2) that an international ban seems like a great improvement, 3) that if such a ban was proposed, they would not oppose it and 4) they would in fact support it. If not the experts, then who? To be clear, I don’t think the experts within the lab racing to build the tech are necessary here (this is not what the post is about). There are experts outside of the lab also (and they don’t have the [huge conflicts of interest]/pressure to filter(/falsify?) their speech). But if not the experts, then who would be better placed to say the above? if there is no one to say it, how does it get understood? if it doesn’t get understood, coordination to actually move out of the status quo towards some kind of international agreement is much harder. The CEOs of some of the lab could say it and that would definitely have an impact, but will they (lol)? Politicians could say it, but probably the backing of many experts would make this much easier for the politicians to say.

I think “there are catastrophic risks” is way too weak and doesn’t substitute. Partly because “there are catastrophic risk, so please give more money to me/so put me in charge/so we must beat those less careful folks” are also possible readings. I also happen to have it on very good authority that some politicians, when informed that many experts recognize the risks of extinctions and told the reasons why we should stop the mad AI race, will ask “but do the experts support stopping?” with perhaps a side of (“or do they just want more money for their thing?”)

Ishual 22 Sep 2025 10:00 UTC
8 points
3
in reply to: Zack_M_Davis’s comment on: Safety researchers should take a public stance
I probably should have said “norm execution” (ie follow the norm). This might just be a cultural gap, but I think norm enforcement/execution/implementation works in many ways that are not threats. For instance, there is pizza at a conference. there is a norm that you shouldn’t take all the pizza if there is a big line behind you. some people break this norm. what happens? do they get threatened? no! they just get dirty looks and people talking behind their backs. maybe they get the reputation as the “pizza taker”. In fact, nobody necessarily told them before this happened that taking all the pizza would break the norm.

I think there is a strange presumption that one is owed my and others’ maximum respect and friendship. anything less than that would be a “punishment”. that is pretty strange. if I have money in my pocket but I will only give some to you based on how many “good deeds” I have seen you do, this is not a threat. I guess that if you did not understand the motives or if the motives were actually to get a specific person to do more “good deeds” (by telling them in advance what the reward would be), you could call it a bribe. but calling it a threat is obviously incorrect.

I think norm enforcement/execution/implementation can and is in my case motivated by an aesthetic preference for “points” that are person A to give such as respect and friendship 1) not go to someone who does not deserve them (in my eyes) and instead 2) go to someone who does deserve them. It is not primarily driven by a consequentialist desire for more people to do respect-and-friendship-deserving things. It is primarily driven by a desire for the points to match reality, and thus enable greater cooperation and further good things down the line.

I realized based on a few comments that the three norms I discuss in the post were seen by some as like one giant strategy to produce more public stances from safety researchers. This is not the case. I am just talking to three different audiences and I explain a norm that I think makes sense (independently) for them.

Ishual 21 Sep 2025 19:44 UTC
1 point
0
in reply to: Neel Nanda’s comment on: Safety researchers should take a public stance
(optional: my other comment is more important imo)
I’m not concerned about someone being fired for this kind of thing, that would be pretty unwise on the labs’ part as you risk creating a martyr

I think you ascribe too much competence/foresight/focus/care to the labs. I’d be willing to bet that multiple (safety?) people have been fired from labs in a way that would make the lab look pretty bad. Labs make tactical mistakes sometimes. Wasn’t there a thing at OpenAI for instance (lol)? Of course it is possible(/probable?) that they would not fire in a given case due to sufficient “wisdom”, but we should not assign an extreme likelihood to that.

Ishual 21 Sep 2025 19:14 UTC
4 points
0
in reply to: Neel Nanda’s comment on: Safety researchers should take a public stance
Rather, I’m concerned about eg senior figures thinking worse of safety researchers as a whole because it causes a PR headache, eg viewing them as radical troublemakers, and this making theories of impact around influencing specific senior decision makers harder (and I’m more optimistic about those, personally)
Thank you Neel for stating this explicitly. I think this is very valuable information. This matches what some of my friends told me privately also. I would appreciate it a lot if you could give a rough estimate of your confidence that this would happen (ideally some probability/percentage). Additionally, I would appreciate if you could say whether you’d expect such a consequence to be legible/visible or illegible (once it had happened). Finally, are there legible reasons you could share for your estimated credence that this would happen?
(to be clear: I am sad that you are operating under such conditions. I consider this evidence against expecting meaningful impact from the inside at your lab.)

Ishual 21 Sep 2025 10:43 UTC
2 points
0
in reply to: Rohin Shah’s comment on: Safety researchers should take a public stance
I conclude from this that you really do see this post as a threat (also you admitted there is no threat in your first comment so this comment now seems contradictory/bad-faith).

some thoughts:
- this isn’t a threat by proxy and isn’t a threat (but if it were a tbp then it would be a t sure)
- I am in the “others” group. I implement the norm I endorse in the post, and I am not threatening you. I don’t want to sound dismissive but you are not giving me a lot to work with here, and it sounds to me like either 1) you have a vague model of what a threat is that includes things that aren’t threats or 2) you are misunderstanding the post and our intent such that you model us as having made a threat.
- we say what we think you should do as a safety researcher. not a threat. it is a recommendation.
- separately, we say how we think others should relate to safety researchers in a way that is more robust and functional. Maybe I should clarify that if safety researchers don’t take a public stance you find acceptable, you shouldn’t be sad that they “called your bluff” (because I don’t endorse you bluffing or threatening). You should not be doing this to change individual safety researchers actions. You should be doing this for the benefit of being less foolable and choosing where you put your respect/friendship in a way more functional for society and more beneficial for you. I would endorse this part of the norm even if not a single additional safety researcher took a public stance (heck, even if some of them tried to invert my preferences by removing their public stance, even then would I endorse this norm. partly because I endorse not giving in to actual threats, but also because it would still be a good norm to have on net).

Ishual 21 Sep 2025 10:14 UTC
3 points
0
in reply to: Neel Nanda’s comment on: Safety researchers should take a public stance
if it was genuinely all the authors of this post wanted then I suggest they write a different post
Leo’s statement is quite good without being all we wanted. (indeed, of the 3 things we wanted, 1 is about how we think it makes sense for others to relate to safety researcher based on what they say/[don’t say] publicly. and 1 is about trying to shift the lab’s behavior toward it being legibly safe for employees to say various things, which Leo’s comment is not about.) I internally track a pretty crucial difference between what I want to happen in the world (ie that we shift from plan B to plan A somehow) and how I believe people ought to relate to the public stance/[lack thereof] of safety researchers within frontier labs. I think there are maybe stronger stances Leo could have taken, and weaker ones, and I endorse having the way I relate/model/[act towards] Leo depend on which he takes. I think the public stance that would max lead to me maximally relating well to a safety researcher ought to be something like “I think coordinating to stop the race (even if in the form of some ban which I won’t choose the exact details of) would be better than the current race to ever more capable AI. I would support such coordination. I am currently trying to make the situation better in case there is no such coordination, but I don’t think the current situation is sufficiently promising to justify not coordinating. Also there is a real threat of humanity’s extinction if we don’t coordinate.” (or something to that effect)

Ishual 21 Sep 2025 9:49 UTC
6 points
0
in reply to: Rohin Shah’s comment on: Safety researchers should take a public stance
Seeing the post as a threat misses the intended point. It is important to state explicitly: The goal of the three norms argued for in the post was never to force people to publicly support something they don’t in fact believe in. It was also never to force people to be more honest about what they believe. The post explicitly says what we think you should be doing, so that there can be a discussion about it. But the norm enforcement part is about what we think others (who are not necessarily working at frontier labs) should be doing.

Separately, I am not sure I understood what you meant by “I aim to not give in to ‘norm enforcement’”, but it seems to me that there is a culture inside the labs that make many people working there uncomfortable taking a public stance. To be more explicit, does that also activate your will to not give in to ‘norm enforcement’? (if not, why not?)

> I think you may want to rethink your models of how norm enforcement works.

I didn’t get what you were trying to communicate here. Continuing to rethink (publicly) my models of how norm enforcement works is why we wrote this post on LW.

Ishual 20 Sep 2025 15:20 UTC
3 points
0
in reply to: leogao’s comment on: Safety researchers should take a public stance
Thank you. I endorse (and internally implement) the norm of respecting you based on this :)

Ishual 19 Jun 2025 16:05 UTC
1 point
0
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
One potential issue with “non-EA ideologies don’t even care about stars” is that in biological humans, ideologies don’t get transmitted perfectly across generations.

It might matter (a lot) whether [the descendent of the humans currently subscribing to “non-EA ideologies” who end up caring about stars] feel trapped in an “unfair deal”.

The above problem might be mitigated by allowing migration between the two zones (as long as the rules of the zones are respected). (ie the children of the star-dwellers who want to come back can do so unless they would break the invariants that allow earth-dwellers to be happy with perhaps some extra leeway/accommodation beyond what is allowed for native earth-dwellers and the children of earth-dwellers who want to start their own colony have some room to do so, reserved in the contract)

one potential source of other people’s disagreement is the following intuition: “surely once the star-dwellers expand, they will use their overwhelming power to conquer the earth.” Related to this intuition is the fact that expansion which starts out exponential will eventually be bounded by cubic growth (and eventually quadratic, due to gravitational effects, etc...) Basically, a deal is struck now in conditions of plenty, but eventually resources will grow scarce and the balance of power will decay to nothing by then.

Ishual 19 Jun 2025 15:40 UTC
1 point
0
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
I like the idea from Pretentious Penguin that, IIA might not be satisfied in general, but if you first get the agent to read A, B, C, and then offer {A,B} as options and {A,B,C} as options, (a specific instance of) IIA could be satisfied in that context.

You can gain info by being presented with more options, but once you have gained info, you could just be invariant to being presented with the same info again.

so you would get IIA*: “whether you prefer option A or B is independent of whether I offer you an irrelevant option C, provided that you had already processed {A,B,C} beforehand”

You can’t have processed all possible information at a finite time, so above is limited relative to the original IIA.

I also didn’t check whether you get additional problems with IIA*.

Ishual 19 Jun 2025 15:26 UTC
1 point
0
in reply to: Cleo Nardo’s comment on: Why I am not a successionist
Is there actually (only) a small number of moral worldviews?

My own moral worldview cares about the journey and has a bunch of preferences for “not going too fast”, “not losing important stuff”, “not cutting lives short”, “not forcing people to grow up much faster than they would like”. But my own moral worldview also cares about not having the destination artificially limited. From my vantage point (which is admittedly just intuitions barely held together with duck tape), it seems plausible that there is a set of intermediate preferences between MM and GG, somewhat well-indexed by a continuous “comfortable speed”. Here are some questions on which I think people might differ according to their “comfortable speed”:
- how far beyond the frontier are you entitled to remain and still live a nice life? (ie how much should we subsidize people who wait for the second generation of upload tech before uploading?)
- how much risk are you allowed to take in pushing the frontier?
- how much consensus do we require to decide the path forward to greater capabilities? (eg choosing which of the following is legitimate: genetic edits, uploading, or artificial intelligence)
- how much control do we want to exert over future generations?
- how much do you value a predictible future?
- how comfortable are you with fast change?

If my “comfortable speed” model is correct, then maybe you would want to assign regions of the lightcone to various preferences according to some gradient.

There can also be preferences over how much the present variance within humanity keeps interacting in the future.

Ishual 17 May 2025 11:31 UTC
1 point
0
on: An Advent of Thought
With this picture in mind, one can see that whether an endeavor is infinite depends on one’s measure — and e.g. if all you’re interested in in mathematics is finding a proof of some particular single theorem, then maybe “math” seems finite to you.
Informally and intuitively, securing the future (in a conflict) naturally induces a measure that makes math seem finite, but enjoying the future (after you have somewhat secured it) naturally induces a measure that makes math seem infinite.

Ishual 14 May 2025 13:08 UTC
1 point
0
on: An Advent of Thought
- If $E$ is an infinite endeavor, then “how should one do $E$ ?” is also infinite.^[18] For example: math is infinite, so “how should one do math?” is infinite; ethics is infinite, so “how should one do ethics?” is infinite.^[19]
- If $E$ is an infinite endeavor and there is a “faithful reduction” of $E$ to another endeavor $F$ , then $F$ is also infinite. (In particular, if an infinite endeavor $E$ is “faithfully” a subset of another endeavor $F$ , then $F$ is also infinite.)^[20] For example, math being infinite implies that stuff in general is infinite; “how should one do math?” being infinite implies that “how should one think?” is infinite.
- If an endeavor constitutes a decently big part of an infinite endeavor, then it is infinite.^[21]^[22] For example, to the extent that language is and will remain to be highly load-bearing in thinking, [figuring out how thinking should work] being infinite implies that [figuring out how language should work] is also infinite.
Important and useful, but also keep in mind:
a’. If “how should one do E?” is a finite endeavor then E is finite.
b’. If F finite, then any E that reduces to F must also be finite.
c’. If an endeavor contains not too much more than a finite endeavor, then it is finite.

perhaps “infinitude clusters”? perhaps “infinitudes don’t exist alone?” perhaps “infinitudes are not isolated?” perhaps “non-solitary infinitudes?”

Ishual 13 May 2025 13:23 UTC
2 points
0
on: An Advent of Thought
I want to preclude scenarios which look like doing a bunch of philosophy
A (perhaps related) pathology: a mind that, whenever it sees the symbol “0”, it must prove a novel theorem in ZFC in order to complete the perception of the symbol. For such a mind, many (but not all!) repetitive and boring tasks will induce an infinite endeavor (because the mind contains such an endeavor).

Ishual 13 May 2025 13:17 UTC
2 points
0
on: An Advent of Thought
I think it is a mistake to think of these as finite problems^[12] — they are infinite.
It is often possible to build multiple instances of a “big/small” dichotomy, such that the theory has similar results. (e.g. [small=”polynomial time”] vs [small=”decidable”]).

When I imagine using your concept to think about AI, I want to say that cloning a strawberry/proving the Riemann hypothesis is a finite task that nevertheless likely implies an endeavor of a different character than [insert something you could do manually in a few months].

I wonder if the [small=finite] version of your concept is the one we should be using.