Unfortunately, many others have disagreed, and the public debate is nowhere near settled.
It would be great to have a debate society that was fun to watch, facilitates expressing positions clearly and in their strongest / most relevant forms, and makes the disagreements actually face each other. (Some speculations here, but mainly just someone iterating on hosting lots of debates and building tools to help debaters find truth either together or adversarially.
(I don’t have much to say but I’d generically note: I’d suspect significant gains from medium-depth theorization here. Like, my speculative guess is: There’s various dynamics and resources and stuff at play, which are fairly obvious if you sit down to write them out; and then if you think about various one- or two-step inferences from those basics, you get interesting novel ideas about how to structure media platforms. Examples: attention, boosting, piggybacking, replying, affiliations, trust. Etc. Then ask, what is “a LW post” or “a LW comment” or “a LW quicktake” or “a tweet” or “a tweet reply” or “a QT” or “a substack post” or etc etc in these terms? What are the affordances and natural patterns of attention and information processing associated with each one? What space of possible such [media-chunk roles] does that suggest?)
As an additional point, there are also potentially quite significant negative externalities to slotting in—because you validate the strategic vision, which makes it able to pick up other slotters. (Cf. https://tsvibt.blogspot.com/2024/09/the-moral-obligation-not-to-be-eaten.html )
I think there’s a middle ground statement:
While the outcomes that drive up the expected dollars are mainly in the regime where your utility is linear in dollars, you should bet crazily; after that point, you should be hedging more; Kelly works (uniquely?) well in the long-run; but in the real world you could know what utility regime you’re in / how long of a run you’re in or something like that.
Yeah I had a convo about Kelly a few years ago where (I may be misremembering who said what) I was like “But my utility is basically linear in money, or in some sense it’s even significantly superlinear (because it would be easier to get a bunch of information and connections and good investments in a snowbally way), so I obviously should just bet as much as I can if I really think I have an edge.”. I think this is basically right, but not that relevant in practice just because there aren’t really opportunities like that unless you really go looking for them, and it’s much better to have some money even if you do have a safety net.
I do generally find myself more concerned about genetic engineering for extreme intelligence than other people in this cluster seem to be.
Tangent of course, but happy to discuss, whether in private or on a podcast.
I suggest that another underlying source of disagreement could be about the general factor of: what is this function, approximately:
[the similarity of all previous tries put together, to try X]
--> [how much should we expect to fail on try X on the grounds that it’s first-try-ish, i.e. how much is it Murphy’s Cursed]
If you think that even a fair amount of similarity still doesn’t get you to success on one-shot problems, then you’d talk about oneshotness as being a strong argument against AGI alignment attempts working out well. This kinda sounds like what Yudkowsky is arguing in this post. Someone else could disagree with that.
They may think it’s critical but not very firsty, i.e. sufficiently similar to / comparable to / generalizable from previous tries.
Another underlying disagreement could be about the general factor of: what is this function, approximately:
[the similarity of all previous tries put together, to try X]
--> [how much should we expect to fail on try X on the grounds that it’s first-try-ish, i.e. how much is it Murphy’s Cursed]
I would imagine that this isn’t the main source of disagreement, but I do find it hard to see how [create an alien mind that’s smarter than humanity] ends up less firsty than [make a Mars rover] etc., so I’m wondering if that’s not the claim.
I very clearly said that in my comment… Anyway, I guess there’s nothing to discuss here, I’m just saying that abortability is a relevant dimension to these scenarios. It’s something that’s brought up often, and also it bears on first-try-ness. If there is a situation that is akin to the eventual critical first try, but is abortable, then that would imply that when you get the eventual critical try, it doesn’t have to be your first try. There’s a nontrivial argument to make about “when it’s abortable, it’s not akin enough to the eventual critical try”.
I’m not actually sure exactly what “critical” means here. I’m taking it to just mean “you absolutely must get this try right”. That’s closely connected to abortability, in that if you can abort, it’s not fully lethal / critical yet. I don’t think it’s really the same thing, e.g. you could imagine an LLM-based bacterial package (a more complex “computer virus”) that permanently lives on many computer systems and is basically impossible to abort (short of scouring the planet of all computers with more than 16 GB of memory or whatever).
There’s whether or not you get to try again after your first try, and there’s how late in the game you can decide to not fully do the try at all. There’s at least 3 kinds of outcomes:
You abort (don’t fully do the try).
You do the try and succeed.
You do the try and fail (and can’t try again).
Because unaligned AGI is lethal, you don’t get to try again.
You kids need to turn off the car.
Right, another dimension to these scenarios is abortability. At some point, we cross out of technically feasible abortability—we (humans) wouldn’t be able to abort the AI’s growth even if we tried. Whether things are abortable before then depends on how humans react over time / new information (e.g. heeding arguments, heeding warning shots, being credulous about apparent alignment, etc.).
Well, one can point concretely at the exampled but not climaxed possibility of being better at putting ideas into words: https://www.lesswrong.com/posts/7RFC74otGcZifXpec/the-possible-shared-craft-of-deliberate-lexicogenesis
(Speaking for myself of course)
But what counts as a first try?
A given try is more firsty if it’s less like all previous tries put together. A one-shot problem is one where you try a pretty firsty try, and that try is likely to kill humanity.
How many previous tries are in the same class (e.g. small asteroid / big asteroid, or GPT2 / GPTN) is relevant in that a priori more such tries might suggest that future tries are less firsty. But it’s also perfectly plausible a priori to have lots of tries that you survive, and then a one-shot problem (lethal and very firsty).
You could even have a series of one-shot problems. Imagine for example that you have a lethal asteroid—but you saw it 10 years in advance, and it’s small enough that you can stop it with nukes. It’s one-shot (lethal and firsty), but maybe you survive. Then you have another similar asteroid, but you only saw it 6 months in advance—that might be another one-shot problem (do it all again but way faster). Then you have an asteroid that’s so big, all the nukes in the world wouldn’t stop it. One-shot again; there’s totally new, crucial challenges to solve.
I think that with AI, you very likely get a one-shot problem in the ballpark of superhuman AGI. It’s lethal, in that it would by default go on and extinct humanity, and very firsty, in that many core alignment difficulties first show up there.
I guess one thing I’m saying is that one can have any given beliefs without taking the Slighted affiliations/attitudes too far. It’s not a mere choice, it’s a skill, potentially a big one (like lots of subskills, like “programming” is a skill), that one could learn over time.
(Sorry, your post seems pretty interesting but I don’t have enough spare bandwidth; I’ll just note that those things you list here sound of course related to Slight but being Slighted is always a kind of choice you’re making, similar to being Consensus, and neither is that great—they both have advantages and disadvantages. They’re social and attitudinal attractors to some extent, but far from absolute.)
Yeah that’s fair.
My wild guess would be that there is a significant local persuasion overhang in the sense that if someone set up an RL context (like, any scaled up persuasive context, e.g. bots on various social platforms with clear feedback), there would be fast gains, maybe to the point of being really disruptive in certain classes of contexts. (There is another theory which states that this has already happened.) I think you’d then hit an asymptote below being relevant to most important contexts.
(Because today’s systems would not be able to follow along with how humans change their stances in response to things like this. For example, image generation can easily fool people, but for most people there would only be a brief period during which that person would send money based mainly on receiving a realistic image which, if real, would make them want to send money. They’d just learn to not do that.)
A Slighted might view a Consensus as sanctimonious, cowardly, conformist, overconfident, boring, virtue signaling, envious, self-deceiving, power-thirsty (in the sense of wanting to be in control of arbitrary social consensus), wanting to cut down tall poppies, arbitrary / not truth seeking, getting sucked into a locked-in equilibrium, tribalistic.
A Consensus might view a Slighted as delusional, aggressive, contrarian, overconfident, wanting to feel special, power-thirsty (in the sense of wanting to avoid social or legal accountability), needing to be put in their place / needing to be socialized / needing moral correction / needed to have values transmitted to them, overly focused on the perceived Slight, having a big ego about their contrarian positions and not updating, getting sucked into spirals of networks of misinformation, reckless, uncaring, tribalistic, paranoid/conspiratorial.
It’s an interesting question. Some thoughts:
Generally, cognition is aimed at solving problems, so we are drawn to think of downside risks and conflicts.
As a second order effect, conflict and downsides are more memetically fit. I can easily name media imagining bad results of reprogenetics (Gattaca, Brave New World, Sandel, Habermas, Fukuyama, etc. etc.); harder to name media imagining good results.
As Kaarel notes, genuine utopias are also alien, and our real values largely route through developmental processes (learning more, understanding more, reflecting more, empathizing more, regularizing through childish understanding, etc.).
As a corollary of the previous point, contra the Anna Karenina principle, there’s something that’s much more relatable about pain, suffering, failure, problems, death, conflict, and bad outcomes generally. There’s many ways to fail, but they all have stuff in common. In contrast, consider free creation of a life in general. Or as a metonymy, consider the free creation of a piece of music. There are so many degrees of freedom and an infinite range of structures to express in the music. Cf. Fun theory https://www.lesswrong.com/w/fun-theory.
Hope is painful. Thus, the infinite hopes of children are worn away with age; and for an adult, regaining hope would be painful.
Cf. “Border Guards” by Greg Egan. https://www.gregegan.net/BORDER/Complete/Border.html