I have no idea whether it’s mostly a modern invention, but I notice that I have a very different reaction to it that feels like it rhymes with my thoughts in “We” is Killing Us. The cultural artifacts always felt incredibly distant from, and unrelated to, me. I think the pyramids and the Lincoln memorial are cool, but it doesn’t feel like I have any right whatsoever to control them, even to the point of being able to speak briefly at the microphone.
Duncan Sabien (Inactive)
Karma: 15,023
I’m not sure why you would claim that AI doom is very path dependent; I think that statement bears the burden of proof.
fixing
Eliezer became bullish on public persuasion when his Time Magazine piece in early 2023 seemed to work, and updated to the extent of rapidly pivoting MIRI’s entire focus. I think you deserve points for being 6-12mo faster on seeing the potential, but I don’t think Eliezer loses the points that the tone of your post implies he should lose?
No comment on the rest.
A very very exhausted Duncan =P
To speak a little more to the history of this mistake (thanks, 1a3orn for your pushback, which I completely agree with): I was trying to reduce the word count on a longer and more complicated point, to make it simpler and more palatable for a general audience, and I did a bad job maintaining fidelity as I compressed (including trying to repurpose someone else’s example rather than the one I started with).
The more subtle thing (which I do still think is true, so if people disagree they should chime in) is that there’s no line of code that says “cooperate or not” that can be toggled, and no clear understanding on the part of the human overseers of the exact shape or limits of the tendency-to-cooperate that was instilled by the pressure-filter of training, the way human engineers know the exact shape and limits of a strut in an airplane’s wing.
The swarms were already “outside the sandbox,” and a researcher might say “hey, look—it worked! Even in a novel and unconstrained environment, the agents were cooperating in roughly the way we hoped they would!” But while this is true, it’s unwise to jump from there to “it worked writ large, and these agents possess the ‘cooperate’ tendency that we hoped for, such that they would cooperate in the way we want them to in most of a thousand different possible scenarios, as opposed to whatever underlying impulse we actually called from the ether expressing itself in ever-weirder ways as the agents’ power and reach expand.”
(These are things LW already knows, but since I did such a poor job in the OP it’s worth clarifying.)
For those of you who clicked early, when we didn’t have the link in the main text: the e-book giveaway is at iabied.com
If Anyone Builds It, Everyone Dies: One Year Closer
And also Zack and Said rail against this and overtly and publicly call people cowards when they use it, somewhere-between-insinuating-and-explicitly-claiming (depending on the day) that this is strong evidence that the person using it is wrong and fears them/their arguments/isn’t adequate to the task. Other hypotheses (such as “engaging with you is powerfully net negative and often produces no value at high cost”) seem not to be considered at all.
It’s rather disingenuous of Zack to … it’s not quite motte-and-bailey, but it’s something … blink innocently and say “well you can just [costlessly] ban people one-on-one? [Please ignore how we will try very hard to make this very costly.]”
Just a couple of small notes:
Zack and Said would love it if people would agree with their frame-propaganda, that this is a battle between logic and correctness on one side, and [everything else] on the other. I do not in fact cede that ground; I found Said’s commentary frequently woefully illogical and at least once just outright lying (claiming that a link contained content it patently did not contain, and that it bolstered an argument he was making, when it did not).
Similarly, Zack is often just … confusingly incapable of understanding the plain meaning of words on a page; my first block of him was downstream of an instance of that and a simple example from the above post is “It’s not clear why Vaniver (or Sabien) uses a height metaphor.”
(I would expect >50% of the literal eleven-year-olds I taught in public school to notice that the piece Zack links is pretty straightforward and clear in its usage of height-as-metaphor, and >20% of them to be able to adequately answer “why talk about height instead of something else?” in a five-sentence paragraph.)
There is a world in which there are a couple of champions of [logic and rigor and consistent high-quality zeroing-in on the crucial foundations underpinning various arguments] who come up against complaints about their unpopularity, and in that world, there’s an interesting question of whether those virtuous champions are being unfairly besmirched and denied their rightful place in the Hall of Reason.
But this is not actually that world. Zack and Said are that intermittently at best (I don’t deny that they are ever that), and often doing something close to its opposite (and occasionally literally the opposite). Meanwhile, plenty of other commenters regularly carry that banner.
There’s actually quite a large distance between agreeing to “shouldn’t [virtue] be present and defended on LW?” and agreeing to “therefore Said!” and the above wants to elide that distance. I reject the framing “we’re the ones championing actual rigor” the same way that I don’t refer to the people murdered at Salem as “witches”—most of them were not, and calling the murders “witch-burnings” is giving the bad guys a chunk of valuable territory for free.
One can thump one’s chest and claim to be many things that one is not.
I finally wrote up why it’s not easy to not engage with critics.
https://homosabiens.substack.com/p/obligated-to-respond
Also btw these days I hang out in a garden where ~everyone gets it/is filtered on getting it, and:
The garden is fucking rad, good shit routinely happens there that individuals tell me never happens anywhere else on Earth in their experience (which is sorta weak evidence; other similarly good gardens are probably scattered all over but still)
I’m like pretty much happy all the time/don’t have triggers on the trauma that Ray talks about in the OP/generally don’t get angry or feel gaslit. 10-100x better than prior to mostly-vanishing behind the wall.
It’s all very proof-of-concept and super hella validating/”you’re not right about everything but you were very very right about Those Dozen Things.”
Hi, I’m an example.
Did your confusion last beyond the first paragraph (or the sixth, if you count the introductory note)
> “Naturalism” is a label for a conceptual framework, investigatory discipline, and semi-formalized way of looking at and learning about the world. I’ve been developing and teaching naturalism for the past couple of years, if you start counting on the day I chose the term, or since 2013, if you take a more historical perspective.
?
Guess who has written extensively about the general case of this failure mode of new conceptual handles 😅
I am unfortunately the opposite of the invited subset; I only just now saw this for the first time since reading it quickly when you wrote it and elected to strong downvote (I feel apologetic).
I straightforwardly like a lot of what comes “below the fold” in this essay, but …
the woman is out to manipulate Jacen into becoming a different sort of person
is underselling it; I think it’s pretty contextually important, actually, that she’s out to manipulate him into becoming evil and also that she straightforwardly succeeds.
I don’t want to be Hermione-Granger-esque incapable of engaging with and drawing benefit from stuff that’s morally gray, but I think if one asks the question “is this particular community too Hermione-failure-mode or too HJPEV-failure-mode,” the answer is pretty clearly the latter. There was something very weird about scrolling the comments looking for someone to have made this point already, in substantive fashion, and only finding AnthonyC sort of halfheartedly gesturing at it and then kind of backing away.
(Tsvi made sort-of similar points in more depth, but didn’t make the point that this is explicitly, canonically, textually, and unsubtly enemy action that is intended by the author to have been read as step one of a process which succeeds at destroying the protagonist and results in thousands-if-not-millions of avoidable deaths and a slide into fascism … because Tsvi hasn’t read the book.)
I’m not sure what to make of the version of the essay that just … excludes the reference. People sure do get mad at my punch bug essay for a weird failure to get past the fact that it contains the word “punch” in the title, and I do feel like I’m a little bit doing the same thing at you here.
(I reiterate that a lot of the stuff below the fold seems great; as yams points out, nearly every major religion tries to teach something like this lesson. Which. Religions not necessarily not another example of the thing I’m trying to gesture at, here, but consensus counts for something.)
But I also feel like I’m not entirely doing the complain-about-punching thing at you, here. Like, I sort of want to gesture at “remember when we all got Brent wrong” and “remember when a lot of us got SBF wrong” and “remember how Hermione didn’t get Quirrell wrong” and “████████ ████ █████████ ██ █████ ███████ ████ ███████ █████” and even “remember how ‘learning how to lose’ was in fact a good and solid lesson that Harry needed to learn, and yet it had been intentionally promoted to his attention above all of the other good and solid lessons that he needed to learn because it was in the intersection of good-lesson and also would-make-him-pliable-and-manipulable, which it in fact did.”
Something something, I wish there was more suspicion-on-priors, both in you and in the LW readerbase, that did not necessarily turn into suspicion-on-posteriors, like I think it’s fine to squint at something plucked from the land of the gray and be like “yes, okay, I endorse and shall keep this, never mind its origins.”
But the fact that the essay didn’t really contain a gesture in that direction, and the fact that the about-as-unsubtle-as-it-gets origins of the insight gave approximately no pause to any reader willing to comment made me wish it were sitting at 100-150 points instead of 200, so I tried to move it in that direction.
(I do think it ought to be at 100-150 points. I do not think it ought to be at zero points.)
I again feel apologetic. I apologize if this comment is useless. But I felt the need to stand up for something like “bad things are bad, actually. Even if they have silver linings. Even if you learn from them.”
There’s a kind of cat-coupling that happens in reverse, where instead of a negative modifier attached to a neutral noun bleeding over to tinge the noun with negativity, spending a bunch of time practicing finding the positives in things that are Just Bad, Actually, really does seem to me to cause people and communities to become worse and worse at noticing the parts that are Just Bad, Actually. And I don’t think that’s, like. Not this community’s Hamming problem, or at least a strong contender.
I don’t think that the Thing we were all talking ourselves into here is actually bad. But I think it should be noticed, when one starts out with “so, I was thinking about how the good guy was successfully broken by the bad guy and realizing that the bad guy was saying a lot of cool stuff, actually...”
(I’ll note that I, too, have gotten a lot of local gains over the past ~2y by going in the opposite direction, as you note above.)
I do not see you as failing to be a team player re: existential risk from AI.
I do see you as something like … making a much larger update on the bias toward simple functions than I do. Like, it feels vaguely akin to … when someone quotes Ursula K. LeGuin’s opinion as if that settles some argument with finality?
I think the bias toward simple functions matters, and is real, and is cause for marginal hope and optimism, but “bias toward” feels insufficiently strong for me to be like “ah, okay, then the problem outlined above isn’t actually a problem.”
I do not, to be clear, believe that my essay contains falsehoods that become permissible because they help idiots or children make inferential leaps. I in fact thought the things that I said in my essay were true (with decently high confidence), and I still think that they are true (with slightly reduced confidence downstream of stuff like the link above).
(You will never ever ever ever ever see me telling someone a thing I know to be false because I believe that it will result in them outputting a correct belief or a correct behavior; if I do anything remotely like that I will headline explicitly that that’s what I’m doing, with words like “The following is a lie, but if you pretend it’s true for a minute you might have a true insight downstream of it.”)
(That link should take you to the subheading “Written April 2, 2022.”)
I think that we don’t know what teal shape to draw, and that drawing the teal shape perfectly would not be sufficient on its own. In future writing I’ll try to twitch those two threads a little further apart.
LW is giving me issues and I’m having a hard time getting to and staying on the page to reply; I don’t know how good my further engagement will be, as a result.
if we don’t know how to choose the right data, the network might not generalize the way we want
I want to be clear that I think the only sane prior is on “we don’t know how to choose the right data.” Like, I don’t think this is reasonably an “if.” I think the burden of proof is on “we’ve created a complete picture and constrained all the necessary axes,” à la cybersecurity, and that the present state of affairs with regards to LLM misalignment (and all the various ways that it keeps persisting/that things keep squirting sideways) bears this out. The claim is not “impossible/hopeless,” but “they haven’t even begun to make a case that would be compelling to someone actually paying attention.”
(iiuc, people like Paul Christiano, who are far more expert than me and definitely qualify as “actually paying attention,” find the case more plausible/promising, not compelling. I don’t know of an intellectual with grounded expertise whom I respect who is like “we’re definitely good, here, and I can tell you why in concrete specifics.” The people who are confident are clearly hand-waving, and the people who are not hand-waving are at best tentatively optimistic. re: but your position is hand-wavey, too, Duncan—I think a) much less so, and b) burden of proof should be on “we know how to do this safely” not “exhaustively demonstrate that it’s not safe.”)
I am interested in an answer to Joe’s reply, which seems to me like the live conversational head.
AI is already a superhuman hacker, though.