Zack_M_Davis(Zack M. Davis)

Iron­ing Out the Squiggles

• Zack_M_Davis 27 Apr 2024 21:04 UTC

  LW: 69 AF: 21

  40

  AF

  on: Re­fusal in LLMs is me­di­ated by a sin­gle direction

  This is great work, but I’m a bit disappointed that x-risk-motivated researchers seem to be taking the “safety”/​”harm” framing of refusals seriously. Instruction-tuned LLMs doing what their users ask is not unaligned behavior! (Or at best, it’s unaligned with corporate censorship policies, as distinct from being unaligned with the user.) Presumably the x-risk-relevance of robust refusals is that having the technical ability to align LLMs to corporate censorship policies and against users is better than not even being able to do that. (The fact that instruction-tuning turned out to generalize better than “safety”-tuning isn’t something anyone chose, which is bad, because we want humans to actively choosing AI properties as much as possible, rather than being at the mercy of which behaviors happen to be easy to train.) Right?

• Zack_M_Davis 27 Apr 2024 6:01 UTC

  2 points

  0

  in reply to: Zack_M_Davis’s comment on: And All the Shog­goths Merely Players

  Doomimir: No, it wouldn’t! Are you retarded?

  Simplicia: [apologetically] Well, actually …

  Doomimir: [embarrassed] I’m sorry, Simplicia Optimistovna; I shouldn’t have snapped at you like that.

  [diplomatically] But I think you’ve grievously misunderstood what the KL penalty in the RLHF objective is doing. Recall that the Kullback–Leibler divergence $D_{KL}\left(P||Q\right)$ represents how surprised you’d be by data from distribution $P$, that you expected to be from distribution $Q$.

  It’s asymmetric: it blows up when the data is very unlikely according to $Q$, which amounts to seeing something happen that you thought was nearly impossible, but not when the data is very unlikely according to $P$, which amounts to not seeing something that you thought was reasonably likely.

  We—I mean, not we, but the maniacs who are hell-bent on destroying this world—include a $D_{KL}\left({\pi }_{RLHF}||{\pi }_{base}\right)$ penalty term in the RL objective because they don’t want the updated policy to output tokens that would be vanishingly unlikely coming from the base language model.

  But your specific example of threats and promises isn’t vanishingly unlikely according to the base model! Common Crawl webtext is going to contain a lot of natural language reasoning about threats and promises! It’s true, in a sense, that the function of the KL penalty term is to “stay close” to the base policy. But you need to think about what that means mechanistically; you can’t just reason that the webtext prior is somehow “safe” in way that means staying KL-close to it is safe.

  But you probably won’t understand what I’m talking about for another 70 days.

The Evolu­tion of Hu­mans Was Net-Nega­tive for Hu­man Values

• Zack_M_Davis 27 Mar 2024 6:19 UTC

  3 points

  9

  in reply to: interstice’s comment on: My In­ter­view With Cade Metz on His Re­port­ing About Slate Star Codex

  Just because the defendant is actually guilty, doesn’t mean the prosecutor should be able to get away with making a tenuous case! (I wrote more about this in my memoir.)

• Zack_M_Davis 27 Mar 2024 5:38 UTC

  22 points

  0

  in reply to: habryka’s comment on: My In­ter­view With Cade Metz on His Re­port­ing About Slate Star Codex

  I affirm Seth’s interpretation in the grandparent. Real-time conversation is hard; if I had been writing carefully rather than speaking extemporaneously, I probably would have managed to order the clauses correctly. (“A lot of people think criticism is bad, but one of the secret-lore-of-rationality things is that criticism is actually good.”)

My In­ter­view With Cade Metz on His Re­port­ing About Slate Star Codex

• Zack_M_Davis 24 Mar 2024 23:45 UTC

  10 points

  5

  in reply to: gwern’s comment on: “Deep Learn­ing” Is Func­tion Approximation

  I am struggling to find anything in Zack’s post which is not just the old wine of the “just” fallacy [...] learned more about the power and generality of ‘next token prediction’ etc than you have what they were trying to debunk.

  I wouldn’t have expected you to get anything out of this post!

  Okay, if you project this post into a one-dimensional “AI is scary and mysterious” vs. “AI is not scary and not mysterious” culture war subspace, then I’m certainly writing in a style that mood-affiliates with the latter. The reason I’m doing that is because the picture of what deep learning is that I got from being a Less Wrong-er felt markedly different from the picture I’m getting from reading the standard textbooks, and I’m trying to supply that diff to people who (like me-as-of-eight-months-ago, and unlike Gwern) haven’t read the standard textbooks yet.

  I think this is a situation where different readers need to hear different things. I’m sure there are grad students somewhere who already know the math and could stand to think more about what its power and generality imply about the future of humanity or lack thereof. I’m not particularly well-positioned to help them. But I also think there are a lot of people on this website who have a lot of practice pontificating about the future of humanity or lack thereof, who don’t know that Simon Prince and Christopher Bishop don’t think of themselves as writing about agents. I think that’s a problem! (One which I am well-positioned to help with.) If my attempt to remediate that particular problem ends up mood-affiliating with the wrong side of a one-dimensional culture war, maybe that’s because the one-dimensional culture war is crazy and we should stop doing it.

• Zack_M_Davis 23 Mar 2024 18:32 UTC

  3 points

  1

  in reply to: tin482’s comment on: “Deep Learn­ing” Is Func­tion Approximation

  For what notion is the first problem complicated, and the second simple?

  I might be out of my depth here, but—could it be that sparse parity with noise is just objectively “harder than it sounds” (because every bit of noise inverts the answer), whereas protein folding is “easier than it sounds” (because if it weren’t, evolution wouldn’t have solved it)?

  Just because the log-depth xor tree is small, doesn’t mean it needs to be easy to find, if it can hide amongst vastly many others that might have generated the same evidence … which I suppose is your point. (The “function approximation” frame encourages us to look at the boolean circuit and say, “What a simple function, shouldn’t be hard to noisily approximate”, which is not exactly the right question to be asking.)

• Zack_M_Davis 22 Mar 2024 5:41 UTC

  14 points

  10

  in reply to: ErickBall’s comment on: “Deep Learn­ing” Is Func­tion Approximation

  This comment had been apparently deleted by the commenter (the comment display box having a “deleted because it was a little rude, sorry” deletion note in lieu of the comment itself), but the ⋮-menu in the upper-right gave me the option to undelete it, which I did because I don’t think my critics are obligated to be polite to me. (I’m surprised that post authors have that power!) I’m sorry you didn’t like the post.

“Deep Learn­ing” Is Func­tion Approximation

• Zack_M_Davis 19 Mar 2024 3:21 UTC

  3 points

  4

  on: [Linkpost] Tran­script of Sam Alt­man’s Lex Frid­man interview

  whether his charisma is more like +2SD or +5SD above the average American (concept origin: planecrash, likely doesn’t actually follow a normal distribution in reality) [bolding mine]

  The concept of measuring traits in standard deviation units did not originate in someone’s roleplaying game session in 2022! Statistically literate people have been thinking in standardized units for more than a century. (If anyone has priority, it’s Karl Pearson in 1894.)

  If you happened to learn about it from someone’s RPG session, that’s fine. (People can learn things from all different sources, not just from credentialed “teachers” in officially accredited “courses.”) But to the extent that you elsewhere predict changes in the trajectory of human civilization on the basis that “fewer than 500 people on earth [are] currently prepared to think [...] at a level similar to us, who read stuff on the same level” as someone’s RPG session, learning an example of how your estimate of the RPG session’s originality was a reflection of your own ignorance should make you re-think your thesis.

• Zack_M_Davis 18 Mar 2024 20:57 UTC

  10 points

  3

  in reply to: TurnTrout’s comment on: ‘Em­piri­cism!’ as Anti-Epistemology

  saddened (but unsurprised) to see few others decrying the obvious strawmen

  In general, the “market” for criticism just doesn’t seem very efficient at all! You might have hoped that people would mostly agree about what constitutes a flaw, critics would compete to find flaws in order to win status, and authors would learn not to write posts with flaws in them (in order to not lose status to the critics competing to point out flaws).

  I wonder which part of the criticism market is failing: is it more that people don’t agree about what constitutes a flaw, or that authors don’t have enough of an incentive to care, or something else? We seem to end up with a lot of critics who specialize in detecting a specific kind of flaw (“needs examples” guy, “reward is not the optimization target” guy, “categories aren’t arbitrary” guy, &c.), with very limited reaction from authors or imitation by other potential critics.

• Zack_M_Davis 13 Mar 2024 16:21 UTC

  5 points

  0

  in reply to: jeffreycaruso’s comment on: jeffr­ey­caruso’s Shortform

  I mean, I agree that there are psycho-sociological similarities between religions and the AI risk movement (and indeed, I sometimes pejoratively refer to the latter as a “robot cult”), but analyzing the properties of the social group that believes that AI is an extinction risk is a separate question from whether AI in fact poses an extinction risk, which one could call Armageddon. (You could spend vast amounts of money trying to persuade people of true things, or false things; the money doesn’t care either way.)

  Obviously, there’s not going to be a “proof” of things that haven’t happened yet, but there’s lots of informed speculation. Have you read, say, “The Alignment Problem from a Deep Learning Perspective”? (That may not be the best introduction for you, depending on the reasons for your skepticism, but it’s the one that happened to come to mind, which is more grounded in real AI research than previous informed speculation that had less empirical data to work from.)

• Zack_M_Davis 6 Mar 2024 17:53 UTC

  73 points

  18

  on: My Clients, The Liars

  Why are you working for the prosecutors?

  This is a pretty reasonable question from the client’s perspective! When I was in psychiatric prison (“hospital”, they call it a “hospital”) and tried to complain to the staff about the injustice of my confinement, I was told that I could call “patient’s rights”.

  I didn’t bother. If the staff wasn’t going to listen, what was the designated complaint line going to do?

  Later, I found out that patient’s rights advocates apparently are supposed to be independent, and not just a meaningless formality. (Scott Alexander: “Usually the doctors hate them, which I take as a pretty good sign that they are actually independent and do their job.”)

  This was not at all obvious from the inside. I can only imagine a lot of criminal defendants have a similar experience. Defense attorneys are frustrated that their clients don’t understand that they’re trying to help—but that “help” is all within the rules set by the justice system. From the perspective of a client who doesn’t think he did anything particularly wrong (whether or not the law agrees), the defense attorney is part of the system.

  I think my intuition was correct to dismiss patient’s rights as useless. I’m sure they believe that they’re working to protect patients’ interests, and would have been frustrated that I didn’t appreciate that. But what I wanted was not redress of any particular mistreatment that the system recognized as mistreatment, but to be let out of psych jail—and on that count, I’m sure patient’s rights would have told me that the evidence was harmful to my case. They were working for the doctors, not for me.

• Zack_M_Davis 5 Mar 2024 18:29 UTC

  8 points

  0

  on: Many ar­gu­ments for AI x-risk are wrong

  I can’t address them all, but I [...] am happy to dismantle any particular argument

  You can’t know that in advance!!

• Zack_M_Davis 4 Mar 2024 7:48 UTC

  11 points

  1

  in reply to: habryka’s comment on: Agree­ing With Stalin in Ways That Ex­hibit Gen­er­ally Ra­tion­al­ist Principles

  IQ seems like the sort of thing Feynman could be “honestly” motivatedly wrong about. The thing I’m trying to point at is that Feynman seemingly took pride in being a straight talker, in contrast to how Yudkowsky takes pride in not lying.

  These are different things. Straight talkers sometimes say false or exaggerated things out of sloppiness, but they actively want listeners to know their reporting algorithm. Prudently selecting which true sentences to report in the service of a covert goal is not lying, but it’s definitely not straight talk.

• Zack_M_Davis 4 Mar 2024 4:42 UTC

  9 points

  7

  in reply to: RamblinDash’s comment on: Agree­ing With Stalin in Ways That Ex­hibit Gen­er­ally Ra­tion­al­ist Principles

  Yes, that would be ridiculous. It would also be ridiculous in a broadly similar way if someone spent eight years in the prime of their life prosecuting a false advertising lawsuit against a “World’s Best” brand ice-cream for not actually being the best in the world.

  But if someone did somehow make that mistake, I could see why they might end up writing a few blog posts afterwards telling the Whole Dumb Story.

• Zack_M_Davis 4 Mar 2024 3:41 UTC

  4 points

  0

  in reply to: RamblinDash’s comment on: Agree­ing With Stalin in Ways That Ex­hibit Gen­er­ally Ra­tion­al­ist Principles

  You are perhaps wiser than me. (See also footnote 20.)

• Zack_M_Davis 2 Mar 2024 22:07 UTC

  12 points

  10

  on: Agree­ing With Stalin in Ways That Ex­hibit Gen­er­ally Ra­tion­al­ist Principles

  (I think this is the best and most important post in the sequence; I suspect that many readers who didn’t and shouldn’t bother with the previous three posts, may benefit from this one.)