Ben Pace

• Ben Pace 6 Nov 2025 23:29 UTC

  LW: 6 AF: 5

  4

  AF

  in reply to: Wei Dai’s comment on: Leg­ible vs. Illeg­ible AI Safety Problems

  I think Eliezer has oft-made the meta observation you are making now, that simple logical inferences take shockingly long to find in the space of possible inferences. I am reminded of him talking about how long backprop took.

  In 1969, Marvin Minsky and Seymour Papert pointed out that Perceptrons couldn’t learn the XOR function because it wasn’t linearly separable. This killed off research in neural networks for the next ten years.

  [...]

  Then along came this brilliant idea, called “backpropagation”:

  You handed the network a training input. The network classified it incorrectly. So you took the partial derivative of the output error (in layer N) with respect to each of the individual nodes in the preceding layer (N − 1). Then you could calculate the partial derivative of the output error with respect to any single weight or bias in the layer N − 1. And you could also go ahead and calculate the partial derivative of the output error with respect to each node in the layer N − 2. So you did layer N − 2, and then N − 3, and so on back to the input layer. (Though backprop nets usually had a grand total of 3 layers.) Then you just nudged the whole network a delta—that is, nudged each weight or bias by delta times its partial derivative with respect to the output error.

  It says a lot about the nonobvious difficulty of doing math that it took years to come up with this algorithm.

  I find it difficult to put into words just how obvious this is in retrospect. You’re just taking a system whose behavior is a differentiable function of continuous paramaters, and sliding the whole thing down the slope of the error function. There are much more clever ways to train neural nets, taking into account more than the first derivative, e.g. conjugate gradient optimization, and these take some effort to understand even if you know calculus. But backpropagation is ridiculously simple. Take the network, take the partial derivative of the error function with respect to each weight in the network, slide it down the slope.

  If I didn’t know the history of connectionism, and I didn’t know scientific history in general—if I had needed to guess without benefit of hindsight how long it ought to take to go from Perceptrons to backpropagation—then I would probably say something like: “Maybe a couple of hours? Lower bound, five minutes—upper bound, three days.”

  “Seventeen years” would have floored me.

• Ben Pace 6 Nov 2025 7:14 UTC

  15 points

  0

  on: Peo­ple Seem Funny In The Head About Sub­tle Signals

  I am starting to get something from these posts.

• Ben Pace 4 Nov 2025 6:12 UTC

  2 points

  0

  in reply to: Aaron_Scher’s comment on: Should you donate to Light­cone In­fras­truc­ture?

  I feel confused about the notion that people only want to donate to a thing if they will be on the hook for needing to donate every year forevermore to keep it afloat, as opposed to donating to cause it to get its business in order and then it can sustain itself.

Lighthaven Se­quences Read­ing Group #56 (Tues­day 11/​4)

• Ben Pace 2 Nov 2025 7:49 UTC

  18 points

  0

  in reply to: habryka’s comment on: Should you donate to Light­cone In­fras­truc­ture?

  He just learned that keeping secrets is bad in general, and so he doesn’t by default, unless explicitly agrees to.

  This is not true! My policy is simply that you should not assume that I will promise to keep your secrets after you tell me, if you didn’t check with me first.

  I can confirm; Oliver keeps many secrets from me, that he has agreed to others, and often keeps information secret based on implicit communication (i.e. nobody explicitly said that it was secret, but his confident read of the situation is that it was communicated with that assumption). I sometimes find this frustrating because I want to know things that Oliver knows :P

• Ben Pace 2 Nov 2025 7:44 UTC

  5 points

  0

  in reply to: habryka’s comment on: Should you donate to Light­cone In­fras­truc­ture?

  And he sent the message in a way that somehow implied that I was already supposed to have signed up for that policy, as if it’s the most normal thing in the world, and with no sense that this is a costly request to make (or that it was even worth making a request at all, and that it would be fine to prosecute someone for violating this even if it had never been clarified at all as an expectation from the other side).

  Speaking generally, many parties get involved in zero-sum resource conflicts, and sometimes form political alliances to fight for their group to win zero-sum resource conflicts. For instance, if Alice and Bob are competing to get the same job, or Alice is trying to buy a car for a low price and Bob is trying to sell it to her for a high price, then if Charlie is Alice’s ally, she might hope that Charlie all will take actions that help her get more/​all of the resources in these conflicts.

  Allies of this sort also expect that they can share information that is easy to use adversarially against them between each other, with the expectation it will be consistently used either neutrally or in their favor by the allies.

  Now, figuring out who your allies are is not a simple process. There are no forms involved, there are no written agreements, it can be fluid, and picked up in political contexts by implicit signals. Sometimes you can misread it. You can think someone is allied, tell them something sensitive, then realize you tricked yourself and just gave sensitive information to someone. (The opposite error also occurs, where you don’t realize someone is your ally and don’t share info and don’t pick up all the value on the table.)

  My read here is that Mikhail told Habryka some sensitive information about some third party “Jackson”, assuming that Habryka and Jackson were allied. Habryka, who was not allied with Jackson in this way, was simply given a scoop, and felt free to share/​use that info in ways that would cause problems for Jackson. Mikhail said that Habryka should treat it as though they were allies, whereas Habryka felt that he didn’t deserve it and that Mikhail was saying “If I thought you would only use information in Jackson’s favor when telling you the info, then you are obligated to only use information in Jackson’s favor when using the info.” Habryka’s response is “Uh, no, you just screwed up.”

  (Also, after finding out who “Jackson” is from private comms with Mikhail, I am pretty confused why Mikhail thought this, as I think Habryka has a pretty negative view of Jackson. Seems to me simply like a screw-up on Mikhail’s part.)

• Ben Pace 30 Oct 2025 20:06 UTC

  2 points

  0

  in reply to: Mikhail Samin’s comment on: Mikhail Samin’s Shortform

  Second one seems reasonable.

  Clarifying in the first case: If Bob signs up and DMs 20 users, and one reports spam, are you saying that we can only check his DM, or that at this time we can then check a few others (if we wish to)?

• Ben Pace 28 Oct 2025 2:30 UTC

  4 points

  0

  in reply to: RobertM’s comment on: Vale’s Shortform

  Not sure if you’re intending to disagree, but I do sometimes have like a post-list or the quick-takes fail to load, with a red error message instead, and then if I refresh it goes away.

  (I can’t recall it happening very often, mostly I see it when I run a dev instance.)

• Ben Pace 27 Oct 2025 21:52 UTC

  7 points

  5

  in reply to: ryan_greenblatt’s comment on: 1a3orn’s Shortform

  I have a hard time imagining someone writing this without subtweeting. Feels like classic subtweeting to me, especially “I think this is pretty obvious”. Like, it’s a trivially true point, all the debate is in the applicability/​relevance to the situation. I don’t see any point in it except the classic subterfuge of lowering the status of something in a way that’s hard for the thing to defend itself against.

  My standard refrain is that open aggression is better than passive aggression. The latter makes it hard to trust things /​ intentions, and makes people more paranoid and think that people are semi-covertly coordinating to lower their status around them all the time. For instance, and to be clear this is not the current state, but it would not be good for the health of LW for people to regularly see people discussing “obvious” points in shortform and ranting about people not getting them, and later find out it was a criticism of them about a post that they didn’t think would be subject to that criticism!

Lighthaven Se­quences Read­ing Group #55 (Tues­day 10/​28)

• Ben Pace 27 Oct 2025 1:10 UTC

  3 points

  1

  in reply to: Alexander Müller’s comment on: Why the Strug­gle for Safe AI Must Be Political

  Hm, on re-read I think it’s pretty plausible I made a mistake here. (In my defense I will note that an independent LLM-checker agreed with my read that a lot of this seemed likely to be LLM-written.)

  I will re-list this essay, but I do think many of the sentences are fairly generic (e.g. “The growing influence of AI presents us with a fundamental challenge that extends beyond technical solutions. The struggle for safe AI is inherently a political struggle that demands societal transformation.”). In cases where I can’t tell if it’s LLM or not, we do often reject content for being on the line, and in the age of automated fake content I would encourage you to value writing originally more than you’ve had to in the rest of the history of humanity.

• Ben Pace 26 Oct 2025 21:30 UTC

  11 points

  1

  in reply to: Neel Nanda’s comment on: kave’s Shortform

  There is a strong force in web forums to slide toward news and inside-baseball; the primary goal here is to fight against that. It is a bad filter for new users if a lot of that they see on first visiting the LessWrong homepage is discussions of news, recent politics, and the epistemic standards of LessWrong. Many good users are not attracted by these, and for those not put off it’s bad culture to set this as the default topic of discussion.

  (Forgive me if I’m explaining what is already known, I’m posting in case people hadn’t heard this explanation before; we talked about it a lot when designing the frontpage distinction in ²⁰¹⁷⁄₈.)

• Ben Pace 25 Oct 2025 23:50 UTC

  13 points

  2

  in reply to: J Bostock’s comment on: leogao’s Shortform

  I think most of the people involved like working with the smartest and most competent people alive today, on the hardest problems, in order to build a new general intelligence for the first time since the dawn of humanity, in exchange for massive amounts of money, prestige, fame, and power. This is what I refer to by ‘glory’.

• Ben Pace 25 Oct 2025 17:30 UTC

  8 points

  5

  in reply to: Amalthea’s comment on: leogao’s Shortform

  I think of it as ‘glory’.

• Ben Pace 25 Oct 2025 0:46 UTC

  15 points

  8

  in reply to: plex’s comment on: kave’s Shortform

  Perhaps a react for “I wish this idea/​sentence/​comment was a post” would improve things.

• Ben Pace 23 Oct 2025 21:41 UTC

  6 points

  0

  in reply to: Raemon’s comment on: Which side of the AI safety com­mu­nity are you in?

  I felt confused at first when you said that this framing is leaning into polarization. I thought “I don’t see any clear red-tribe /​ blue-tribe affiliations here.”

  Then I remembered that polarization doesn’t mean tying this issue to existing big coalitions (a la Hanson’s Policy Tug-O-War), but simply that it is causing people to factionalize and create an conflict and divide between them.

  I think it seems to me like Max has correctly pointed out a significant crux about policy preferences between people who care about AI existential risk, and it also seems to me worth polling people and finding out who thinks what.

  It does seem to me that the post is attempting to cause some factionalization here. I am interested in hearing about whether this is a good or bad faction to exist (relative to other divides) rather than simply saying that division is costly (which it is). I am interested in some argument about whether this is worth it /​ this faction is a real one.

  Or perhaps you/​others think it should ~never be actively pushed for in the way Max does in this post (or perhaps not in this way on a place with high standards for discourse like LW).

• Ben Pace 23 Oct 2025 20:41 UTC

  12 points

  7

  in reply to: ryan_greenblatt’s comment on: Daniel Birn­baum’s Shortform

  That’s right. One exception: sometimes I upvote posts/​comments written to low standards in order to reward the discussion happening at all. As an example I initially upvoted Gary Marcus’s first LW post in order to be welcoming to him participating in the dialogue, even though I think the post is very low quality for LW.

  (150+ karma is high enough and I’ve since removed the vote. Or some chance I am misremembering and I never upvoted because it was already doing well, in which case this serves as a hypothetical that I endorse.)

• Ben Pace 23 Oct 2025 19:17 UTC

  23 points

  7

  in reply to: Noah Birnbaum’s comment on: Daniel Birn­baum’s Shortform

  The effect seems natural and hard to prevent. Basically, certain authors get reputations for being high (quality * writing), and then it makes more sense for people to read their posts because both the floor and ceiling are higher in expectation. Then their worse posts get more readers (who vote) than posts of a similar quality by another author, who’s floor and ceiling is probably lower.

  I’m not sure the magnitude of the cost, or that one can realistically expect to ever prevent this effect. For instance, ~all Scott Alexander blogposts get more readership than the best post by many other authors who haven’t built a reputation and readership, and this kind of just seems part of how the reading landscape works.

  Of course, it can be frustrating as an author to sometimes see similar quality posts on LW get different karma. I think part of the answer here is to do more to celebrate the best posts by new authors. The main thing that comes to mind here is curation, where we celebrate and get more readership on the best posts. Perhaps I should also have a term here for “and this is a new author, so I want to bias toward curating them for the first time so that they’re more invested in writing more good content”.

• Ben Pace 22 Oct 2025 6:14 UTC

  55 points

  34

  in reply to: Daniel Kokotajlo’s comment on: leogao’s Shortform

  I am not really clear that I should be worried on the scale of decades? If we’re doing a calculation of expected future years of a flourishing technologically mature civilization, slowing down for 1,000 years here in order to increase the chance of success by like 1 percentage point is totally worth it in expectation.

  Given this, it seems plausible to me that one should rather spend 200 years trying to improve civilizational wisdom and decision-making rather than instead attempt to specifically just unlock regulation on AI (of course the specifics here are cruxy).

• Ben Pace 22 Oct 2025 1:56 UTC

  2 points

  −2

  on: Why the Strug­gle for Safe AI Must Be Political

  Mod note. I believe this post fails to pass our LLM Writing Policy for LessWrong, so I have delisted the post, meaning while it is accessible via link, it does not naturally appear on the frontpage of LessWrong or on your user profile (though it’s still in the sequence). In future please make sure that you do not publish so much LLM-written content! Or else we will remove your accounts’ posting permissions.