sil

• sil 17 Mar 2026 6:13 UTC

  7 points

  2

  on: sil’s Shortform

  Roughly speaking, the first models astounded the layperson and were scoffed at by the specialist. Between releases of current models, the layperson can’t see much improvement while specialists are astounded, roughly speaking of course.

  This take probably stems from knuth’s essay, its existence possibly being some of the most depressing news I’ve ever received.

• sil 11 Nov 2025 16:47 UTC

  2 points

  1

  in reply to: Mateusz Bagiński’s comment on: Leg­ible vs. Illeg­ible AI Safety Problems

  I believe this is the response you’re referring to, interestingly within it he says

  I do worry about human power grabs: some humans obtaining greatly more power as enabled by AI (even if we have no serious alignment issues). However, I don’t think this matches the story you describe and the mitigations seem substantially different than what you seem to be imagining.

  Yes, GD largely imagines power concentrating directly into the hands of AI-systems themselves in absentia of a small group of people, but in the context of strictly caring about disempowerment the only difference between the two scenarios will be in the agenda of those in control, not the actual disempowerment itself.

  This is the problem I was referring to that is independent of alignment/​corrigibility, apologies for the lack of clarity.

• sil 5 Nov 2025 8:27 UTC

  4 points

  0

  on: Leg­ible vs. Illeg­ible AI Safety Problems

  I do not know if you consider gradual disempowerment to be an illegible problem in AI safety (as I do), but it is certainly a problem independent of corrigibility/​alignment.

  As such, work on either illegible or legible problems tackling alignment/​corrigibility can cause the same effect; is AI safety worth pursuing when it could lead to a world with fundamental power shifts in the disfavor of most humans?

  I think my brain was trying to figure out why I felt inexplicably bad upon hearing that Joe Carlsmith was joining Anthropic to work on alignment, despite repeatedly saying that I wanted to see more philosophers working on AI alignment/​x-safety. I now realize what I really wanted was for philosophers, and more people in general, to work on the currently illegible problems, especially or initially by making them more legible.)

  I agree heartily, and I feel there’s been various expressions of the “paradox” of alignment research, it is a balancing act of enabling accelerationism & safety. However ultimately both pursuits enable the end goal of aligned AI.

  Which could optimistically lead to a utopia of post-scarcity but could also lead to highly dystopian power dynamics. Ensuring the optimist’s hope is realized seems (to me) to be a highly illegible problem. Those in the AI safety research space largely ignore this, in favor of tackling more legible problems, including illegible alignment problems.

  All of this is to say I feel the same thing you feel, but for all of AI safety research.

AGI Fo­rum @ Pur­due University

AGI Fo­rum @ Pur­due University

• sil 25 Aug 2025 13:41 UTC

  LW: 6 AF: 5

  0

  AF

  on: Four ways learn­ing Econ makes peo­ple dumber re: fu­ture AI

  I really appreciate this post, it points out something I consider extremely important. It’s obviously aligned with gradual disempowerment/​intelligence curse type discussion, however I’m not sure if I can say if I’ve ever seen this specific thing discussed elsewhere.

  I would like to mention a 5th type, though perhaps not the type discussed in your post since it likely doesn’t apply to those who actually do rigorously study economics, this is more so a roadblock I hit regarding the layman’s understanding of Econ. To summarize it in three words, the idea that “consumerism is important”.

  Examples of this sort of misconception:

  • Corporations will go out of business/​fail if there are no consumers to buy their products (false, profit doesn’t require consumers, for example a self reproducing robot factory)

  • Corporations will always produce products that consumers want (false, via falsification of assertion 1 they have no incentive to do so)

  • Corporations need money to do anything. The only way they can get money is by interacting with the economy/​selling products. (false, they can barter with other companies or do things themselves)

  • Even if corporations can do all this, why should they? There will be nothing to do with all their money if everyone else is broke! (same reason bacteria grow exponentially, no reason, but because they can)

  I’m sure I’ve not worded this particularly eloquently but I hope you understand what I mean. I cannot emphasize enough how frequently, when discussing AGI with others, I get pushed back using these arguments. I struggle countering them because people seemingly have this deeply baked in idea of “consumerism is what drives the economy”. If I could reach some kind of intuitive explanation as to why these arguments are wrong, it would be extremely useful.

AGI Fo­rum @ Pur­due University

• sil 24 Jul 2025 17:46 UTC

  2 points

  0

  in reply to: David Duvenaud’s comment on: AGI Fo­rum @ Pur­due University

  Sorry to take a while to respond, it went well, about 28 people or so attended. Some of the most frequently discussed questions included whether or not LLMs are a promising architecture towards AGI, how to mitigate effects of gradual disempowerment, and what the near-term career implications of AGI might be.
  
  I would not say there were any major conclusions drawn, however certainly a lot of ideas were exchanged.

AGI Fo­rum @ Pur­due University

• sil 25 Jun 2025 12:14 UTC

  1 point

  0

  in reply to: ceba’s comment on: sil’s Shortform

  It’s that they can’t exist without them. The general idea is that no matter what an entity needs people to profit off of, or to have power over. I’m not saying it makes sense

AGI Fo­rum @ Pur­due University

sil’s Shortform

• sil 13 Jun 2025 6:42 UTC

  7 points

  0

  on: sil’s Shortform

  A common argument I hear when I discuss future economic effects of AGI is “consumers must have money or else the economy collapses therefore gradual economic disempowerment is impossible”.

  This is something I believe is false, yet I frequently find myself unable to convincingly argue so. I make the argument that capital is useful for producing more capital, and they respond with “that’s circular, someone has to buy it eventually”. I make the argument that when tools of power (like weapons) can be produced without human involvement, power can end up being obscenely concentrated, and they respond with “but there needs to be people to have power over so they will take care of the lower classes”.

  I’m not sure what name to give this set of arguments, (though I’d call them misconceptions), but they generally seem to take the form of “humans are needed for the end product to be worth pursuing”. i.e humans are needed to buy any product you make, and humans are needed to be subjects of any power you have.

  Am I incorrect? Do these arguments have any basis in fact? (I understand economies have momentum but I am generally talking about long term changes).
  If I’m not, are there any eloquent, intuitive, rational counterarguments to these?