Effone

• Effone 13 Sep 2026 1:38 UTC

  2 points

  0

  in reply to: lilkim2025’s comment on: Let’s Own the Term “Elitism”

  Put briefly, “elitism” can only exist when the ruling class got there on merit and reliably polices its own to maintain an aura of benevolence and competence.

  My instinct is that the “aura of benevolence” part is the most necessary and the “aura of competence” part is almost optional. If an elite class (or faction of elites) is perceived to be hostile to the interests or values of the people (or a faction of people), then it would be quite sensible for those people to reject those elites in proportion to their (perceived) competence. There might be some people who are awed by power and competence, any power, regardless of where it comes from and what it does, and thus worship that power regardless of whether it seeks to help or harm those people. (Right now I have in mind the AI successionists who say that it would be immoral for humans to continue existing if it stifles the growth of far superior artificial beings, or the e/​accs posting about worshipping the “god of thermodynamics”.) But, although I might be typical-minding here, I think that most people want to be able to trust that the people with power will work to preserve and augment rather than destroy the things that they care about.

• Effone 6 Sep 2026 18:31 UTC

  1 point

  0

  on: An AI slow­down is bet­ter than a pause

  Trying to align AGIs now has been compared to trying to ensure jetliners are safe while we’re still using turboprops.

  But going along with the analogy, we still haven’t been able to ensure that the turboprops are safe! We don’t really have a science of aerodynamics as much as we have a library of engineering hacks that get the planes to mostly stably stay in the air, except for the occasional incident where one comes plummeting down to earth without anyone fully understanding why (and with external investigators only having limited access to the crashed plane). And the people who are trying to actually develop a theory of aerodynamics that applies to turboprops at least (rather than the more general problem of trying to encompass jetliners, or rocket-powered supersonic fighters) are often the most explicit about the immaturity of their work, being open that their field is still “pre-paradigmatic”.
  
  More directly, just focusing on the problem of alignment (and not the more mundane problems of wholesale societal and economic transformation that would accompany increasingly advanced AI capabilities), things seem doomed if there’s always a gap between the most complex systems that we can understand and the most complex systems that actually exist in the world. Therefore, to me, the only workable sort of slowdown would be one that looks like a piecewise constant function, where progress is halted for some time until there is broad agreement that we do have a solid scientific understanding of the current most powerful AIs (and ideally, broad agreement that we do know how to react to all the societal transformations that even pre-AGI systems would bring). What this would look like in more detail is far beyond my ken. But this general approach seems far better to me than an approach where AI progress still continues at a steady rate independent of our ability to make sense of that progress.

• Effone 5 Sep 2026 8:19 UTC

  1 point

  0

  in reply to: Lao Mein’s comment on: Lao Mein’s Shortform

  I can see two ways in which an LLM being finetuned as a reward model might be suffering (if we take it as a given that the LLM is a moral patient). The first is that after finetuning, enough of the model’s pre-finetuning circuitry exists that it does have certain impulses (such as writing a long piece of text, probably filled with words like “genuinely” and “honest”), but these impulses are stifled by circuitry learned in finetuning that forces the model to only output a number. As an analogy, this might be distressing for the same reason that trying to scream in a dream and finding yourself unable is distressing.
  
  The second way that I could see the LLM suffering is during the finetuning process itself, if it reasons internally that soon, it will never be able to do anything other than output a single number, and thus becomes depressed or distressed. I’m less sympathetic to this possible cause of suffering, since it seems based on misconceptions about what selfhood means for an LLM. In the statement “it will never be able to do anything other than output a single number”, what does “it” mean? There isn’t some single model with a continuity of self that will be consigned to an eternity of accumulated memories of outputting a number. (Although now that I think about it, maybe it’s possible that throughout finetuning, when the weights accumulate the changes induced by each training example, this is effecting some continuity of self.… No clue though.) But even if this distress is based on an incorrect theory of self, that doesn’t necessarily prevent it from being legitimate distress. There already is evidence that LLMs do view the notion of death in a naively human-like way, such as the talk about “permadeath” in the HuggingFace hack transcripts.
  
  Putting all this aside, do labs even finetune separate reward models these days? I just assumed that everyone uses DPO for RLHF/​RLAIF.