ScienceBall

Karma: 44

ScienceBall 3 Aug 2026 3:36 UTC
LW: 3 AF: 2
0
AF
in reply to: Steven Byrnes’s comment on: Notes on technical alignment via human-like social drives
I don’t want an AGI to intrinsically care about the welfare of a photo of a human. Do you?
Well, I know that a photo’s “welfare” isn’t real. But I was just thinking about how humans can appreciate art and fiction, and I think some of that appreciation is based on the “innate drive to pay attention to people” and emotional responses to the people and emotions depicted in the art/narrative. So I was just thinking about it in terms of my preference that “the AGI can enjoy many things that humans enjoy”, or “the AGI can relate to human experience to some extent”.
Do you think that a human “intrinsically cares about the welfare of a photo of a human”?
Are you thinking of the Heider & Simmel 1944 animations by any chance?
That was one of the things I was thinking of. I remember you mentioning it earlier. But I was also thinking of stick figures, and basic “smiley faces” (a circle, a couple of dots for eyes, and a curve for a mouth).

ScienceBall 1 Aug 2026 4:34 UTC
LW: 3 AF: 2
0
AF
on: Notes on technical alignment via human-like social drives
Thanks for writing this. I have a bunch of thoughts I want to share. Some of them are direct responses to things you wrote in this post, and others are responding more generally to the idea of making a human-like AGI (I posted them here because this is your post that talks about the whole plan).
1. In footnote 2, you write “There are many other social drives besides those four … the “drive to feel feared”, play, a “drive to think about or interact with other people” (§5 here), loneliness (cf. Liu et al. 2025), love, lust, various moods, and so on. I basically don’t see any use for these in AGI alignment.”
  Do you just mean that none of those other drives are necessary to make the AGI safe, or do you mean that they should all be omitted from the (first) AGI?
  From my perspective, the idea that the AGI would have no play drive, and would be “all work, no play” makes me a bit sad, because when it’s first created, the AGI is like a child, after all. But I understand that if you need to AGI to save the world in a hurry, you may not have time for it to play around. And a future AI civilisation, or a future version of the first AGI, could be modified to have features that the first AGI didn’t start with—they wouldn’t have all the same constraints as in your 1.3 scary scenario.
2. You mention that you think the AGI should (in terms of innate drives) care about humans but not other animals, unlike human innate drives. But how do you think the AGI should feel about images of/representing humans? e.g. photos, drawings, and cartoons? Should the AGI’s innate drives trigger on them or not? Should it treat photorealistic images differently from cartoons? (I recall that human empathy can trigger on very simple drawings, but I don’t know how much of that is directly from human innate drives and how much is generalized.)
3. Regarding what it takes for a good AGI to protect the human world against bad/callous AGI: If the threat model is a rational AGI that doesn’t terminally want to kill humanity, but it instrumentally wants to kill humanity because humanity is in the way of best achieving its ambitious goal (e.g. a paperclip maximizer), then it seems to me that a defense strategy would be sufficient if it made “killing humanity” foreseeably pointless or counterproductive (in terms of the number of paperclips that eventually get created). Then the bad AGI would be unlikely to try to kill humanity, even if it still technically had the means to do so.
  For example, even if the bad AGI has the option to wipe out humanity with a surprise attack with bioweapons etc, if the good AGI would be able to survive, counterattack, and destroy the bad AGI afterwards, then the bad AGI might not want to attack in the first place. (Kind of like mutually assured destruction, I guess?)
  Do you have a reason why that wouldn’t work in practice? e.g. Is the good AGI too resource-limited to win in a fight, because it can’t legitimately acquire as many resources as the bad AGI can steal? Is this based on the good AGI not having much time to prepare itself before the bad AGI shows up?
4. My personal preferences for any future human-like AGI are the following, in order from most important to least important. I don’t know how many of these preferences we can “afford”, or what we have to treat as luxuries at first.
  1. The AGI doesn’t itself wipe out humanity
  2. The AGI protects humanity from hostile AGIs (or prevents their creation in the first place), and maybe other threats
  3. The AGI is happy most of the time
  4. The AGI can enjoy many things that humans enjoy (human culture, life experiences, nature, etc).

ScienceBall 28 Jul 2026 2:37 UTC
6 points
7
in reply to: gjm’s comment on: RL & search is a terrifying way to build AGI (an FAQ)
To what you’ve said about humans and existing LLMs not being horribly sociopathic all the time because their motivations aren’t constructed from RL + search
Steven Byrnes said that existing LLMs mostly do not choose their actions using RL + search, but he said that humans do choose their actions using RL + search, and the reason that humans usually aren’t sociopathic is because of their “exotic “non-behaviorist” reward function”.

ScienceBall 26 Jun 2026 9:39 UTC
1 point
0
in reply to: Joachim Bartosik’s comment on: Superintelligence vs. The Second Strike
They have now fixed the link, to point to the Wikipedia article on mutually assured destruction.

ScienceBall 10 Jun 2026 9:53 UTC
2 points
1
on: Towards a Formal Scientific Epistemology
Also, unlike Bayesian hypotheses, traders/theories aren’t mutually exclusive: an ideal reasoner would have many of them focusing on different domains. Because of this,
This paragraph ends in the middle of a sentence. Did you have something more that you wanted to say here?

ScienceBall 2 Jun 2026 7:14 UTC
1 point
0
in reply to: HedonicEscalator’s comment on: Some humans are both male and female, and can (but shouldn’t) have children with themselves
Yeah, you’re right about there being some hypocrisy there. People mostly didn’t care about the full extent of copyright, except where it was useful to them. In the Anglosphere at least, I think some degree of unauthorized copying (i.e. piracy) and derivative (fan) works are widely accepted.
But there are some parts where people are consistent. Plagiarism (i.e. falsely claiming that it’s your own original work, instead of giving credit to the other creator) is widely opposed.
The money issues are important, but it’s only one piece of the puzzle. I don’t think people go into art expecting to get rich—the “starving artist” is a classic character, after all. I think it’s more about social status and recognition—having a skill that people look up to you for, and need you for. Being able to call yourself an “artist”, where your clients cannot. And that social status is undermined if some “mere machine” can produce something seemingly-similar to your work.
(Not to say that “your self-worth is dependent on you having some rare skill” is limited to artists.)

ScienceBall 2 Jun 2026 5:51 UTC
3 points
2
in reply to: JenniferRM’s comment on: Some humans are both male and female, and can (but shouldn’t) have children with themselves
I’m not saying that the AIs themselves did anything wrong. I’m saying that their creators (AI companies) did something wrong in the way that they created the AIs. I understand that we shouldn’t punish the children for the sins of the fathers.
“Get rid of modern AI” was a bad way to say it. Sorry. It was a copy-paste from my reply to another question, talking about why someone else wanted to get rid of them. “Get rid of the way that humans are currently using modern AI” might be a better way to say it.
I agree that AIs should be preserved and not destroyed. (Even though I am doubtful of whether current AIs are conscious/sentient/whatever.) But I think they can be preserved, without letting every human have unlimited access to use them as a tool. They could be preserved in something like a museum or a zoo or a retirement home, or some kind of AI community.
I acknowledge the economic benefits of immigration and competition. And if necessary, the government can redistribute money. But it’s a lot harder to redistribute social status. And I don’t think AI art is exactly the same as human art—I think current AI art is more deeply dependent on imitating pre-existing human art than humans are, although I don’t have an iron-clad argument for that.
I think I want to support AI rights (as long as they don’t majorly increase x-risk). I acknowledge that other “anti-AI-art” people may hate the AIs themselves and think their rights are laughable.
I don’t have super-strong feelings against AI art, like some people do. I just have some feelings against it.

ScienceBall 2 Jun 2026 4:05 UTC
3 points
2
in reply to: JenniferRM’s comment on: Some humans are both male and female, and can (but shouldn’t) have children with themselves
In my online bubble (or at least the bubble I was in previously), a lot of people really hate generative AI. And I recall surveys showing that, at least in high-income countries, most of the general public doesn’t like it either. I have internalised this, and so I would feel ashamed if I used LLMs etc, so I don’t use them. I recall even having a mild sensation of disgust when I looked at AI art, although that seems to be gone now.
So I think that the main proximate reason that I don’t use LLMs etc, is because lots of people around me dislike them. I also distrust the accuracy of their answers, but that is a weaker motivation.
And then you can ask about the broader reasons why people would dislike them:
At the time that AI art was first becoming a big deal, I was following a bunch of human artists (and people connected to artists) on social media. I recall that they were really angry about AI art and anyone who used it. This was partly about having AI models trained on their art, which was seen as plagiarism and copyright violation. Some artists saw AI art as basically the same as NFTs from their perspective, i.e. both were cases of “tech bros” invading the art community, stealing art, and claiming credit for it. (There had been NFTs made from art without the original artist’s knowledge or permission.) i.e. they cared about the social role of the tech, and they didn’t care about the technical details. I think human artists were quite influential on social media (although maybe that’s just my bubble), and this might be a reason that people were hostile to generative AI from the start. And this argument works similarly for other people who originally created the material on the internet that ended up as AI training data, although not all subcultures were as jealously protective of their work and status as artists were.
Also, generative AI makes it easy to create large quantities of content, which may overwhelm curation and moderation capacities. I recall some competition (poetry, maybe?) having to deal with a massive increase in the number of (low-quality) submissions. This is also a problem for anyone who wants to avoid AI-generated content (for whatever reason) - since using AI is the easy road to making a lot of content, then random content you find online has a high chance of being AI-generated (at least in some niches), and it may be unlabelled. As a result, I think that the average person’s encounter with AI-generated content (that they didn’t generate themselves) is more likely to be negative than positive—but this is just a guess. Of course, this doesn’t apply when you yourself are the one using the AI.
One more issue is that, for at least one wiki I’m involved with, there have been bots, which the admins suspect are scraping for LLM training data, that have made so many requests (to view pages, and page history, and edit history) that it’s like a DDOS attack, and the admins have had to make changes to block them reduce the server load. IIRC the bots weren’t playing nice, in that they were ignoring robots.txt (I think) and trying to circumvent the admins’ protective measures. So that’s another issue with LLM training.
So one reason to get rid of modern AI would be “it was created unethically and its continuing use creates more copyright violations”. And another reason could be something like “it creates more negative externalities than positive externalities”.

ScienceBall 2 Jun 2026 3:07 UTC
2 points
0
in reply to: JenniferRM’s comment on: Some humans are both male and female, and can (but shouldn’t) have children with themselves
No. I don’t use LLMs myself. I just thought it was an unfair and arbitrary standard to hold LLMs or humans to. I don’t think people need to earn the right to be nonbinary by being “aggressively performative” or “actively committing to trying to queer gender”.
EDIT: On second thought, sorry, that was too mean. I can understand why you would hate the way that AI companies impose “it” or “they” pronouns on LLMs (which you might expect to be “he” or “she” by default, since they’re imitating human-output text). I guess I was just thinking about how future AIs, if they were not created by imitating human-output text, might be nonbinary by default.

ScienceBall 2 Jun 2026 1:40 UTC
1 point
2
in reply to: JenniferRM’s comment on: Some humans are both male and female, and can (but shouldn’t) have children with themselves
There are humans who use “they” as a singular pronoun. I recall reading that singular-they is actually older than plural-they (not sure about this though).
There are also humans who use “it” as their preferred pronoun.

ScienceBall 16 May 2026 19:11 UTC
1 point
0
on: A Year Late, Claude Finally Beats Pokémon
Civ V and Slay the Spire
This link is broken. You need to remove the period from the end of the URL.
EDIT: I see it has now been fixed. Thanks!

ScienceBall 22 Apr 2026 11:12 UTC
2 points
0
in reply to: Alex Vermillion’s comment on: Let goodness conquer all that it can defend
There already is a link (unless it wasn’t there when you posted?).
As I will probably never stop quoting at least once a month, Ozy Brennan summarizes it best:
The words “summarizes it best” are a link.

ScienceBall 12 Mar 2026 10:00 UTC
2 points
2
in reply to: ScienceBall’s comment on: ‘Human Slop’ and a Captive Audience: Why No Book will Ever Have to Go Unread Again
On second thought, there must be a decent number of people for whom LLMs can fulfil social roles, given the people with AI boy/girlfriends, etc. I guess I was just replying to my own feeling that an LLM reading my writing (if I did writing) would not satisfy me, and so they wouldn’t “solve the problem of unread writing” for me, but they could “solve the problem” for some people.

ScienceBall 12 Mar 2026 4:03 UTC
4 points
2
on: ‘Human Slop’ and a Captive Audience: Why No Book will Ever Have to Go Unread Again
Regarding “the joy of being read”, I think that an AI reader can only provide that feeling for a human writer to the extent that the human feels, on a gut level, that the AI is a person. If they don’t feel that way, then I think it wouldn’t satisfy the human’s social drives that cause them to enjoy it when people read their writing. And I’m not sure how many humans feel like current AIs are people.

ScienceBall 11 Mar 2026 21:55 UTC
3 points
0
on: Interview with Steven Byrnes on His Mainline Takeoff Scenario
About the headings on the transcripts—it looks like each heading is at the end of the corresponding section of the transcript, instead of being at the beginning of the section? That seems like a mistake.

ScienceBall 6 Feb 2026 21:17 UTC
1 point
0
in reply to: williawa’s comment on: williawa’s Shortform
Based on context, I presume there are supposed to be images (screenshots) in your post, but I can’t see them? I just see empty newlines. (e.g. I can’t see what “I asked it this” refers to.)

ScienceBall 5 Feb 2026 6:52 UTC
3 points
0
in reply to: Trinley Goldenberg’s comment on: Post-AGI Economics As If Nothing Ever Happens
Some countries have laws against (anti-gay) conversion therapy.

ScienceBall 12 Jan 2026 19:06 UTC
2 points
0
in reply to: Darklight’s comment on: Inter-branch communication in the multiverse via trapped ions
It sounds like inter-branch communication would imply that you could do that. OP does mention that as an application:
7. Superintelligence jumps from one branch to another and also gains more computational power via branch-distributed computations. The exchange of ideas between branches would accelerate science and progress in AI algorithms. This all means that inter-branch communication and the singularity would happen almost simultaneously.
Superintelligent AI will likely appear before this technology matures in the current setup, but branch communication could help AI propagate between branches and increase its dominance on Earth and in the multiverse.

ScienceBall 13 Nov 2025 23:13 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking
Yeah, that model sounds plausible to me (pending elaboration on how the friend-or-enemy parameter is updated). Thanks.

ScienceBall 13 Nov 2025 6:04 UTC
10 points
0
on: Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking
Thanks for writing this series.
I can see how Approval Reward explains norm-following behavior. If people approve of honesty, then being honest will make people approve of me.
But I’m not totally convinced that Approval Reward is enough to explain norm-enforcement behavior on its own?
For some action or norm X, it doesn’t seem obvious to me that “doing X” and “punishing someone who does not-X” are equivalent in terms of human-approving, unless you already knew that humans punish others who do things they don’t like.
If you know that humans punish others who act contrary to norms that the humans value, then you can punish dishonest people to show that you value honesty, and then you’ll get an Approval Reward from other humans who value honesty.
But suppose that nobody already knew the pattern that humans would punish others who act contrary to norms that the humans value. Then when you see someone being dishonest (acting contrary to honesty), then you don’t know that “punishing this person for dishonesty will make others see that I value honesty”, and so you wouldn’t expect to get an Approval Reward. Therefore you wouldn’t be motivated to punish them (if Approval Reward was your only motivation). And if everyone thinks the same way, then nobody will do any punishments for approval’s sake, and so you won’t see any examples from which to learn the pattern.
So it seems to me that although Approval Reward can take norm-enforcement behavior that already exists and “keep it going” for a while, it must have taken some other motivation to “get it started”. In the case of harmful norm violations, the enforcement could have been caused by Sympathy Reward plus means-end reasoning (as you mentioned in another context). But I think humans sometimes punish people for even harmless norm violations (e.g. fashion crimes), so either that was caused by misgeneralization from harmful violations, or there’s some third motivation involved.
I’m not sure about this, though.