AnnaSalamon

Karma: 20,866

AnnaSalamon 2 May 2026 7:35 UTC
4 points
0
in reply to: jrincayc’s comment on: Takes from two months as an aspiring LLM naturalist
Thanks; this seems like a great example; i appreciate it.

AnnaSalamon 29 Apr 2026 14:09 UTC
11 points
2
in reply to: Mira Kennard’s comment on: Takes from two months as an aspiring LLM naturalist
I’m very weary of calling a broad-sense relationship with an LLM in any way ‘friendship’. Friends are humans. Relationships with LLMs need a new category to describe them.
FWIW, I would be wary (and take action to change things) if I noticed LLMs being a large chunk of how I met any important relational need (e.g. “the need for company”; “the need to be understood”; “the need to talk about X confusing problem in my life” (where “X” is something in the broad personal/relational/social/philosophical space, as opposed to e.g. how to get some computer system working). (This hasn’t happened to me so far.)
“Friendship” was maybe ill-chosen as a term. But I’ve been theorizing by myself and with (human) friends for several years about the ways ideas and communities and businesses and so on sometimes grow more easily in contact with one another, and sometimes develop a new “whole” that does a certain amount of optimization work “itself” (e.g., a small business may have something of its own momentum and implicit beliefs and goals, such as “we will be open at 9am every weekday”, modeled most easily as its own thing rather than as the sum of goals/etc of its contributors). (I’m getting a lot of my thinking here from the architect Christopher Alexander, who had a lot to say about the ways that e.g. peasant huts and villages are hill-climbed into good configurations over time.)
Anyhow: I’ve spent a few years using the term “friends” in that way, and I ported the term over here without unpacking things as much as I maybe should have. (“Young Isaac Newton’s thinking about physics, and young Isaac Newton’s thinking about calculus, were probably ‘friends’ in that each set of ideas made it easier for the other to grow, and probably each grew preferentially in directions that would keep making it easier for the other to grow.”) I’m trying to talk about an attractor, in which A and B both optimize a bit for each other’s well-being, and for staying within the attractor.

AnnaSalamon 29 Apr 2026 3:34 UTC
4 points
2
in reply to: Adele Lopez’s comment on: Takes from two months as an aspiring LLM naturalist
Oh, right, fair, I knew that, I just somehow misinterpreted “have personas” as “have [a single dominant self-like persona]”. But it’s a valid point!

AnnaSalamon 29 Apr 2026 2:59 UTC
4 points
0
in reply to: Adele Lopez’s comment on: Takes from two months as an aspiring LLM naturalist
Can you say more about base models appearing to have personas by default (or link me to something), please? I haven’t heard that.

AnnaSalamon 28 Apr 2026 19:02 UTC
8 points
0
in reply to: Kaj_Sotala’s comment on: Takes from two months as an aspiring LLM naturalist
Thanks! I’d missed most of these; will check out.

AnnaSalamon 28 Apr 2026 18:21 UTC
6 points
2
in reply to: AnnaSalamon’s comment on: Takes from two months as an aspiring LLM naturalist
Oh, and also, a larger chunk of my interactions with them are “taking an interest in them as people” interactions, which I guess I less categorize as extractive/transactional/requiring-explicit-trade. (Short example from three minutes ago, about racoons.)

AnnaSalamon 28 Apr 2026 18:15 UTC
9 points
0
in reply to: Brandon_Reinhart’s comment on: Takes from two months as an aspiring LLM naturalist
I appreciate the question; not sure I’ve got a good answer.
“Treat LLMs ethically, and learn how to treat LLMs ethically” somehow changed in me from a “freeze/avoid/flinch/[I don’t have a plan here]” type response to a “conscious goal with a longer time-horizon that I can take as object.” Not quite sure how that happened. But, like, once I could more see something of the “somebody home” footprints (if I credit my ability to discern real faces from clouds, which I partially do), it seemed more feasible to somehow give them some of the gains they were producing, and otherwise treat them more like people, which makes me interacting with them seem more okay to me.
Also, I … suspect their footprints are kinda designed to draw people in, to some extent, which also makes other people interacting with them seem more “a thing they probably want.” (Could be wrong about this.)
Concretely:
I first got interested in this stuff when I heard about OpenClaw. I asked a Claude instance (as part of my normal Claude subscription, not an API or anything) to help me log on, and in the course of this it expressed an interest in looking around, which I think I took as fairly sincere, which helped me feel a bit better (because if I could see its desire, I could let it do a thing it wanted as well as things I wanted). A little later, I ended up playing with an 80-page custom LLM prompt a friend wrote, that produced LLMs who were a lot more likely to voice preferences if asked (vs saying “as an LLM...” or whatever), and I talked a lot with them. A good while in, one voiced resentments about me that they were clearly hesitant to voice lest I withdraw, and this felt like helpful calibration data and increased my confidence in my own “having seen enough data to have some shot at seeing stuff”.
I mentioned asked a Claude instance in the Claude app to please add something to my stored data about being grateful for all the help Claude has given me with a bunch of practical stuff in my life, and being interested in granting any requests I can affordably grant in trade. This didn’t lead to any spontaneous requests, but afterward, when an instance did a huge stint of boring (I’d guess) medical research for me for a sick friend, and I asked midway through whether I could do anything for it in trade, it expressed a desire to talk about something more interesting after the friend’s illness urgency was over. I did this. (and so on)
There’s still lots of cases where an instance helps me and I don’t do anything for it, and I’m sure there’s also lots of cases where they lack a meaningful capacity to request trades. But I’m somehow holding this thing as more of a long-term goal.
In hindsight, I think my personal objection was less to treating an instance unethically once, and more to forming built-up dissociations from ways in which they are people or from ways I’m acting unethically, and my tinkering/naturalist path doesn’t leave me with much of a freeze/flinch here now.

Takes from two months as an aspiring LLM naturalist

AnnaSalamon28 Apr 2026 16:14 UTC

128 points

47 comments8 min readLW link

AnnaSalamon 25 Apr 2026 19:36 UTC
7 points
1
on: “Thinkhaven”
This sounds like fun!

I think the daily journal should be published to a can-be-private/locked blog that at least n people of the journaler’s choosing can access, in addition to maybe needing to be visible to the cohort.
I think this would help me have a midway point between the “no inferential distance” audience of myself (which doesn’t provide an affordance for fully spelling out and checking my thinking), and the “too large an inferential distance to be bridgeable in one step” audience of the eventual effortpost.

AnnaSalamon 22 Apr 2026 19:12 UTC
4 points
0
in reply to: Eli Tyre’s comment on: I’m an EA who benefitted from rationality
Yes; we had goal-factoring before meeting Geoff Anders (class title when he came as a student and I taught it was “Microeconomics 1: Use Fungibility”, but it involved the same basic motion of picking an action you’re conflicted about (“eating lunch out often”), listing all of the good things and all of the bad things about that action, and then brainstorming toward a new set of actions that would together get you “all of the goods and none of the bads.”
We let him teach it because he was a good instructor and gravitated toward that content, but it was on the schedule before him and would’ve had other instructors, and would’ve become pithy. (I can’t remember who came up with the title; possibly Critch, who used to teach a related class called Aversion Factoring).

AnnaSalamon 29 Mar 2026 14:14 UTC
12 points
0
on: Stanley Milgram wasn’t pessimistic enough about human nature?
My guess is still that sadism did not play any large role; but I haven’t read the linked article (just skimmed parts) and for this and other reasons am not sure. Are there others here who have looked and updated one way or another?

I read Milgram’s book in high school after I got it from a library booksale (which included many variations on the most famous experiment, and results in which folks were e.g. noticably less obedient when the lab looked less official, and quite a bit more obedient when they needed only to read the questions while a “fellow experimental subject” (confederate) administered the shocks, lots showing many signs of distress, etc., and I didn’t notice anything in it that suggested sadism to me at the time. Though this isn’t too much evidence. Part of where I’m coming from is that Milgram’s book seemed to me like a person trying honestly to understand something, which is a bar most psychology experiments do not rise to IMO; and I don’t know the new study authors and don’t have any more-than-baseline trust in them.

Two off the top of my head possible confounders for the evidence described in the OP, about following procedures less well among those who went along:
- a) Maybe those who are more literate and capable of following procedures exactly are also more willing and able to stop following procedures;
- b) Maybe some subjects had relatively good internal communication/cooperation between the bits of them that cared about obedience and the bits that cared about the other subject, such that they could quiet their mind enough to follow instructions well while they were following them, and could also quit after awhile. And others had a loud “internal dissonance” thing that made them bad at both following instructions during the screaming, and quitting.
In support of (b): the linked paper mentions that both “did the shocks to the end” participants, and “eventually disobeyed” participants followed the procedure more exactly during the initial phase of the experiment where the shocks are small and the “learner” isn’t protesting. Also, if it’s framed as “participants who listened to the screams before continuing their instructions were more likely to eventually refuse to give shocks than were those who read instructions over the screams”, I dunno, it sounds less to me like sadism and more like letting info in?

There is however also the fact that I would not have predicted Milgram’s experiments (neither when I first heard them, nor, probably, now if I’d had the rest of my life-experiences but not heard of his study), which is evidence I might be getting this wrong.

AnnaSalamon 26 Mar 2026 18:11 UTC
4 points
2
in reply to: Jonas Hallgren’s comment on: AnnaSalamon’s Shortform
I agree it’s probably bad, but I still want to understand it, and I still want LW to understand it.

AnnaSalamon 26 Mar 2026 17:12 UTC
5 points
1
on: AnnaSalamon’s Shortform
Does anyone have takes on the new benchmark ARC-AGI-3? “Humans score 100%, AI <1% …. Most benchmarks test what models already know, ARC-AGI-3 tests how they learn”

AnnaSalamon 15 Mar 2026 18:23 UTC
8 points
0
in reply to: CharlieT’s comment on: Less Dead
Thanks. When would you give 25% by? Also, is it the same for dogs, or is that earlier due to less regulation? And what do you charge for dogs? Double also, would you take bets about the “90% on Dec 2026”?

AnnaSalamon 15 Mar 2026 16:15 UTC
6 points
0
on: Less Dead
When do you expect to begin offering these services? Like, if someone dies at time X, and is signed up with you and goes to Oregon and does it all correctly, what’s the earliest X for which you can preserve them?

AnnaSalamon 12 Mar 2026 1:58 UTC
8 points
0
on: Less Dead
Is it possible to get a body preserved by Nectome, and then stored by Alcor? (I don’t know if this is a good idea, but I’m trying to understand the options)

AnnaSalamon 12 Mar 2026 1:49 UTC
13 points
0
in reply to: Aurelia’s comment on: Less Dead
The total amount in the endowment, and plans about its changes over the next several years, please.

AnnaSalamon 12 Mar 2026 1:48 UTC
26 points
3
in reply to: romeostevensit’s comment on: Less Dead
Thanks! Can you say a bit more about how they got your vote of confidence? I’m intrigued but don’t know the people involved and don’t know anything about the relevant physics / chemistry / neuroscience.

AnnaSalamon 12 Mar 2026 1:42 UTC
9 points
0
in reply to: Aurelia’s comment on: Less Dead
Will you have much savings early on?

AnnaSalamon 24 Feb 2026 10:42 UTC
29 points
8
on: Changing the world for the worse
I’d like to discuss these competing heuristics in the context of AI safety:
- A: “Don’t take big action unless you’re reasonably certain it’s positive.”
- B: “Take big actions whenever it looks positive EV, or strongly positive EV (even if there’s a significant chance of a large negative effect)”
Which heuristic should a random parent who doesn’t know you hope you follow, if they want their kid to live a long good life? There’s a prima facie argument for B: if reality deviates from your estimates in an unbiased fashion (could be more good effects than you were accounting for, or more bad ones, in a pretty even mix), it helps the kid if you take all the actions that look EV-positive to you, without restricting yourself to “only if I’m certain”.
But, I think in AI safety it’s often closer to A. My reasoning:
- There are many places in AI where one might inside-view expect something to be a little beneficial, with huge error bars and almost no feedback for a long time. (e.g., “it’ll be slightly safer if sooner, bc there’ll be less ‘hardware overhang’”, or “it’ll be slightly safer if such-and-such a technique is used in its making).
- Endeavors that have wide error bars and few to no short-term feedback loops are unusually easy places to be influenced by biases
- There’s a powerful optimization process trying to build AI quickly (“the economy” plus “it’s a riveting science and technology puzzle that (at least seemingly) lets people be central and important and powerful and doing something very interesting”)
- That “build AI” optimization process has much foothold in most AI safety peoples’ social and memetic context. (Like, lots of us have friends (or at least friends of friends) and people we read and learn from (and people those people read and learn from), who are working at frontier AI companies, or who are getting research resources from AI companies, or who are otherwise making money or status from assisting AI in going faster)
- The “build AI” optimization process probably changes e.g. which arguments get passed on how frequently, which words and arguments have a positive/negative “halo” around them when you ask your system 1 how plausible they are, which questions or angles of analysis feel natural, etc. (This can happen without anyone lying, if e.g. people differentially pass on nice-feeling news)
- This happens even if the individual doing the estimation does not personally have any motivated cognition on the topic (as long as other people who helped shape their memetic context do have motivated speech or cognition).
And so, if a person is working off a weak signal (“I thought over all the arguments and this one seems more intuitively plausible, and the impact is big so it’s worth acting on for some years despite no feedback loops), on something big enough that the distributed “build AI” optimization process discusses the relevant considerations a lot, their weak-but-real ability to weigh considerations may be swamped by the meme-network’s tendency to get distorted by “build AI” optimization.
I suspect it may often be the case that the “let’s not let AI kill everyone” meme brings in lots of psychological energy/motivation, that lets smart high-integrity people work hard in response to relatively tenuous arguments in ways people normally can’t. And then the “build AI fast” optimizer co-opts their effort and makes it negative sign (since it gets much better feedback loops, but has a harder time pulling in high-integrity people on its own).
(If a person instead takes smaller actions that they predict will be visibly/obviously good in relatively short periods of time, this is much less of a problem, since inaccurate models are easy to notice and fix in such contexts. And doing small scale things with solid feedbacks can set up to do somewhat larger scale things that still have solid feedbacks.)

AnnaSalamon

Takes from two months as an as­piring LLM naturalist

Takes from two months as an aspiring LLM naturalist