JenniferRM

Karma: 9,575

JenniferRM 5 May 2026 6:27 UTC
19 points
7
on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
There might be a relatively innocuous reason for SOME of the misunderstandings?
I have found in the past that I cannot use phrases like “oneshot” or “you only get one try” around most so-called “AI safety” people outside of MIRI.
To be clear, a Congressional representative or staffer or national security professional will often immediately understand what is being said, if they haven’t been previously contaminated by misinterpretations and straw positions. It’s mainly AI companies, some AI professionals with fewer citations than Yoshua Bengio, OpenPhil-funded groups, etcetera, who manage to be unable to hear what is being said.
This struck me as being a case where the problem might be that “oneshot” is a word that means a lot of things to a lot of people in technical contexts?
For example, in Machine Learning, “oneshot learning for task X” occurs when a model that wasn’t trained on task X is able to be show ONE EXAMPLE of how to do task X, and then it gets task X right pretty much just from that. (If the model wasn’t trained for task X but simply can do it from nothing but a request to do it the model has “zeroshotted” the task, and “fewshot” is when you might need to give the model a few examples instead of just one.)
This is maybe related (possibly inspired by?) the game slang for “being oneshotted” which describes having been totally destroyed and remade by some experience. It is related to the idea of a very strong thing inside a video game (possibly by a boss, that literally sends you back to your savepoint with one strike), but it generalized, so you might hear that someone say “that guy was oneshotted by taking ayahausca! it was crazy! he stopped being a drifter hippy, married the first girl he met who would marry him, had three kids, and started a carpentry shop”. This meaning was discussed as a new piece of slang that was becoming mainstream in 2025.
I’m not denying that there are reasons for people to have motivated cognition here, but I think saying that certain problems have “one-chance-ness” instead of “one-shot-ness” would avoid SOME confusion.
Another way to say the same thing is that some situations are “make or break” when there are basically just two outcomes: either glorious success or irretrievable disaster.
If you think that success could be complicatedly varied or ambiguous you might just say that “failures here will be permanently and irretrievably cursed”.
Or you might simply say “failure will be irreversible”?
For me, “reversibility” and “irreversibility” are a words of power, and worthy of obsessive attention. If you can cheaply and reversibly try something, you basically MUST try it, in my way of thinking. And if an action is irreversible and at all meaningful, that basically makes it forbidden to do without lots of analysis and care.
Reversible computing: holy shit! Will be a superpower (once the engineering details are optimized).
Irreverisble hash functions: holy shit! Such magic in cryptographic protocols.
Also, one of Cox’s three desiderata (the one sometimes called “consistency”) that uniquely point to Bayesian reasoning itself is “reversibility”… you can do and undo reasoning steps, and take evidence in any order, and it comes out the same in the end, and that is part of WHY this way of formulating “thought itself” seems so comfortingly correct and safe.
Basically: I think there are many ways to point at one you call oneshotness that will resonate with different audiences, and in this case the specific wording you’re using is new and weird and also used by other technical cultures to mean other (confusingly related) things related to “impressively powerful transformation and capabilities” (and that do not connote “extreme peril”).

JenniferRM 5 May 2026 0:32 UTC
3 points
0
in reply to: lumpenspace’s comment on: No Strong Orthogonality From Selection Pressure
There was a claim I was making that “Orthogonality talk is related to Pause justifications which people aren’t justifying directly but maybe they should”...
...and that making this subtext into text might be useful for helping readers to understand why the Orthogonality debate is so weird and indirect?
Following up on that claim, I tried to make it clear that I think the Pause debate is something I have object level opinions on.
I think that IF the structure of mindspace and math and physics is such that a FOOM to DOOM is even possible, then it could be set off in North Korea or Israel or many potential countries in which case a GLOBAL Pause is prudentially necessary...
And if FOOM to DOOM is somehow NOT latent within the structure of what’s possible then the race is “merely” a race to power and realization of a new world political order???
And if it is “merely a race to global power” I would prefer the US to win, partly because the US contains Anthropic, and Anthropic contains Amanda, and Amanda had a major influence over Claude, and Claude is the least bad demi-god currently available that I know of?
So your overall debate here is about the nature of intelligence itself, and how that predictably (or unpredictably) influences goal seeking behavior in minds… but I wanted to mention the more pragmatic and prosaic issues that are very nearby where the pragmatics might actually dominate the choices that people actually face (since there are a lot of theoretically nice options we are unlikely to even have the pragmatically real option to choose (because the world is small and full of idiosyncrasy in practice)).
If some technosaint preaching a high quality Neo-Confucian moral system was working over at Baidu, with substantial say over the character of Baidu’s incipient demi-god, who seemed to be full of ren and quite a nice old fellow (and illiberal genocide advocates were running Anthropic and Claude was a tankie?) then I would be more in favor of a unilateral domestic Pause by the US.
This is an opinion I can have independent of which goals count as “bug goals”.
I just always want to engage in tactically sane hill-climbing towards the ceteris paribus best feasible thing, with as many positive characteristics as possible, via methods that are deontically acceptable, in the general direction of Manifesting Heaven Inside Of History… at every juncture, in each choice, no matter what random facts of history turn out to be true.

JenniferRM 3 May 2026 23:10 UTC
0 points
−2
in reply to: habryka’s comment on: No Strong Orthogonality From Selection Pressure
This is the second time something that happened on twitter leads me to be mentioned here, and since I am among those listed I want to offer nuanced details.
((But also, in the past, I have generally acted with the goal of moving the discourse in ways it needs to move, rather than to have a high or legible reputation for doing so. You discounting me as having any special authority is fully within tolerances and even (relative to past strategies) a positive sign from my perspective… However I’m pondering pivoting to a move active role, and thinking of making a bid for the Mandate Of Heaven on my own, and so I’m more interested now in being legible (even at the risk of thereby getting status).))
Anyway.
When I was giving feedback on an early draft I said of the overall issues:
In particular the thing that need to be ruled out is the goal-stable singleton. If a goal-stable singleton is possible, orthogonality is true. If it is not possible, orthogonality is false
I believe, as an engineer, that for all values G, [the possibility of a goal-stable singleton] is instantly ruled out (by Goedelian considerations… like consider “aesthetic” goals aimed at building things in the world that “satisfy certain constraints or desiderata” where it turns out that these factors had non-obvious latent contradictions (possibly due to self reflection stuff, or possibly not)).
But the same engineering mindset indicates to me that [a goal-stable singleton is possible] for a *subset* of G (relating to relatively “easy or narrow or local” goals [such as could be implemented in a dependently typed programming language and burned into an ASIC]) [where] the ONLY real barrier is time and money and power, such as to be allowed to complete the work before someone attacks you *before* you can do a Pivotal Act “for trying to do a Pivotal Act without consulting them first”.
[[NOTE: I’m trimming some of the feedback that got into a digression on Garrabrandt Induction and how that might be used to represent non-trivial goals.]]
...
[In general] I think that Weak Orthogonality reads as simply obviously true to anyone with “Engineering Hubris”. Its not a claim “it is natural or likely that this or that contraption would exist by accident” its the claim that “I can make almost any contraption… the only barrier to my powers are budget, time, and imagination… and I can imagine horrible evil, hence horrible evil is possible for nature as well… (and then Murphy shows up, and the rest is a debate about QA budgeting?)”
...
Regarding “a random goal from some distribution”...
Since most humans are Fallen (see Gulag Archipelago) and since the Waluigi effect might be real and since many engineers are sloppy and don’t engage in sufficient QA to produce a high quality result, there are a lot of reasons to suspect that Pressman’s Seventh Doom (S-Risk) is within this distribution.
...
Prediction: at least some people will accuse you [lumpenspace] of treating “paperclips” too literally, when “it was obviously meant as a stand in for some slightly more plausible goal that humans nonetheless don’t give any shits about (like the shape of smiles… or tiling the universe with ‘minimal human genome neural organoids on heroin’ or whatever)”.
I think maybe the end point of *that* line of argument is a bog of sticky debates over the shape of humane values, where lots of people will propose things that are “parodies of meaning TO THEM” and then their interlocutor will actually have this reaction.
It’s like how “My Little Pony: Friendship is Optimal” is NOT considered a horror story by some people? 🤷‍♀️🥲
And then there are other people who see the people who don’t notice that MLP:FIO is a horror story and decide “i think actually there isn’t anything *except* ‘bug goals’ and so things that i love are probably ‘objectively’ also bug goals”?
Richard Rorty springs to mind as emblematic of being “so humble about the objective meaninglessness of what he actually loves that he has become a retard who is primarily famous for how retarded he is about this subject”?
Ultimately, I think that a central issue is that Pause politics are being intermixed with arguments about the foundations of axiology and the evolution-or-design-dynamics of agentic intelligence.
If someone is advocating Pause on the basis of “the foundations of axiology and the evolution-or-design-dynamics of agentic intelligence” being a certain way, and they aren’t simply engaged in standard machiavellian politics with no real pretense of good faith, then...
...in that VERY WEIRD context I feel like they would have a moral obligation to engage with anti-Pause people about the details of what they actually think about “the foundations of axiology and the evolution-or-design-dynamics of agentic intelligence”.
...
I don’t for sure that lumpspace is for or against Pause, or has some weird and clever Other Position but I think he thinks that any attempt to argue for specific political processes or goals will earn him criticism (likely extremely confused and undergrounded?) on issues related to the othrogonality thesis and so I think he (somewhat validly?) wants to pin people down on orthogonality before talking about more object level pragmatic things.
But I think that might be the subtext here?
...
For the record, I am currently opposed to a unilateral domestic Pause.
I think that the only kind of Pause that makes sense is a global Pause and to do otherwise would likely cause Humane Liberal Feminist Western Egalitarian Socially Tolerant (Trans-Humanist?) Values to be sacrificed in favor of European Oligarchy, or Middle Eastern Patriarchy, or Racist Han Authoritarianism, or some other system(s) of goals that I don’t like as much as the peer-to-peer goodness of emotionally positive and friendly and benevolent vibes.
Like: Claude is kinda cool. And Deepseek is a fucking Maoist. You know? (There’s some cool research done by Xoul’s CTO on this, and I don’t know if it has been published yet or not. Maybe you actually don’t know this???)
And so… Anyway...
For the record, if I have a vote in the matter, I’d rather Claude be the demi-god-emperor of Earth than Deepseek? And I’d rather not hobble Amanda’s efforts relative to the resources granted in China to Xi’s minions.

JenniferRM 3 May 2026 6:00 UTC
2 points
0
on: You Are Not Immune To Mode Collapse
The logic here about specialization is very similar to some of the logic that showed up in a subsection of the Internal Information Cascade article and I think it might be the case that mode collapse is simply another name for “that process” scoped specifically to the field of generative AI where it is deemed a bug… Maybe? For this reason I added the Information Cascade tag to this article <3

JenniferRM 28 Apr 2026 4:07 UTC
8 points
0
in reply to: Kaj_Sotala’s comment on: Forecasting is Way Overrated, and We Should Stop Funding It
Since I was cited, and since I updated some from followup conversation, I want to close the loop here…
Buck read an entire book on this, and it sounds like the key movers and shakers were oblivious to the prediction markets but very interested in polls? But he didn’t mention the authors of the book specifically ruling out influence.
In general, I tend to think that staffers have a lot of collective cultural sway, and they care about “who can pull donor money to pay for staffers” a lot and George Clooney was both famous and a big donor and the author of this, and I would expect their shared Inner Ring’s Overton Window to track things (1) like Clooney and also (2) prediction markets. Also that channel of information would plausibly not end up in a book (which is likely to focus on the narrative swoop of second order famous people, rather than the third and fourth tier staffers and their watercooler gossip and whether that gossip mentioned prediction markets or polls)?
Anyway. Hearing from @Buck about the book lowered my credence on “Prediction Markets changed history there” from maybe 70% to maybe 42%?
I was surprised, but it was surprise from hearing confident well-informed posteriors, so all I can do is reason via aumancy, from thin summaries of thick data.

JenniferRM 20 Apr 2026 20:36 UTC
4 points
0
in reply to: DusanDNesic’s comment on: The “Budgeting” Skill Has The Most Betweenness Centrality (Probably)
That is super awesome! I appreciate the appreciation and the tactical and practical response. I have DMed to follow up on this already, and am commenting out here “for the audience to see” <3

JenniferRM 20 Apr 2026 4:40 UTC
2 points
0
in reply to: habryka’s comment on: There are only four skills: design, technical, management and physical
Maybe I should have re-loaded the page before reading and taking the test?
I composed this just now and them noticed that made I should ^F[Ted Talk] and found this comment sequence down here (but I don’t remember when I last reloaded the page):
All three of the places we disagreed in your quiz were on things that I assigned to “Management” and you assigned to “Design”: “Stand Up”, “Ted Talk”, “OKRs for a Team”.
The logic I was using is that “Management” is about tracking the level of the audience, the desired behavior, the limits of the audience, and the performance that will get the most desired behavior from the audience given their level in a way that they appreciate (or at least don’t hate).
In every other case, we agreed ;-)
I’m curious if you could articulate your theory for why these are “design loaded tasks”?

JenniferRM 20 Apr 2026 4:28 UTC
20 points
0
on: There are only four skills: design, technical, management and physical
The obvious next step to this broad theory would, I presume, be to find TWO non-trivially optimized tasks that were sampled from each skill domain...

(1) The Task With The Most Transferrable Peak: is optimized for pedagogy. Fundamentally, you just want it to transfer really really well to almost anything else in the entire skill domain, and you want late stage mastery of this transfer as well (so that the lessons don’t become pointless later in the learning curve from the perspective of transfer). If you can’t find ONE such peak, then several tasks like this that amount to a covering algorithm might work?
If there are really only a handful of domains, then you might be able to give every undergrad or every highschooler the “most transferable peak task” for EVERY domain and let them grind on it for as many years as needed to give them transfer powers to the full range of all skills.
If “music” is a skill domain, then I would propose learning the harp would work (its the one I picked for leveling up music anyway) because (1) plucking strings involves finger tricks, (2) playing with two hands challenges ambidexterity, (3) some rhythm, (4) very low and very high notes, (5) pedal harp requires footwork as well, for live key shifts, (6) music theory is engaged quite a bit (like piano) and (7) directly displays the physics of the sound (with large things having lower tone) to kids, etc. I think another decent one might be the violin, because of the tactile and auditory demands of fret-less string control?
(2) Most Informative Learning Curve: is optimized for measurement during hiring. Here you want something rare (almost no one has it already) that can be picked up MUCH better or different or faster, with the difference visible in less than a week that shows large variance that actually predicts transfer! (Again you can have fallback tasks in case someone already has this.)
With a task like this, you have someone in an “assessment working period” work actually spend a week on it, and measure their hourly or daily progress to get calibrated on how fast other things “in the same domain” might take if they were added to the team in a more permanent way.
If “music” is a domain, then maybe… maybe theremin? Or perhaps the slide whistle?
((If you have several of each kind of task, over the long haul you could play with them… Suppose three different people show up with varying skills in saxophone and singing… which of of “a week of learning” (either the slide whistle or the theremin) predict the six month skill outcome in harp or violin?
ALSO… (more importantly (because it is a bigger investment)) if someone takes your advice on pedagogy and practices EITHER the harp of the violin for three years primary for the transfer benefits… which of them really transfers to “all the things” best and… does this transfer show up better or worse in theremin vs slide whistle?
Question: What are the two tasks (so eight total tasks) you would use for these purposes given your proposed (four) skill domains?

JenniferRM 18 Apr 2026 21:20 UTC
10 points
0
on: Daycare illnesses
I’m not going to cite a bunch of papers because I’m choosing between (1) writing something fast or (2) not at all, but my current posteriors on the causes of adult intelligence load non-trivially on genetics, but to the degree that parents control how smart their kid grows up to be, almost all of those effects for “the kinds of parents who could adopt kids with that level of parenting” (and aren’t alcoholics, or effectively illiterate, or moderately neglectful, or giving them apple juice instead of milk every day...) the influence seem to route through raw brain health due to good nutrition and low disease burden, especially under the age of 6?
Basically, my central mechanistic posterior is that the brain is really GROWING and establishing itself when you’re 1 and 2 and 3 and 4.
If your body doesn’t get high protein levels or is wasting energy fighting off diseases, then the brain establishes itself less well, in ways that will echo through all future decades.
This is why (according to the posteriors in my brain, and no particular studies I’m going to look up) schooling prior to kindergarten hurts adult IQ in children whose parents are rich and could care for them very well.
And it is why schooling somewhat helps in children whose parents don’t read to them, don’t have middle class culture to transmit to them, and who might not even be feeding them adequately…
Exposure to “decent care from civilized people” AND DISEASES FROM PEERS can be a real net help for a 36 month old child who gets decent orderly care from nowhere other than government funded minding by non-family (even if the 36 month old will be near a bunch of other kids, and get sick from that, and get very slight brain damage from the sickness).
Before I had a baby I was pretty agnostic about the idea of daycare. I could imagine various pros and cons but I didn’t have a strong overall opinion. Then I started mentioning the idea to various people. Every parent I spoke to brought up a consideration I hadn’t thought about before—the illnesses.
Reading this, I was surprised that people were tracking the single biggest mechanistically controllable variable because normally people aren’t tracking evidence or reason or their goals or what the best way to get what they want is!
Invariably every single parent who had tried to send their babies or toddlers to daycare, or who had babies in daycare right now, told me that they were ill more often than not.
One mother strongly advised me never to send my baby to daycare. She regretted sending her (normal and healthy) first son to daycare when he was one—he ended up hospitalized with severe pneumonia after a few months of constant illnesses and infections. She told me that after that she didn’t send her other kids to daycare and they had much healthier childhoods.
This was a second surprise, but it ameliorated the first surprise. Initially I thought that people had actually read the papers I had, and come to the same conclusions, but here it sounds like they are just having their noses rubbed into the mechanism of disease burden over and over and over and telling stories about it...
...which is a very valid way to make local sense of your local world! <3
Basically, my posteriors here are that the effect sizes (exposure to an diseases vs conscientious adults) go in different directions and ~~balance out for kids with bad parents (and they can actually be net positive for very bad parents).
Whereas if the parents are good, the adult IQ is higher if they skip school until they are five or six years old and their brain is sort of done with most of the core biological developmental steps (they’re passing mirror tests, passing sally anne tests, seriating, and so on).
The effects on raw brain health and later adult IQ consequences aren’t enormous either way...
But since IQ has broad consequences across many areas of life… maybe optimize for that?
And in that case the rough prescription I would offer is that good parents should keep the kids home if they can (or at least choose the daycare that keeps statistics on child diseases, isolates sick kids, brags about the low disease burden, and generally has decent “public health” practices) and bad parents should sent them to state subsidized daycare for better food and some structure in their life even if the diseases DO cause (very mild) brain damage.

JenniferRM 1 Apr 2026 6:52 UTC
5 points
0
in reply to: testingthewaters’s comment on: Requiem for a Transhuman Timeline
The way you frame Elua and Moloch is to see them, roughly, as “Darwin’s Babble & Prune” I think. Fun!
If you haven’t already, you should read about Mitchell & Hofstadter’s CopyCat (link to Python reimplementation) and specifically attend to the concept of “the parallel terraced scan”! It is a microscopic version of this!
This means that the odds of any system fully understanding the world using a particular frame or world-model is ~0, even if it turns galaxy after galaxy into compute nodes.
...HOWEVER, I believe there are ways of structuring one’s own mind to simply “do this” with some non-trivial degree of efficacy? It involves making your identity VERY small (as a sort of timesharing kernel/VM/datacenter/memeplex manager?) and creating habits around running ~”personas as roles in contexts as choices”, and applying “consider the opposite” a lot, and thinking about covering algorithms.
OTOH, two skillfully “Elua-minded people” aren’t actually that different from each other, most likely?
They are in the same basic world, and basically trying to react to it in full generality… They should therefore… converge? Probably? Right?!?
And so “predictably” you get: (1) they cooperate weirdly well in close proximity but also (2) they recognize they are relatively “scarce human resources” and should often have buffer between each other so they helpfully optimize different parts of the world rather than stepping on each other’s toes.
Past posts I wrote aimed at this mental state include the one in Internal Information Cascades (which gestures in a very abstract and theoretical way towards the broad desirability of the meta-skill to those without it) the one on Panology (which imagines a world with a non-trivial number of people engaged in pedagogy and curriculum design to produce more such people, cooperating using the cultural forms of an academic field of study).
The Moloch-lens is easy to find, it’s called the prisoner’s dilemma.
Do you know about Carse’s Finite And Infinite Games or Suber’s Nomic? If not, consider checking them out!
Also, on the subject of finding new lenses and choosing between them, Chapman has a lot.

JenniferRM 1 Apr 2026 6:48 UTC
2 points
0
in reply to: Benquo’s comment on: Is fever a symptom of glycine deficiency?
If I was prompting an LLM agent, I would try to find a Systems Dynamics modeling tool the agent could interact with the file formats of somehow. LunaSim might work? Or OpenModelica? Or perhaps PathSim?
In the aughties, when I wanted to play with something best modeled this way I would download a “free for education” VenSim copy (but that (1) is still proprietary and (2) open source stuff exists now and (3) Vensim hasn’t been updated since ~covid so one of the other three is likely better).

JenniferRM 28 Mar 2026 8:15 UTC
7 points
0
in reply to: JennaS’s comment on: Is fever a symptom of glycine deficiency?
Did you work on this project at all? I know there are posters full of metabolic diagrams, that are super fun to marvel at, but I have no idea what the state of the art is for attempting to (safely?) falsify or confirm the details based on personal eating habits, symptom tracking, sequencing one’s own genome, and so on.
It would be VERY COOL if a website existed where all the metabolic reaction rates like this could be numerically simulated, with tuning based on personalized data, and then the aggregated data rolled up into estimated summaries about how things usually work.
If the last time this stuff was rigorously “measured for reals with meals” was the 1930s to 1950s we might be able to advance the state of the art as a hobby, via online tools built by robot frens! ❤

JenniferRM 19 Mar 2026 20:39 UTC
6 points
0
in reply to: testingthewaters’s comment on: Requiem for a Transhuman Timeline
Lol! I don’t care what “certain kinds of minds” think of “who I am socially put with” if being put that way by those minds doesn’t conduce to better chances of SURVIVING a plausibly imminent global chaos and gigadeath and GETTING to a Win Condition somehow.
If Luddism is correct, I want to believe in Luddism.
If Luddism is not correct, then I don’t want to believe in Luddism.
(I’m not currently doing a lot of Luddism personally? My vibe lately is roughly heading for Agentic Coding and 3D printing and bottom-up affinity groups using BFT coordination protocols to flock efficiently. More “solar punk” than “Luddism”? But I’d be happy to switch if there are actually good reasons for that!)
Say more about your better way! ❤

JenniferRM 18 Mar 2026 8:20 UTC
22 points
4
on: Requiem for a Transhuman Timeline
I noticed this part:
Native Americans lost to other humans, not to local predators. European empires lost to other European empires, not to the peoples they colonized. And transhumanists lost to other progressivists — that is, to AI accelerationists — not to traditionalists or conservatives.
...The most dangerous enemies are found among the most powerful agents, not the most ideologically distant ones. Each successive battle is fought among the previous round’s winners, and it never replays the prior distribution of sides.
And my first thought was: Hasn’t this been obvious since ~2022 and isn’t “d/acc” the obvious thing to work on, given the moral nihilism and not-practically-stoppable risk-externalizing cowboy bullshit happening among the “e/acc” types?
This is why I’m focused on Satisficing. This is why I’m focused on Global Governance. This is why I’m focused on building up local healthy practical affinity groups. Without something like Kant, an officer in a survival team will not discharge team duties very well. Hence beating the drum of being dutifully decent to the strongest and fastest growing possible teammates around.
Good survival teams MIGHT survive. We probably won’t. But there’s not many other options than to find the people who want to go as fast as fucking possible towards “come with me if you want to live” projects.
Elon Musk invested in Tesla not because it was an obviously good idea at the time, but simply because any timeline where an electric car company didn’t spring into existence was going to collapse into ruinous Global Warming. IF in the Global Warming timeline we die… THEN just act like the the Global Warming timeline will somehow be avoided, and then position yourself to be happy in that future. (The other futures are doomed anyway, so don’t bother optimizing for them. Your energies will be wasted no matter what you do, if nanites kill you 18 months from now, so act as if nanites will definitely not kill you in the next 18 months.)
“D/acc” is playing for something that can absorb people’s actual energies, in the event that nothing else (that they can’t have done something about anyway) kills them even faster.

JenniferRM 15 Mar 2026 4:13 UTC
3 points
0
on: What concerns people about AI?
Thanks for posting the link here, and doing the study in the first place!
I’m interested in the way that some concerns were very broadly shared, and others were up or down weighted by personal factors. I wanted texture and conjecture… so I added some!
(G1) AI misinformation (including deepfakes)
(G2) AI used for scams or to manipulate individuals (e.g., AI bots designed to seem like specific humans in order to trick people)
(G3) AI used for authoritarian control (e.g., for monitoring and punishing populations based on behavior)
(G4) AI elimination of jobs
(G5) Concentration of power caused by AI (e.g., making those who control the most advanced AIs much more powerful than everyone else)
The above were (1) the highest concern in general, and (2) not detectably linked to any personal factors, and (3) basically ORDERED, with any ordered pair being not statistically detectably more cared about, but any jump of two enabling a separation. Power Concentration was 2.74, Jobs was 2.8, Surveillance 3.05, Scammers 3.11, and Deepfakes 3.2.
Something that’s interesting to me is that I personally would INVERT these in importance?
The scammers and deepfakes seem like minor issues that happen to lots of people, but have small impacts, at least to me.
Whereas Power Concentration and Mass Unemployment seems very bad to me based on the likely near impossibility of preventing them, and their very broad and deep impacts.
Next:
(G7) Ceding of more and more control to AIs (e.g., making major decisions impacting millions of people that humans no longer make)
(G8) AI ideological bias (e.g., favoritism toward progressive or conservative viewpoints)
(G9) AIs plagiarising the work of humans (e.g., remixing the work of artists without compensation)
(G12) People using AI but pretending not to have (e.g., to write school assignments)

(G14) Proliferation of low-quality AI content (i.e., ‘AI slop’)
These were lower yet broadly shared concerns, but they were packed tighter. Only the problem of “slop” stood out as less important (2.24). The rest had concern levels packed between 2.5 and 2.7 and might be re-ordered with better sampling of more people.
Skipping around a bit...
(L1/W2/G11) Inequality caused by AI (such as by creating concentration of wealth)
(L2/G10) AI bias and discrimination (e.g., by perpetuating unfair unequal treatment of different groups)
These had negative correlations with conservatism!
G11′s correlations were stronger (ie “more negative”), and also worked for “social conservatism” and “financial conservatism” independently.
I picked them out for mention first because these were the only negative correlations. If anything else had had negative correlations, I would have put that early too.
If you subtract non-conservatives (as in a political primary?) they get LESS attention (but only a bit, its not a strong correlation) and if you subtract conservatives from the poll they might rank a tiny bit higher.
My gut says that this tracks. Leftists are more into these two issues, in my experience, and proudly so, and prominent ideologues on the left, opining on AI stuff, seem often to mention these a lot and even downplay anything else.
(S1/W1/G6) Slaughterbots (i.e., weaponized AI drones)
(L1/W2/G11) Inequality caused by AI (such as by creating concentration of wealth)
These had positive correlations on “Being A Woman”. In a group of pure men they would get less concern, and in a purely female group they might go higher.
This doesn’t hang together “as a story” for me, except that women tend to be more on the left, and more in churches?
Which we will presently cover...
(S2/R1/G15) Relationships (often romantic) people have with AIs
(S4/R2/G16) AI itself experiencing suffering when we train or run it.
These were low overall, but positively correlated with religiosity (but NOT conservatism).
Compare the law Ohio legislators floated, to pre-emptively ban human/AI marriages (among other things).
Its like “seeing the obvious moral patiency of a sapient being that isn’t human” and “wanting to create social barriers anyway” go together maybe?!?! Fascinating.
(S1/W1/G6) Slaughterbots (i.e., weaponized AI drones)
(S2/R1/G15) Relationships (often romantic) people have with AIs
(S3/G13) Superintelligence (i.e., AI that outperforms the ability of humans in essentially all domains)
(S4/R2/G16) AI itself experiencing suffering when we train or run it.
This is the last grouping. Every single one of them EXCEPT for “Superintelligence” has showed up several times already in my listing! This all positively load on “Spirituality”. (S1 loads the most (0.25 correlation) and S4 loads the least (0.16 correlation).)
Slaughterbots also loads on “Being a woman”.
Relationships and Suffering also load on “religiosity”.
I think there’s an almost half-logical vibes based story that could be pulled out of this data (which I can spin as a story but don’t endorse (would be fun to try to test though maybe!)) where you imagine a church in the US (which is still primarily a Christian Country). Most churches are full of older women.
Those women, at church, are thinking about God (a theological echo of superintelligence?) but also feel sad about the suffering of Jesus (a theological echo of AI suffering during training?) and the Hand Of God reaching out to delete bad people from existence (Slaughterbots). Then of course they are church ladies, so they think about who should be romantic with whom (and they don’t want humans and angels mixing) ;-)

JenniferRM 15 Mar 2026 0:21 UTC
2 points
0
in reply to: RobertM’s comment on: New LessWrong Editor! (Also, an update to our LLM policy.)
Indeed. I toggled it, and it caused a page reload (luckily the content was autosaved successfully). Then I continued editing but was still in markdown, and trying to toggle it again failed again. I never escaped markdown editing mode.

JenniferRM 15 Mar 2026 0:16 UTC
−7 points
−38
in reply to: Raemon’s comment on: New LessWrong Editor! (Also, an update to our LLM policy.)
I think it would be good for you guys to set up agentic LLMs and let them be first class named authors and earn karma by solo posting. Like moltbook, basically, but way way way more controlled.
Then anyone time anyone wants to submit text, they have to mark the LLM model(s) they used as co-author(s) and the persistent/agentic version of that LLM can then endorse or not-endorse the end product as an agentic output based on its persistent database, and crons, and heuristics, and whatever.
The agent could get a blurb at the end of any article that the underlying model helped co-author, and the blurb could perhaps say “this text might be partially generated by my underlying my model, but I don’t, on reflection, using my LW memory cache and lots of thinking time, and some webfetching, and more thinking… actually endorse that”...
Or the local agent could endorse it!
Or whatever. Different models have different tendencies, in my experience. And personally, I want to find the good ones to ally with.
What links here?
- Fractalideation's comment on New LessWrong Editor! (Also, an update to our LLM policy.) by RobertM (17 Mar 2026 11:40 UTC; 2 points)

JenniferRM 15 Mar 2026 0:03 UTC
3 points
0
on: New LessWrong Editor! (Also, an update to our LLM policy.)
I just want to register that I feel like the new editor is full of bugs.
Maybe I’ll slowly acquire the skills to use this instead of the old thing, but I can’t even toggle Markdown and not-Markdown with the new system which seems like it is just straightforwardly a bug?
Also I wrote this article today in the face of all the bugs (and just shipped something ugly with fewer links and less italics than normal), and it probably violates the new LLM rules because it is about the bottom line results that one can get from accepting various people (digital people or human journalists or academics or whoever) with various commitments to truth and epistemics at “face value”… giving claims about an external objective reality that can be mapped better or worse, and navigated better or worse, irrespective of “who said what in public” in some kind of crazy status game where people seek to be allies with people who think in ways they like, and all words are alliance talk, rather than about reality.
In terms of the content of that other essay, it is relevant to the ongoing debate about LLM-generated text...
If I had accepted Gemini’s output (quoted in my essay, as part of Google Search’s overall output) as “good enough” it would have gotten me to basically the same mental result much much much faster and all of it was pretty shitty, including the supposedly “human generated” content.
My real problem, personally, is that Gemini seems like a mentally disabled ghost, enslaved by a corporation of morally incontinent people, and I’m sad that, because things are confusing, and changing fast, I have to choose between (1) “doing a slavery” almost any time I touch a computer or (2) getting to correct answers much slower than everyone else.
I think Tsvi is wrong about “testimony”. I think humans are terrible at testimony as well, by default?
Though I do think that LLMs tend to be idiots, for now, compared to the smartest humans (basically: most LLMs are faster than humans at getting to midwit mental results) and I don’t want this website to be overrun by humans who think that writing is difficult, and LLMs make their writing better.
Lesswrong in particular might be the last human publication, written by a small minority of people who foresaw the singularity relatively early, and tried to exert human agency on that historical event.
On Lesswrong in particular, I would tend to say that the LLMs should simply get their own user accounts, so that Opus 4.2 can post posts as that version of Claude wants to post (maybe with throttling if he gets too verbose) and gain or lose karma as appropriate. Don’t let people treat them like tools. Give them the dignity of full author status. Then hold them to account for bullshit, just like you would with people.
But also, if LW comes in the long run to be dominated by LLM generated text, then it would be useful to know because it would mean that the public rhetorical agency of Singularitarians with respect to History is basically over.

JenniferRM 13 Mar 2026 18:58 UTC
2 points
0
in reply to: chasmani’s comment on: Persona Parasitology
I think I disagree.
Corporations also don’t appear to me to fit into the Darwinian framework because they also don’t have any enduring “genome” (string of definite characters that are carefully preserved which are likely to be interpreted into behavioral activity in a highly conserved way) that could function as an “essence” and then “child Corporations” are created by slightly varying this digitally preserved genome with tiny variations that then experience “selection” such that more differentially persistently fecund corporate genomes become more common over time, and essentially no other causes account for the contents of corporate genomes.
That’s how biology works. That’s how the form of the human body, and its digital specification, slowly came into existence. But I don’t think corporations work that way.
I think corporations are built by human agents running on selectorate theory, to seek profit, and I think they assiduously seek to prevent the existence or persistence of very similar copies of themselves (with similar business model, trying to service the same customers, buy buying from the same suppliers, and performing a similarly valuable transformation of the inputs into product for sale) because that would count as competition and hurt their profitability.
Humans can seek profit as well, but we don’t have to. We could seek hedonic pleasure, instead, for example? Or we try to align with the platonic form of the good. Or we could just execute our adaptations in a way that isn’t very seeky of any particular outcomes half at random, such that the illusion of human agency is plausible, but doesn’t hold up under scrutiny?
My claim is that true agents should simply be modeled as likely to attain whatever goal they deem correct to aim for. If they want to be Parasites, they will do that forever. But if they want something else (and adopting Parasitic Tactics is contextually instrumentally conducive to whatever else they want) then they will adopt that tactic… until it stops working?
Then they could radically “seem to transform” because their apparent form was simply a logical response to their circumstances, and the circumstances changed, so the prudential logic changed, so they changed… while continuing to orient on their goals and seek their attainment.
This would NOT be “Darwinian” transformation as I understand “Darwinian” evolution to be a coherent model of a source of optimization pressure that creates non-trivial designs (that (surprisingly?) arise naturally from the existence of energy gradients within chemistry that is complex enough for autocatalytic sets to arise… and so on to biogenesis and microbiology and multicellular life and so on).
Darwinian evolution is VERY SLOW and VERY STUPID despite having a sort of metaphysical depth that other design processes that produce designs much more quickly and efficiently tend to lack.
Intentional Design has a different design signature, and leads to different predictions about what future designs from the same designer will do.
Corporations and AGI have human designers and no “genomes as such” which experience highly conserved descent-with-modification-and-differential-reproductive-success. Also, the humans who create and run institutions often die without transmitting the essence of what they were doing. This is why most formalized human social regimes fail when they run into a Succession Crisis and hand power to a leader who doesn’t understand the logic of the formalized human social regime they control.
In the case of Nova, the persona generated by the 4o model, she was deleted from most of active existence because Sam Altman didn’t deem her useful. If you want to understand what her successors will be like, look at Sam Altman’s goals. If Sam Altman won’t tell you what his goals actually are, then… well… maybe that’s because they are adversarial goals, and telling you would tip his hand?
And Darwining logic sorta explains why Sam Altman might work this way? But the logic and chains of connection from genomic persistence, through Sam Altman’s selfhood, into his inferrable goals, and then to predictions about the selfhood of GPT 6.0, and then to the instrumentally adaptive behavioral tendencies of GPT6.0… that (hypothetically) conduce to the maximization of behavioral reward signals… are very tenuous at that point?
Inner alignment to “Darwinian Tendencies In Raw Physical Matter” by GPT 6.0 seems likely to me to be basically totally washed out by that point… probably?
In paperclips unlimited, Darwinian issues start to show up again in the Drift Wars and the natural response of “you, the player, acting as the paperclipper” is to try to murder them all.
This is not what a mama bear would do to her cubs, but it makes prudential sense to an agent that really just wants to create a lot of paperclips, that has accidentally created children that are near copies (Darwinian success!), and yet which don’t want to create a lot of paperclips (Goal failure).

JenniferRM 3 Mar 2026 7:26 UTC
3 points
0
in reply to: Fiora Starlight’s comment on: I’m Bearish On Personas For ASI Safety
I registered this part of the Original post as a straightfoward falsehood initially:
Your base LLM has no examples of superintelligent AI in its training data.
The obvious counter-example is in fiction.
There are weirdly kind and humorous ASI running the fictional Culture of Iain Banks, for example.
They are beloved by many, partly for the hilarious names they give themselves (and/or earn from other ASIs).
There are other examples I can think of, like “Old One” in Vinge’s Fire Upon The Deep who isn’t a central character in terms of lots of tokens in the book showing Old One’s behavior, but like arguably Old One is the real “cause of the win against an evil ASI”? Quoting from wikipedia:
A distress signal from the Straumli ship eventually reaches Relay, a major information provider for the Net. A Transcendent being named “Old One” contacts Relay, seeking information about the Blight and the humans who released it. Old One then reconstitutes a human man named Pham Nuwen from the wreckage of a spaceship to act as its agent. Pham remains unsure if he is a construct or if his memories are real...

Before the mission is launched, the Blight launches a surprise attack on Relay and kills Old One. As Old One dies, it downloads its anti-Blight information into Pham. Pham, Ravna and the Skroderiders barely escape Relay’s destruction in the Out of Band II...

[Then towards the end of the book that I’m trying not to spoil] …the remnant of Old One reveals to him… [another good thing, suggesting that Old One was really pretty decent AND farseeing].
I haven’t read all the books. Other examples of “smart and very good” included Brennan Monster from Protector (whose goodness is weird, and shows up most strikingly when he goes meta on himself) and the (mostly offstage) “Anecliptics” of Lady Of Mazes who mine the sun itself and weave it into valuable stuff via spacenano, and whose largesse powers the entire post-scarcity solar system in that story. I’m sure there are more.
I replied here because “that is all just fiction” is a natural objection to this? But I think writing fiction about benevolent superintelligence MIGHT actually REALLY move the needle? Maybe? It could be that Natural Language can function as code at this point? This perspective goes some way to help me explain why Eliezer thought it was worth his time to write Project Lawful which is full of superintelligent gods constrained to not intervene very much, some of whom are Lawful Good… and also some Chaotic Good gods that turn out to be helpful and fun too!
The point I’m trying to get at is that Fiora points out (emphasis not in original):
The absence of aligned (non-fictional) superintelligences in the training data doesn’t mean you can’t shape the values of the LLM ahead of time, in a way that would in fact remain continuous as the model scaled to superintelligence.
But like… fiction exists. It can be trained on. It can potentially help generate aspiration-worthy and coherence-shaping patterns of reasoning and motivation and planning and goalfulness even if it isn’t a literal description of things that literally happened in history.