Benjy_Forstadt

Karma: 160

Benjy_Forstadt 1 Jul 2026 2:39 UTC
2 points
2
in reply to: RobertM’s comment on: Functional Decision Theory: Not Even Wrong, Also Wrong
>Certainly you are welcome to define your way to victory
I mean, you‘re free to use the term “decision theory“ to refer to whatever you want. But the claim that the theory about what decisions to make is the same thing as the theory about how to program an AI or what are the most desirable dispositions to have is what actually requires an argument.

Benjy_Forstadt 1 Jul 2026 2:20 UTC
1 point
0
on: Functional Decision Theory: Not Even Wrong, Also Wrong
My personal ranking of decision theories goes EDT > UDT/FDT > CDT
I don’t really see why I should believe the only important thing to consider about my decisions is their causal impact. Even more convincing to me than Newcomb’s problem is the perfect deterministic twin prisoner’s dilemma. On the other hand, the UDT/FDT belief that the true theory of rationality can’t be diachronically inconsistent seems like wishful thinking to me. There is a certain sort of beauty and elegance in thinking that the correct theory about what actions to take is the same as the theory about what dispositions it would be luckiest to have, but I don’t know why I should believe that either. EDT is closer to my common sense: it never makes you sacrifice your expected utility on the alter of theoretical ideas about causality or subjunctive dependency or ideal-agenthood. It never forces you to think about possible worlds that you know aren’t the actual world.

Benjy_Forstadt 13 May 2026 18:52 UTC
6 points
1
on: Who Got Breasts First and How We Got Them
Thanks for this post, very interesting.
It should be obvious that the Venus of Willendorf is late in the game, because Subsaharan Africans diverged from Eurasians well before the Upper Paleolithic, something like ~70-100k years ago iirc, with some African groups having even earlier dates.
Perhaps I’m misunderstanding something, but I don’t see why the sexual selection story has to be tied to a post-Neanderthal divergence date. Modern humans are diagnosed in the fossil record by their skull shapes and there’s no particular reason to assume that the development of modern looking skulls would have happened at the same time as other anatomical features

Benjy_Forstadt 23 Apr 2026 5:16 UTC
21 points
26
on: (Re)introduction of a rationalist dragon, and clarifications on Ziz’s character
If you’re trying to come across as sane, safe, reasonable and trustworthy in the future, my advice would be to try to write a little more normally. It’s fine if you want to become a dragon in the future, really. I just don’t see why it’s relevant to the topics at hand. Another example, and this is a little mean to say, but I don’t see why it’s necessary to go on for multiple paragraphs of purple prose describing your hopes and aspirations for humanity.
I’m sorry if the reason you brought these things up is that they are very important to your identity. But many people entertain transhumanist fantasies of body modification or have utopian hopes for the future of the world. The impression I am getting from this post is of someone who is lost in their own head, someone who has adopted a very insular frame and doesn’t feel comfortable stepping outside of it. Writing in a more ordinary fashion would go a little ways to put people like me at ease that you’re grounded in reality.
I would also try to avoid unnecessary jargon and references, especially to works of fanfiction. Yes, it’s true that intellectual communities often develop these ways of marking members of an ingroup. But if people are constantly mistaking you for a cultist, it’s probably best to lay off the esotericism.

Benjy_Forstadt 24 Mar 2026 17:55 UTC
1 point
0
on: I’m Bearish On Personas For ASI Safety
If the choice was between Claude—and you personally - “growing up“ and becoming master of the universe, then I understand why it being Claude instead of you might be upsetting. But if the choice is between Claude and humanity as a whole, I don’t see why you’re so sure humanity is the better choice here.

Benjy_Forstadt 24 Mar 2026 16:39 UTC
3 points
0
in reply to: cubefox’s comment on: Does Hebrew Have Verbs?
That’s true, but when we’re looking at natural languages that’s not a super useful perspective to take because natural languages care a lot about how information is packaged. From a linguistic point of view, predication is a way of packaging information with a topic-comment structure. The topic can be thought of as a ”discourse file” and the comment can be thought as information that is being added to that file. Reference can be thought of as opening a new file or accessing a pre-existing one.
I slightly mispoke earlier when I paired adjectives with “attributing”, the function Croft pairs it with is actually “modification”. Modification is like adding supplementary information to a file.
It‘s a linguistic universal that when it comes to concrete material object concepts like “being a cat“, “being a rock” etc, there is a strong preference for packaging this information as a topic in the discourse rather than a comment. On the other hand, for action concepts like “walking”, there is a strong preference for packaging this information as a comment that is explicitly being added to a discourse file. All languages allow you to use nonprototypical combinations of semantic class and information packaging function though—It is possible to add the info that something is a cat to a pre-existing discourse file, it’s just not as common.
Predicate logic doesn’t really have discourse files, unless you think of quantification as opening an empty file.

Benjy_Forstadt 20 Mar 2026 16:22 UTC
3 points
0
on: Does Hebrew Have Verbs?
When we’re talking about grammatical categories like “noun” and “verb”, it is VERY important that we make a sharp distinction between language-internal categories and categories that are useful for making comparisons across languages. If you are trying to say that some languages don’t have verbs, you better have a very clear criterion for what counts as a “verb” that can be applied to any language, otherwise you’re just going to get yourself hopelessly confused.

The best way to define “noun”, “adjective” and “verb” in a cross-linguistically valid way is actually the naive schoolboy approach of defining a noun as a word for an object, an adjective as a word for a property, and a verb as a word for an activity. A refinement of this idea is that a noun is a linguistic form used to refer to an object concept, an adjective is a form used to attribute a property concept, and a verb is a form used to predicate an action concept. All languages have ways of expressing these functions, but they can differ widely in the kinds of forms they use. I’m getting this way of talking from the linguist William Croft’s excellent textbook Morphosyntax: Constructions of the World’s Languages.

This definition has some unintuitive consequences. For example, in the sentence “the man is a doctor”, the word “doctor” is actually not a noun in the crosslinguistic sense. This is because it is being used to predicate an object concept, not to refer to it. You might see this as pathological, but this kind of thing is unavoidable when we’re defining categories neutrally across languages. In English, when we predicate object concepts, we use a construction with an indefinite article and a copula. In Nahuatl, the roots denoting objects take agreement markers like verbs—they would say something like “the man doctors”. Does that mean that in Nahuatl, “they don’t have nouns, only verbs”? Well, you could say something like that, but not all languages have indefinite articles or agreement marking, so how are you going to define what you even mean by “noun” and “verb”?

Spinoza is pointing out something important and interesting about the Hebrew infinitive (a form used to refer to an action concept) when he says that it doesn’t carry tense like the Latin infinitive. But not all languages have tense! The point I’m trying to make isn’t that languages are actually all pretty similar, they all really have nouns and verbs “deep down”. Actually, languages differ widely in the strategies they use, and the parts-of-speech categories that come naturally to them, even in the same language family (English is actually pretty flexible when it comes to parts-of-speech). They differ so much, in fact, that if we’re going to make comparisons between them, we need good, language-neutral definitions of grammatical categories.

Benjy_Forstadt 19 Feb 2026 0:31 UTC
4 points
0
in reply to: Seth Herd’s comment on: Why we should expect ruthless sociopath ASI
The claim isn’t that minds are safe and nice by default. It’s that they’re not sociopaths.

If in your view, most humans are basically ruthless sociopaths, then that’s good news, isn’t it? Sociopathic AIs would fit in well to our culture. It would mean our laws and norms do a remarkably good job of restraining us, so there’d be hope they’d do the same for future AIs.

Benjy_Forstadt 13 Feb 2026 18:23 UTC
56 points
22
on: Why You Don’t Believe in Xhosa Prophecies
According to Google, the Xhosa currently number between 9.2 and 9.6 million people, and are the second-largest ethnic group in South Africa. Looks like they’ve done fine for themselves.

It’s weird you would use them as an intuition pump for human extinction being plausible, as if they were some extinct tribe.

Maybe you’ll think I’m missing the main point of the essay, which is about what happens when memes or viruses jump species, but I don’t think that works either. This argument proves too much. You could just as well say that since most ideas in the world don’t need Xhosa brains to reproduce, they are not under selective pressure to keep the Xhosa alive, and so we should expect the world’s memeplex to evolve in a manner maximally incompatible with Xhosa survival.

Benjy_Forstadt 22 Sep 2025 6:33 UTC
6 points
8
on: This is a review of the reviews
I understand that when a person feels a lot is on the line it is often hard for that person to not come across as sanctimonious. Maybe it’s unfair of me, but that is how this comes across to me. Eg “people who allegedly care”.

Death with Dignity:

>Q2: I have a clever scheme for saving the world! I should act as if I believe it will work and save everyone, right, even if there’s arguments that it’s almost certainly misguided and doomed? Because if those arguments are correct and my scheme can’t work, we’re all dead anyways, right?
A: No! That’s not dying with dignity! That’s stepping sideways out of a mentally uncomfortable world and finding an escape route from unpleasant thoughts!”
This is a good insight about a possible reasoning mistake. Likewise, if more optimistic assumptions about AI are correct, you should not “step sideways” into an imaginary world where MIRI is right about everything “just to be safe”. Whatever problems come with AI need to be solved in the actual world, and in order to do that it is very very important to form good object-level beliefs about the problems

Benjy_Forstadt 19 Aug 2025 17:33 UTC
1 point
0
in reply to: bhauth’s comment on: SSC retrospective
>Do you like cinnamon when it’s not combined with sugar? If not, is it really cinnamon per se that you like?

How do you feel about butter

Benjy_Forstadt 26 Jul 2025 2:54 UTC
17 points
5
on: English writes numbers backwards
The order of our numeral notation mirrors the order of our spoken numerals. I’m not sure if there are any languages that consistently order additive numerals from smallest to largest—“two and fifty and three hundred” instead of “three hundred and fifty two”.
>💡Flipping the local ordering of pronunciation: If we’re truly optimizing, we might as well say “twenty and hundred-three” while we’re at it. The first words “and three-” don’t tell you much until you know “three of what”? Whereas “and hundred-three” tells you the order of magnitude as soon as possible.

This suggestion is aesthetically in tension with your principle of ordering from smallest to largest. Why should we go with informativeness for multiplication and small-to-large order for addition? The larger number in a sum is more informative about the size of the value, that is probably why languages tend to pronounce additive numerals from larger to smaller.
Interestingly, many languages actually do use the “hundred-three” order. You may be interested in this paper: https://www.nature.com/articles/s41599-023-02506-z, they have a striking geographic distribution.

Benjy_Forstadt 24 Jan 2025 5:41 UTC
2 points
−6
on: Do you consider perfect surveillance inevitable?
I don’t think perfect surveillance is inevitable.

I would prefer it, though. I don’t know any other way to prevent people from doing horrible things to minds running on their computers. It wouldn’t need to be publicly broadcast though, just overseen by law enforcement. I think this is much more likely than a scenario where everything you see is shared with everyone else.

Unfortunately, my mainline prediction is that people will actually be given very strong privacy rights, and will be allowed to inflict as much torture on digital minds under their control as they want. I’m not too confident in this though.

Benjy_Forstadt 23 Sep 2024 15:15 UTC
1 point
0
in reply to: Benjy_Forstadt’s comment on: Another argument against utility-centric alignment paradigms
Basically people tend to value stuff they perceive in the biophysical environment and stuff they learn about through the social environment.

So that reduces the complexity of the problem—it’s not a matter of designing a learning algorithm that both derives and comes to value human abstractions from observations of gas particles or whatever. That’s not what humans do either.

Okay then, why aren’t we star-maximizers or number-of-nation-states maximizers? Obviously it’s not just a matter of learning about the concept. The details of how we get values hooked up to an AGI’s motivations will depend on the particular AGI design but probably reward, prompting, scaffolding or the like.

Benjy_Forstadt 23 Sep 2024 6:40 UTC
1 point
0
in reply to: Thane Ruthenis’s comment on: Another argument against utility-centric alignment paradigms
I don’t think the way you split things up into Alpha and Beta quite carves things at the joints. If you take an individual human as Beta, then stuff like “eudaimonia” is in Alpha—it’s a concept in the cultural environment that we get exposed to and sometimes come to value. The vast majority of an individual human’s values are not new abstractions that we develop over the course of our training process (for most people at least).

Benjy_Forstadt 22 Sep 2024 23:21 UTC
13 points
4
in reply to: Thane Ruthenis’s comment on: Another argument against utility-centric alignment paradigms
There is a difference between the claim that powerful agents are approximately well-described as being expected utility maximizers (which may or may not be true) and the claim that AGI systems will have an explicit utility function the moment they’re turned on, and maximize that function from that moment on.

I think this is the assumption OP is pointing out: “most of the book’s discussion of AI risk frames the AI as having a certain set of goals from the moment it’s turned on, and ruthlessly pursuing those to the best of its ability”. “From the moment it’s turned on” is pretty important, because it rules out value learning as a solution

Benjy_Forstadt 6 Jun 2024 2:30 UTC
1 point
0
in reply to: Benjy_Forstadt’s comment on: MIRI 2024 Communications Strategy
Edit: Retracted because some of my exegesis of the historical seed AI concept may not be accurate

Benjy_Forstadt 2 Jun 2024 14:31 UTC
1 point
0
in reply to: quetzal_rainbow’s comment on: MIRI 2024 Communications Strategy
There will be future superintelligent AIs that improve themselves. But they will be neural networks, they will at the very least start out as a compute-intensive project, in the infant stages of their self-improvement cycles they will understand and be motivated by human concepts rather than being dumb specialized systems that are only good for bootstrapping themselves to superintelligence.

Benjy_Forstadt 1 Jun 2024 21:53 UTC
1 point
2
in reply to: quetzal_rainbow’s comment on: MIRI 2024 Communications Strategy
How does the question of whether AI outcomes are more predictable than AI trajectories reduce to the (vague) question of whether observations on current AIs generalize to future AIs?

Benjy_Forstadt 1 Jun 2024 15:27 UTC
38 points
5
in reply to: quetzal_rainbow’s comment on: MIRI 2024 Communications Strategy
To be blunt, it’s not just that Eliezer lacks a positive track record in predicting the nature of AI progress, which might be forgivable if we thought he had really good intuitions about this domain. Empiricism isn’t everything, theoretical arguments are important too and shouldn’t be dismissed. But-

Eliezer thought AGI would be developed from a recursively self-improving seed AI coded up by a small group, “brain in a box in a basement” style. He dismissed and mocked connectionist approaches to building AI. His writings repeatedly downplayed the importance of compute, and he has straw-manned writers like Moravec who did a better job at predicting when AGI would be developed than he did.

Old MIRI intuition pumps about why alignment should be difficult like the “Outcome Pump” and “Sorcerer’s apprentice” are now forgotten, it was a surprise that it would be easy to create helpful genies like LLMs who basically just do what we want. Remaining arguments for the difficulty of alignment are esoteric considerations about inductive biases, counting arguments, etc. So yes, let’s actually look at these arguments and not just dismiss them, but let’s not pretend that MIRI has a good track record.