TsviBT

Karma: 7,685

TsviBT 4 Jul 2025 14:32 UTC
16 points
16
in reply to: boazbarak’s comment on: A case for courage, when speaking of AI danger
The apparent aim of OpenAI (making AGI, even though we don’t know how to do so without killing everyone) is evil.

TsviBT 4 Jul 2025 0:26 UTC
42 points
28
in reply to: geoffreymiller’s comment on: A case for courage, when speaking of AI danger

Sharing our technical concerns about these abstract risks isn’t enough. We also have to morally stigmatize

I’m with you up until here; this isn’t just a technical debate, it’s a moral and social and political conflict with high stakes, and good and bad actions.

the specific groups of people imposing these risks on all of us.

To be really nitpicky, I technically agree with this as stated: we should stigmatize groups as such, e.g. “the AGI capabilities research community” is evil.

We need the moral courage to label other people evil when they’re doing evil things.

Oops, this is partially but importantly WRONG. From Braχot 10a:

With regard to the statement of Rabbi Yehuda, son of Rabbi Shimon ben Pazi, that David did not say Halleluya until he saw the downfall of the wicked, the Gemara relates: There were these hooligans in Rabbi Meir’s neighborhood who caused him a great deal of anguish. Rabbi Meir prayed for God to have mercy on them, that they should die. Rabbi Meir’s wife, Berurya, said to him: What is your thinking? On what basis do you pray for the death of these hooligans? Do you base yourself on the verse, as it is written: “Let sins cease from the land” (Psalms 104:35), which you interpret to mean that the world would be better if the wicked were destroyed? But is it written, let sinners cease?” Let sins cease, is written. One should pray for an end to their transgressions, not for the demise of the transgressors themselves.

Not everyone who is doing evil things is evil. Some people are evil. You should hate no more than necessary, but not less than that. You should hate evil, and hate evildoers if necessary, but not if not necessary.

Schmidhuber? Evil. Sutton? Evil. Larry Page? Evil. If, after reflection, you endorse omnicide, you’re evil. Altman? Evil and probably a sociopath.

Up-and-coming research star at an AI lab? Might be evil, might not be. Doing something evil? Yes. Is evil? Maybe, it depends.

Essentializing someone by calling them evil is an escalation of a conflict. You’re closing off lines of communication and gradual change. You’re polarizing things: it’s harder for that one person to make gradual moves in belief space and social space and life-narrative space, and it’s harder for groups to have group negotiations. Sometimes escalation is good and difficult and necessary, but sometimes escalation is really bad! Doing a more complicated subtle thing with more complicated boundaries is more difficult. And more brave, if we’re debating bravery here.

So:

Good:

The work you’re doing is evil.

Good:

The goal of this company is among the evilest possible goals ever.

Good:

If you ignore the world-class experts saying that your work might kill everyone, then you are being a disgusting creep and will be responsible for killing everyone.

Bad:

You’re a creep / evil / bad person.

Sidenote:

Because too many rationalists, EAs, tech enthusiasts, LessWrong people, etc still see those AI guys as ‘in our tribe’, based on sharing certain traits we hold dear

I agree that this is an improper motivation for treating some actions with kid gloves, which will lead to incorrect action; and that this is some of what’s actually happening.

TsviBT 3 Jul 2025 21:38 UTC
6 points
−1
on: Call for suggestions—AI safety course
As a category suggestion, I’d suggest including “obstructions/barriers” to making safe/aligned AGI, in the sense of the natural proofs barrier for P vs. NP. They aren’t going to be nearly as crisp in AGI safety/alignment as in math, but I still think it’d be valuable to present fundamental problems, that seem like they must be solved, as key parts of the field—or in other words, sets of approaches (such as ignoring those problems) that (most likely) won’t work.

Some examples are in “AGI Ruin: A List of Lethalities”, though not directly stated as barriers, e.g. “you only get one shot at making a safe fully-superhuman AI” could be stated as “solutions that require observing the behavior of a fully-superhuman AI don’t work [given such and such background conclusions]”.

(I also give examples in “The fraught voyage of aligned novelty”, though maybe not presented in a way you’d like. E.g. “if the AI is genuinely smarter than you, it’s thinking in ways that are alien to you” --> “solutions that require the AI to only think in ways that you already understand don’t work”.)

TsviBT 29 Jun 2025 19:33 UTC
8 points
1
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
I mean, they’re great as search engines or code-snippet writers (basically, search engine for standard functions). If someone thinks that gippities know stuff or can think or write well, that could be brainrotting.

TsviBT 29 Jun 2025 3:54 UTC
7 points
8
in reply to: habryka’s comment on: Habryka’s Shortform Feed
Well, like, if a company tried out some new robotics thing in one warehouse at a small scale in Q1, then in Q2 and Q3 scaled it up to most of that warehouse, and then in Q4 started work applying the same thing in another warehouse, and announced plans to apply to many warehouses, I think it’d be pretty fair to call this lasting adoption (of robotics, not LLMs, unless the robots use LLMs). On the other hand if they were stuck at the “small scale work trying to make a maybe-scalable PoC”, that doesn’t seem like lasting adoption, yet.

Judging this sort of thing would be a whole bunch of work, but it seems possible to do. (Of course, we can just wait.)

TsviBT 28 Jun 2025 21:24 UTC
24 points
13
in reply to: habryka’s comment on: Habryka’s Shortform Feed
Echoing robo’s comment:

Modest lasting corporate adoption

Has there been such adoption? Your remark

Every single corporation in the world is trying to adopt AI into their products. Even extremely slow-moving industries are rushing to adopt AI.

is about attempts to adopt, not lasting adoption. Of course, we can’t make “lasting adoption” mean “adopted for 5 years” if we’re trying to evaluate the prediction right now. But are you saying that there’s lots of adoption that seems probably/plausibly lasting, just by eyeballing it? My vague is impression is no, but I’m curious if the answer is yes or somewhat.

(TBC I don’t have a particularly strong prediction or retrodiction about adoption of AI in general in industry, or LLMs specifically (which is what I think Marcus’s predictions are about). At a guess I’d expect robotics to continue steadily rising in applications; I’d expect LLM use in lots of “grunt information work” contexts; and some niche strong applications like language learning; but not sure what else to expect.)

TsviBT 27 Jun 2025 23:53 UTC
2 points
0
in reply to: gwern’s comment on: Genomic emancipation
Interesting. (I don’t immediately see where you’re going with that, so sounds like I have something to learn!)

In practical terms, it should be feasible sooner to do small amounts of personality nudging using what data we already have, operating on linear variance. Later on we’ll have more data, better psychometrics, and better ways of modeling some of the nonlinear effects. My current take is that it’s better to use the weaker versions while the strong ones are infeasible (https://www.lesswrong.com/posts/rdbqmyohYJwwxyeEt/genomic-emancipation#Genomic_engineering_overhang), but not sure.

TsviBT 27 Jun 2025 6:12 UTC
3 points
0
in reply to: don't_wanna_be_stupid_any_more’s comment on: Genomic emancipation
(Had a good conversation—will think more about the request to ban research into certain personality traits (the traits that an oppressive regime could force upon its populace to enforce long-term subjugation).)

TsviBT 26 Jun 2025 21:08 UTC
3 points
0
in reply to: quetzal_rainbow’s comment on: TsviBT’s Shortform
That seems like part of the picture, but far from all of it. Manufactured stone tools have been around for well over 2 million years. That’s the sort of thing you do when you already have a significant amount of “hold weeks-long goal in mind long and strong enough that you put in a couple day’s effort towards it” (or something like that). Another example is Richard Alexander’s hypothesis: warfare --> strong pressure toward cognitive mechanisms for group-goal-construction. Neither of these are mainly about programmability (though the latter is maybe somewhat). I don’t think we see “random self-preserving terminal goals installed exogenously”, I think we see goals being self-constructed and then flung into long-termness.

TsviBT 26 Jun 2025 20:01 UTC
20 points
2
on: TsviBT’s Shortform
Humans are (weak) evidence for the instrumental utility for mind-designers to design terminal-goal-construction mechanisms.

Evolution couldn’t directly encode IGF into humans. So what was it supposed to do? One answer would be to make us vibe-machines: You bop around, you satisfy your immediate needs, you hang out with people, etc. And that is sort of what you are. But also there are the Unreasonable, who think and plan long-term, who investigate secrets, who build things for 10 or 100 or 1000 years—why? Maybe it’s because having terminal-like goals (here meaning, aims that are fairly fixed and fairly ambitious) is so useful that you want to have them anyway even if you can’t make them be the right ones. Instead you build machines to guess / make up terminal goals (https://tsvibt.blogspot.com/2022/11/do-humans-derive-values-from-fictitious.html).

TsviBT 25 Jun 2025 6:26 UTC
2 points
1
in reply to: mishka’s comment on: A regime-change power-vacuum conjecture about group belief
Ok, thanks for the info. (For the record, these do not sound like what I would remotely call “strokes of genius”.)

TsviBT 25 Jun 2025 3:45 UTC
2 points
0
in reply to: mishka’s comment on: A regime-change power-vacuum conjecture about group belief

I am seeing plenty of “strokes of genius” in LLMs, so the potential is there. They are not “dumb”, they have good creativity (in my experience).

Examples? What makes you think they are strokes of genius (as opposed to the thing already being in the training data, or being actually easy)?

TsviBT 25 Jun 2025 3:12 UTC
3 points
1
in reply to: mishka’s comment on: A regime-change power-vacuum conjecture about group belief

Even with the original GPT-4, asking for some rather non-standard things which I was unable to find online (e.g. immutable transformations of nested dictionaries in Python, so that I could compute JAX gradients with respect to variables located in the nodes of the trees in question) resulted in nice implementations (I had no idea how to even start doing that in Python, I only knew that trying to literally port my Clojure code doing the same thing would be a really bad idea).

Ok cool, but this is just not the sort of thing we’re supposed to be talking about. We’re supposed to be talking about novel ideas about minds / intelligence, and then implementing them in a way that really tests the hypothesis.

It’s obvious that LLMs are more than “half-way there”,

Go ahead and make an argument then.

TsviBT 25 Jun 2025 2:44 UTC
3 points
0
in reply to: mishka’s comment on: A regime-change power-vacuum conjecture about group belief

But if I can ask them to implement my ideas end-to-end in prototype code and to conduct computational experiments for me end-to-end, my personal velocity would grow an order of magnitude, if not more...

If your ideas are things that are commonly implemented, then sure—but then they are much much less likely to be novel ideas progressing towards AGI. If they are novel ideas, it’s much much harder for current LLMs to rightly implement them.

It’s not that people don’t have good AGI ideas, it’s more that the road from a promising idea to a viable implementation is difficult,

No I think it’s the first one.

TsviBT 25 Jun 2025 2:22 UTC
3 points
0
in reply to: mishka’s comment on: A regime-change power-vacuum conjecture about group belief
I’m not talking about a slightly different model, I’m talking about having ideas. I don’t see clearly how you use LLMs including reasoning models to have good AGI ideas much faster. I also don’t think anyone else does, they’re just vaguely thinking there might be.

TsviBT 25 Jun 2025 0:10 UTC
4 points
2
in reply to: mishka’s comment on: A regime-change power-vacuum conjecture about group belief
I agree that you can speed up [the research that is ratcheting up to AGI] like this. I don’t agree that it is LIKELY that you speed [the research that is ratcheting up to AGI] up by A LOT (>2x, IDK). I’m not saying almost anything about “how fast OpenAI and other similar guys can do whatever they’re currently doing”. I don’t think that’s very relevant, because I think that stuff is pretty unlikely (<4%) to lead to AGI by 2030.

Why not likely? Because
- The actual ideas aren’t things that you can just crank out at 10x the rate if you have 10x the spare time. They’re the sort of thing that you usually get 0 of in your life, 1 if you’re lucky, several if you’re an epochal genius. They involve deep context and long struggle. (Probably. Just going off of history of ideas. Some random prompt or o5-big-thinky-gippity-tweakity-150128 could kill everyone. But <4%.)
- OpenAI et al. aren’t AFAIK working on that stuff. If you work on not-the-thing 3x faster, so what?

A regime-change power-vacuum conjecture about group belief

TsviBT24 Jun 2025 23:16 UTC

41 points

15 comments3 min readLW link

TsviBT 24 Jun 2025 22:11 UTC
8 points
2
on: Consider chilling out in 2028
A couple remarks:

I’m skeptical of the the trauma model. I think personally I’m more grim and more traumatized than most people around here, and other people are too happy and not neurotic enough to understand how fucked alignment research is; and yet I have much longer timelines than most people around here. (Not saying anyone should become less happy or more neurotic lol.)

Re/ poor group epistemology, we’ll agree that the overall phenomenon is multifactorial. Partially it’s higher-order emergent deference. See “Dangers of deference”.

I agree that people want something with their short timelines. But my guese would be less about trauma, and more about community. This sort of echoes what you say about a (lack of a) positive vision. I would say that making positive visions is a fundamental gap in the modern world, rationalists have and feel that gap, and consequently they feel badly. And they feel some taste of shared vision from being all worried about short timelines together. The strong undercurrent powering all this is: huddling around the fire, in the big open dark night. If you work on things that other people aren’t working on, you don’t get to live in the world with other people as much. If you “believe” things that other people don’t believe, then naturally you won’t work on things that other people are working on. If you explicitly point this out as a driver of “belief”, usually people will be like “ah, yeah, true” but then go back to pretending that their beliefs come from reasons that are about the propositional content.

I agree that the lack of a shared positive vision is salient. Part of mine is in “Genomic emancipation”, and I discuss some of the general reasons that having a positive shared vision matters in the appendix “Why envision genomic emancipation?”.
What links here?
- A regime-change power-vacuum conjecture about group belief by TsviBT (24 Jun 2025 23:16 UTC; 41 points)

TsviBT 24 Jun 2025 17:16 UTC
2 points
0
in reply to: don't_wanna_be_stupid_any_more’s comment on: Genomic emancipation

still 40% is pretty bad especially when you control the environment so i am still worried, getting bad actors 40% of the way there is a bad idea.

Ok, but that study, at a glance, seems to be about personality models in the vein of Big Five (https://en.wikipedia.org/wiki/Big_Five_personality_traits). This matches how I interpreted your discussion of “personality”, which you later said actually you meant to discuss “values”—are you now trying to discuss personality, or values? I though you said you’re ok with genomically vectoring personality but not values?

TsviBT 23 Jun 2025 19:32 UTC
4 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
Another aspect of this is that we don’t have a good body of obstructions (they are clear, they are major in that they defeat many proposals, people understand and apply them, they are widely-enough agreed on).

https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=CzibNhsFf3RyxHpvT

TsviBT

A regime-change power-vac­uum con­jec­ture about group belief

A regime-change power-vacuum conjecture about group belief