rvnnt

Karma: 119

rvnnt 17 Jun 2023 8:51 UTC
13 points
6
in reply to: Richard_Ngo’s comment on: The ones who endure
Right; that would be a silly thing to think.

My intended message might’ve been better worded as follows

If staring into abysses is difficult/rough, then adequately staring into the darker abysses might require counter-intuitively large amounts of effort/agency. And yet, I think it might be necessary to grok those darker abysses, if we are to avoid falling into them. That makes me worried.

OTOH, you seem exceptionally reflective, so perhaps that worry is completely unfounded in your case. Anyway, I’m grateful for the work you do; I wish there were more people like you in this world. (Also, your attention is extremely valuable, so please feel free to promptly drop/forget this conversation.)

rvnnt 28 Feb 2023 11:51 UTC
12 points
5
on: A case for capabilities work on AI as net positive
I really wish there was an “agree/disagree” button for posts. I’d like to upvote this post (for ~~epistemic virtue /~~ presenting reasonable “contrarian” views and explaining why one holds them), but I also strongly disagree with the conclusions and suggested policies. (I ended up voting neither up nor down.)

EDIT: After reading Akash’s comments, and re-reading the post more carefully: I largely agree with Akash (and updated towards thinking that my standards for “epistemic virtue” are/were too low).

rvnnt 26 Feb 2023 14:10 UTC
10 points
6
in reply to: Kaj_Sotala’s comment on: The Preference Fulfillment Hypothesis
I’m guessing this might be due to something like the following:
- (There is a common belief on LW that) Most people do not take AI x/s-risk anywhere near seriously enough; that most people who do think about AI x/s-risk are far too optimistic about how hard/easy alignment is; that most people who do concede >10% p(doom) are not actually acting with anywhere near the level of caution that their professed beliefs would imply to be sensible.
- If alignment indeed is difficult, then (AI labs) acting based on optimistic assumptions is very dangerous, and could lead to astronomical loss of value (or astronomical disvalue)
- Hence: Pushback against memes suggesting that alignment might be easy.
I think there might sometimes be something going on along the lines of “distort the Map in order to compensate for a currently-probably-terrible policy of acting in the Territory”.

Analogy: If, when moving through Territory, you find yourself consistently drifting further east than you intend, then the sane solution is to correct how you move in the Territory; the sane solution is not to skew your map westward to compensate for your drifting. But what if you’re stuck in a bus steered by insane east-drifting monkeys, and you don’t have access to the steering wheel?

Like, if most people are obviously failing egregiously at acting sanely in the face of x/s-risks, due to those people being insane in various ways

(“but it might be easy!”, “this alignment plan has the word ‘democracy’ in it, obviously it’s a good plan!”, “but we need to get the banana before those other monkeys get it!”, “obviously working on capabilities is a good thing, I know because I get so much money and status for doing it”, “I feel good about this plan, that means it’ll probably work”, etc.),

then one of the levers you might (subconsciously) be tempted to try pulling is people’s estimate of p(doom). If everyone were sane/rational, then obviously you should never distort your probability estimates. But… clearly everyone is not sane/rational.

If that’s what’s going on (for many people), then I’m not sure what to think of it, or what to do about it. I wish the world were sane?

rvnnt 16 Jun 2023 17:18 UTC
8 points
7
on: The ones who endure
When people call things like this post “rough to write/read”, and consider them to require a content warning, I wonder if most people are able to think clearly (or at all) about actually terrible scenarios, and worry that they aren’t. (I’m especially worried if those people have influence in a domain where there might be a tradeoff between mitigating X-risks vs mitigating S-risks.)

I liked the description of the good future, though. Thanks for the reminder that things can (maybe) go well, too.

rvnnt 26 Feb 2023 14:23 UTC
7 points
2
in reply to: MSRayne’s comment on: The Preference Fulfillment Hypothesis
Some voice in my head started screaming something like

“A human introspecting on human phenomenology does not provide reliable evidence about artificial intelligences! Remember that humans have a huge amount of complex built-in circuitry that operates subconsciously and makes certain complicated things—like kindness—feel simple/easy!”

Thought I’d share that. Wondering if you disagree with the voice.

rvnnt 12 Oct 2023 9:09 UTC
6 points
0
on: Evolution Solved Alignment (what sharp left turn?)

evolution did in fact find some weird way to create humans who rather obviously consciously optimize for IGF! [...]

If Evolution had a lot more time to align humans to relative-gene-replication-count, before humans put an end to biological life, then sure, seems plausible that Evolution might be able to align humans very robustly. But Evolution does not have infinite time or “retries”—humanity is in the process of executing something like a “sharp left turn”, and seems likely to succeed long before the human gene pool is taken over by sperm bank donors and such.

rvnnt 24 Apr 2023 15:23 UTC
5 points
2
on: Contra Yudkowsky on AI Doom
This post raised some interesting points, and stimulated a bunch of interesting discussion in the comments. I updated a little bit away from foom-like scenarios and towards slow-takeoff scenarios. Thanks. For that, I’d like to upvote this post.

On the other hand: I think direct/non-polite/uncompromising argumentation against other arguments, models, or beliefs is (usually) fine and good. And I think it’s especially important to counter-argue possible inaccuracies in key models that lots of people have about AI/ML/alignment. However, in many places, the post reads like a personal attack on a person (Yudkowsky), rather than just on models/beliefs he has promulgated.

I think that style of discourse runs a risk of
- politicizing the topic under discussion, and thereby making it harder for people to think clearly about it
- creating a shitty culture where people are liable to get personally attacked for participating in that discourse
For that, I’d like to downvote this post. (I ended up neither up- nor down-voting.)

rvnnt 7 Mar 2023 13:11 UTC
5 points
2
in reply to: Fergus Fettes’s comment on: Are we too confident about unaligned AGI killing off humanity?
Why not both? Why leave value lying around? (Also, the asteroid belt containing Ceres and Vesta contains several orders of magnitude less matter than Earth. Maybe you meant “why not go colonize the Milky Way and other galaxies”?)

rvnnt 6 Dec 2022 13:48 UTC
5 points
1
on: Who are some prominent reasonable people who are confident that AI won’t kill everyone?
Meta: I agree that looking at arguments for different sides is better than only looking at arguments for one side; but

[...] neutralizing my status-yuck reaction. One promising-seeming approach is to spend a lot of time looking at lots of of high-status monkeys who believe it!

sounds like trying to solve the problem by using more of the problem? I think it’s worth flagging that {looking at high-status monkeys who believe X} is not addressing the root problem, and it might be worth spending some time on trying to understand and solve the root problem.

I’m sad to say that I myself do not have a proper solution to {monkey status dynamics corrupting ability to think clearly}. That said, I do sometimes find it helpful to thoroughly/viscerally imagine being an alien who just arrived on Earth, gained access to rvnnt’s memories/beliefs, and is now looking at this whole Earth-circus from the perspective of a dispassionately curious outsider with no skin in the game.

If anyone has other/better solutions, I’d be curious to hear them.

rvnnt 21 Mar 2024 14:46 UTC
4 points
0
on: AI #56: Blackwell That Ends Well
Referring to the quote-picture from the Nvidia GTC keynote talk: I searched the talk’s transcript, and could not find anything like the quote.

Could someone point out time-stamps of where Huang says (or implies) anything like the quote? Or is the quote entirely made up?

rvnnt 23 Apr 2023 17:20 UTC
4 points
1
on: How did LW update p(doom) after LLMs blew up?

I will bet any amount of money that GPT-5 will not kill us all.

What’s the exchange rate for USD to afterlife-USD, though? Or what if they don’t use currency in the afterlife at all? Then how would you pay the other party back if you lose?

rvnnt 31 Dec 2023 12:29 UTC
3 points
0
on: Conversation Visualizer
I wonder how much work it’d take to implement a system that incrementally generates a graph of the entire conversation. (Vertices would be sub-topics, represented as e.g. a thumbnail image + a short text summary.) Would require the GPT to be able to (i.a.) understand the logical content of the discussion, and detect when a topic is revisited, etc. Could be useful for improving clarity/productivity of conversations.

rvnnt 9 Nov 2023 10:14 UTC
3 points
0
in reply to: orthonormal’s comment on: Vote on Interesting Disagreements
That makes sense; but:

so far outside the realm of human reckoning that I’m not sure it’s reasonable to call them dystopian.

setting aside the question of what to call such scenarios, with what probability do you think the humans^[1] in those scenarios would (strongly) prefer to not exist?
1. ↩︎
  or non-human minds, other than the machines/Minds that are in control

rvnnt 16 Oct 2023 15:20 UTC
3 points
2
in reply to: jacob_cannell’s comment on: Evolution Solved Alignment (what sharp left turn?)
Humans have not put an end to biological life.

Yup. I, too, have noticed that.

Your doom^[1] predictions [...]

C’mon, man, that’s obviously a misrepresentation of what I was saying. Or maybe my earlier comment failed badly at communication? In case that’s so, here’s an attempted clarification (bolded parts added):

If Evolution had a lot more time (than I expect it to have) to align humans to relative-gene-replication-count, before humans put an end to biological life , as they seem to me to be on track to do, based on things I have observed in the past, then [...] But Evolution (almost surely) does not have infinite time [...]

Point being: Sure, Evolution managed to cough up some individuals who explicitly optimize for IGF. But they’re exceptions, not the rule; and humanity seems (based on past observations!) to be on track to (mostly) end DNA-based life. So it seems premature to say that Evolution succeeded at aligning humanity.

In case you’re wondering what past observations lead me to think that humans are unaligned^[2] w.r.t. IGF and on track to end (or transcend) biological life, here are some off the top of my head:
- Of the people whose opinions on the subject I’m aware of (including myself), nearly all would like to transcend (or end) biological life.^[3]
- Birth rates in most developed nations have been low or below replacement for a long time.^[4] There seems to be a negative correlation between wealth/education and number of offspring produced. That matches my impression that as people gain wealth, education, and empowerment in general, most choose to spend it mostly on something other than producing offspring.
- Diligent sperm bank donors are noteworthy exceptions. Most people are not picking obvious low-hanging fruit to increasing their IGF. Rich people waste money on yachts and stuff, instead of using it to churn out as many high-fitness offspring as possible; etc.
- AFAIK, most of the many humans racing to build ASI are not doing so with the goal of increasing their IGF. And absent successful attempts to align ASI specifically to producing lots of DNA-based replicators, I don’t see strong reason to expect the future to be optimized for quantity of DNA-based replicators.
Perhaps you disagree with the last point above?

There are many other scenarios where DNA flourishes even after a posthuman transition.

Interesting. Could you list a few of those scenarios?
1. ↩︎
  Note: I wasn’t even talking (only) about doom; I was talking about humanity seemingly being on track to end biological life. I think the “good” outcomes probably also involve transcending biology/DNA-based replicators.
2. ↩︎
  to the extent that it even makes sense to talk about incoherent things like humans being “(mis/un)aligned” to anything.
3. ↩︎
  My sample might not be super representative of humanity as a whole. Maybe somewhat representative of people involved in AI, though?
4. ↩︎
  At least according to sources like this: https://en.wikipedia.org/wiki/Total_fertility_rate

rvnnt 12 Oct 2023 9:05 UTC
3 points
1
on: Evolution Solved Alignment (what sharp left turn?)
The utility function is fitness: gene replication count (of the human defining genes) ^[1]

Seems like humans are soon going to put an end to DNA-based organisms, or at best relegate them to some small fraction of all “life”. I.e., seems to me that the future is going to score very poorly on the gene-replication-count utility function, relative to what it would score if humanity (or individual humans) were actually aligned to gene-replication-count.

Do you disagree? (Do you expect the post-ASI future to be tiled with human DNA?)
1. ↩︎
  Obviously Evolution doesn’t actually have a utility function, and if it did, gene-replication-count is probably not it, as TekhneMakre points out. But, let’s accept that for now, arguendo.

rvnnt 19 Jul 2023 10:21 UTC
3 points
2
on: Simple alignment plan that maybe works
Thumbs up for trying to think of novel approaches to solving the alignment problem.

Every time the model does something that harms the utility function of the dumber models, it gets a loss function.

A few confusions:
- By “it gets a loss function”, did you mean “it gets negative reward”?
- If yes, doesn’t this plan consist entirely of reinforcement learning? How does this “emulate Evolution”?
- What exactly does the quoted sentence mean? Does the smarter model (S) receive RL signals proportional to… changes in the dumber agents’ (D’s) total utility?
Some problems, off the top of my head:
- GPT-like models don’t have utility functions.
- Even if they did, mechinterp is nowhere near advanced enough to be able to reveal models’ utility functions.
- Humans don’t have utility functions. It’s unclear how this would generalize to human-alignment.
- It’s very much unclear what policy S would end up learning in this RL setup. It’s even less clear how that policy would generalize outside of training.
  - If S is given reward proportional to (changes in) D’s utility, then basically we’re just training S with D’s utility function. I.e., just training some arbitrary RL policy/agent. Not much to do with alignment, AFAICT. ^[1]
  - If S is instead given reward for things like {taking actions that lead to obtaining information about D’s utility function}, then… we’re training an RL policy/agent on proxies to “alignment”. I expect that kind of approach to break down badly (due to Goodhart) when S becomes highly capable.
I don’t know how you arrived at this plan, but I’m guessing it involved reasoning with highly abstract and vague concepts. You might be interested in (i.a.) these tools/techniques:
- https://www.lesswrong.com/posts/GKfPL6LQFgB49FEnv/replace-the-symbol-with-the-substance
- https://www.lesswrong.com/posts/JcpzFpPBSmzuksmWM/the-5-second-level
1. ↩︎
  Except maybe if you somehow managed to have the entire simulation be a very accurate model of the real world, and D’s be very accurate models of humans. But that’s not remotely realistic; and still subject to Goodhart.

rvnnt 15 Jun 2023 10:37 UTC
3 points
0
on: Instrumental Convergence? [Draft]

Firstly, a bias towards choices which leave less up to chance.

Wouldn’t this imply a bias towards eliminating other agents? (Since that would make the world more predictable, and thereby leave less up to chance?)

And thirdly, a bias towards choices which afford more choices later on.

Wouldn’t this strongly imply biases towards both self-preservation and resource acquisition?

If the above two implications hold, then the conclusion

that the biases induced by instrumental rationality at best weakly support [...] that machine superintelligence is likely to lead to existential catastrophe

seems incorrect, no?

Could you briefly explain what is wrong with the reasoning above, or point me to the parts of the post that do so? (I only read the Abstract.)

rvnnt 28 Apr 2023 7:31 UTC
3 points
0
on: My views on “doom”

[...] bad futures without extinction, e.g. that AI systems take over but don’t kill everyone.

What probability would you assign to humans remaining but not being able to kill themseleves; i.e., to unescapable dystopias (vs. dystopias whose badness for any individual are bounded by death-by-suicide)?

rvnnt 5 Apr 2023 11:30 UTC
3 points
0
on: Giant (In)scrutable Matrices: (Maybe) the Best of All Possible Worlds

Well, I ask you—do you think any other ML model, trained over the domain of all human text, with sufficient success to reach GPT-4 level perplexity, would turn out to be simpler?

If we’re literally considering a universal quantifier over the set of all possible ML models, then I’d think it extremely likely that there does exist a simpler model with perplexity no worse than GPT-4. I’m confused as to how you (seem to have) arrived at the opposite conclusion.

rvnnt 5 Apr 2023 11:28 UTC
3 points
0
on: Giant (In)scrutable Matrices: (Maybe) the Best of All Possible Worlds
Imagine an astronomer in the year 1600, who frequently refers to the “giant inscrutable movements” of the stars. [...]

I think the analogy to {building intelligent systems} is unclear/weak. There seem to be many disanalogies:

In the astronomy case, we have
- A phenomenon (the stars and their motions) that we cannot affect
- That phenomenon is describable with simple laws.
- The “piles of numbers” are detailed measurements of that phenomenon.
- It is useful to take more measurements, doing so is helpful for finding the simple laws.
In the {building AIs} case, we have
- A phenomenon (intelligence) which we can choose to implement in different ways, and which we want to harness for some purpose.
- That phenomenon is probably not entirely implementable with simple programs. (As you point out yourself.)
- The “piles of numbers” are the means by which some people are implementing the thing; as opposed to measurements of the one and only way the thing is implemented.
- And so: implementing AIs as piles-of-numbers is not clearly helpful (and might be harmful) to finding better/simpler alternatives.