satchlj

Karma: 132

satchlj May 28, 2025, 3:08 PM
3 points
0
in reply to: RogerDearnaley’s comment on: Can We Naturalize Moral Epistemology?
This comment does really help me understand what you’re saying better. If you write a post expanding it, I would encourage you to address the following related points:
- Can you have some members of a society who don’t share some of the consistent moral patterns which evolved, or do you claim that every member reliably holds these morals?
- Can someone decide what they ought to value using this system? How?
- Is it wrong if someone simply doesn’t care about what society values? Why?
- How can we tell that your story tells us what we ought to value rather than simply explaining why we value the things we do?
- Do you make a clear distinction between normative ethics and descriptive ethics? What is it?

satchlj May 26, 2025, 2:33 PM
9 points
1
in reply to: RogerDearnaley’s comment on: Can We Naturalize Moral Epistemology?
Thanks for explaining.
So to discuss “what we ought to value” you need to judge moral systems and their consequences using something that is both vaguer and more practical than a moral system. Such as psychology, or sociology, or political expedience, or some combination of these.
I think this is tempting but ultimately misguided, because the choice of a ‘more practical and vague’ system by which to judge moral systems is just a second order moral system in itself which happens to be practical and vague. This is metanormative regress.
The only coherent solution to the “ought-from-is” problem I’ve come across is normative eliminativism - ‘ought’ statements are either false or a special type of descriptive statement.

satchlj May 23, 2025, 1:56 PM
10 points
1
in reply to: RogerDearnaley’s comment on: Can We Naturalize Moral Epistemology?
Evolutionary ethics aims to help people understand why we value the things we do. It doesn’t have the ability to say anything about what we ought to value.

satchlj May 22, 2025, 4:17 AM
8 points
0
on: Can We Naturalize Moral Epistemology?
What’s the state of existing empirical evidence on whether Moral Reasoning is Real?
My own observations tell me that it is not. Certainly, some people engage in moral reasoning and are satisfied with their results to varying degrees, but it appears to me that this is a small proportion of humans.
My preliminary investigation into the research confirms my existing belief that most moral reasoning is post-hoc, and that while human values can change it is almost never due to reasoned arguments and instead a social and emotional process. When moral reasoning seems to work, endorsement is often shallow and attitudes can revert within days.
I am frequently reminded that I underestimate the degree to which my own view on this is not universally held, however.

satchlj May 7, 2025, 4:15 PM
1 point
0
in reply to: gwern’s comment on: satchlj’s Shortform
Likely it’s not great news, like I said in the original post I am not sure how to interpret what I had noticed. But right now, after the reflection that’s come from people’s comments, I don’t think it’s bad news, and it might (maybe) even be slightly good news.^[1]
Could you explain to me where the single step / multiple steps aspect comes in? I don’t see an assumption of only a single step anywhere, but maybe this comes from a lack of understanding.
1. ^
  Instead of a world where we need our proxy to be close to 100% error-free (this seems totally unrealistic), we just need the error to have ~ no tails (this might also be totally unrealistic but is a weaker requirement).

satchlj May 6, 2025, 11:16 PM
1 point
0
in reply to: gwern’s comment on: satchlj’s Shortform
I’m not sure I understand what you’re getting at.
If $U (X)$ is the target utility function and $V (X)$ is the proxy, and $X^{*}$ is $arg max V (X)$ then we calculated expected value of $max U - U (X^{*})$ assuming $| V - U |$ is normally distributed over $X$ .
See this other comment.
It seems people have already explored this in depth. I like John Wentworth’s slogan that Goodheart is about generalization, not approximation.

satchlj May 6, 2025, 8:18 PM
1 point
0
in reply to: plex’s comment on: satchlj’s Shortform
Thanks, yeah Gaussian error is a strong assumption and usually not the default. But it’s intuitively a much more realistic target than ~no error, and we want to understand how much error we can tolerate.

satchlj May 6, 2025, 2:50 AM
2 points
1
in reply to: Vladimir_Nesov’s comment on: Why I am not a successionist
The question of ‘what should replace humanity and how should this occur’ isn’t something we can answer with a patchwork of different microworlds.

satchlj May 5, 2025, 8:21 PM
1 point
0
in reply to: Elizabeth’s comment on: Elizabeth’s Shortform
Inside of Google Docs you mean? Yeah, for some subset of tasks the ‘select > menu click > Refine the selected text’ tool has been useful.

satchlj May 5, 2025, 8:00 PM
7 points
2
on: satchlj’s Shortform
Is Goodhart’s Curse Not Really That Bad?
EDIT: It’s bad. Still, it’s good to understand exactly when it’s bad.
I’m not implying I’m on to anything others haven’t thought of by posting this—I’m asking this so people can tell me if I’m wrong.
Goodhart’s Curse is often cited to claim that if a superintelligent AI has a utility function which is a noisy approximation of the intended utility function, the expected proxy error will blow up given a large search space for the optimal policy.
But, assuming Gaussian or sub-Gaussian error, the expected regret is actually something like $σ \sqrt{2 log n}$ where $n$ is the size of the raw search space. Even if search space grows exponentially with intelligence, expected error isn’t really blowing up. If smarter agents make more accurate proxies, then error might very plausibly decrease as intelligence grows.
I understand that there are a lot of big assumptions here which might not hold in practice, but this still seems to suggest there are a lot of worlds where Goodhart’s Curse doesn’t bite that hard.
If this is too compressed to be legible, please let me know and I will make it a full post.

satchlj May 5, 2025, 4:20 PM
12 points
5
on: Why I am not a successionist
why is your definition of value so arbitrary as to stipulate that biological meat-humans are necessary
In this kind of conversation, it’s important that different people will inevitably want fundamentally different things in a way that cannot be fully reconciled.
Most people recognize that humans in their current form will not exist forever, but our preferences about what comes next are really varied. Many people want to minimize change and want the future to be something which they recognize and understand. Many others see this kind of future as both unrealistic and undesirable, and focus on caring about the manner in which we are replaced, or preserving certain qualities of humans in our successors.

Unfortunately, it’s common to assume that there is some correct thing to value, or that a certain class of values (e.g. evolutionary preferences) are well justified or widely accepted.

This isn’t a productive framework.

Perhaps I am overly pessimistic, but I see preferences as so varied on this question that I’d guess there is no possible outcome that is desirable to more than half the population.

satchlj Apr 2, 2025, 7:29 PM
1 point
0
in reply to: Jonas Hallgren’s comment on: Is instrumental convergence a thing for virtue-driven agents?
This doesn’t make complete sense to me, but you are going down a line of thought I recognize.
There are certainly stable utility functions which, while having some drawbacks, don’t result in dangerous behavior from superintelligences. Finding a good one doesn’t seem all that difficult.
The real nasty challenge is how to build a superintelligence that has the utility function we want it to have. If we could do this, then we could start by choosing an extremely conservative utility function and slowly and cautiously iterate towards a balance of safe and useful.

satchlj Apr 2, 2025, 7:17 PM
1 point
0
on: Is instrumental convergence a thing for virtue-driven agents?
I’ve been thinking about a similar thing a lot.
Consider a little superintelligent child who always wants to eat as much candy as possible over the course of the next ten minutes. Assume the child doesn’t ever care about what happens ten minutes from now.
This child won’t work very hard at any instrumental goals like self improvement and conquering the world to redirect resources towards candy production, since that would be a waste of time, even though it might maximize candy consumption in the long term.
AI alignment isn’t any easier here, the point of this is just to illustrate that instrumental convergence is far from given.

satchlj Apr 2, 2025, 12:13 AM
28 points
1
in reply to: Gurkenglas’s comment on: VDT: a solution to decision theory
Claude says the vibes are ‘inherently cursed’
But then it chooses not to pull the lever because it’s ‘less karmically disruptive’

Williamstown – ACX Meetups Everywhere Spring 2025

satchljMar 25, 2025, 11:49 PM

2 points

0 comments1 min readLW link

satchlj Mar 18, 2025, 8:06 PM
3 points
0
in reply to: Fiora Sunshine’s comment on: Against Yudkowsky’s evolution analogy for AI x-risk [unfinished]
Note that if computing an optimization step reduces the loss, the training process will reinforce it, even if other layers aren’t doing similar steps, so this is another reason to expect more explicit optimizers.
Basically, self attention is a function of certain matrices, something like this:
$e_{j} \leftarrow e_{j} + \sum_{h} P_{h} W_{h, V} \sum_{i} e_{h, i} \otimes e_{h, i} W_{h, K}^{T} W_{h, Q} e_{j}$
Which looks really messy when you put it like this but is pretty natural in context.
If you can get the big messy looking term to approximate a gradient descent step for a given loss function, then you’re golden.
In appendix A.1., they show the matrices that yield this gradient descent step. They are pretty simple, and probably an easy point of attraction to find.
All of this reasoning is pretty vague, and without the experimental evidence it wouldn’t be nearly good enough. So there’s definitely more to understand here. But given the experimental evidence I think this is the right story about what’s going on.

satchlj Mar 18, 2025, 6:57 PM
10 points
0
on: Against Yudkowsky’s evolution analogy for AI x-risk [unfinished]
I think you do this post a disservice by presenting it as a failure. It had a wrong conclusion, but its core arguments are still interesting and relevant, and exploring the reasons they are wrong is very useful.
Your model of neural nets predicted the wrong thing, that’s super exciting! We can improve the model now.

satchlj Mar 18, 2025, 6:51 PM
3 points
0
in reply to: Fiora Sunshine’s comment on: Against Yudkowsky’s evolution analogy for AI x-risk [unfinished]
The fundamental idea about genes having an advantage over weights at internally implementing looping algorithms is apparently wrong though (even though I don’t understand how the contrary is possible...)
I’ve been trying to understand this myself. Here’s a the understanding I’ve come to, which is very simplistic. If someone who knows more about transformers than me says I’m wrong I will defer to them.
I used this paper to come to this understanding.
In order to have a mesa-optimizer, lots and lots of layers need to be in on the game of optimization, rather than just one or several key elements which gets referenced repeatedly during the optimization process.
But self-attention is, by default, not very far away from being one step in gradient descent. Every layer doesn’t need to learn to do optimization independently from scratch, since it’s relatively easy to find given the self-attention architecture.
That’s why it’s not forbiddingly difficult for neural networks to implement internal optimization algorithms. It still could be forbiddingly difficult for most optimization algorithms, ones that aren’t easy to find from the basic architecture.

satchlj Mar 16, 2025, 1:37 AM
13 points
36
on: Help make the orca language experiment happen
Why don’t I do the project myself?
Because I think I’m one of the smartest young supergeniuses, and I’m working on things that I think are even more useful in expectation, and which almost nobody except me can do.
Even if this is by some small chance actually true, it’s stupid of you to say it, because from the perspective of your readers, you are almost certainly wrong and so you undermine your own credibility. I’m sure you were aware some people would think this, and don’t care. Have you experimented with trying not to piss people off and see if it helps you?
As for your actual idea, it’s cool and even if it doesn’t work out we could learn some important things. Good luck!

satchlj Mar 7, 2025, 3:48 PM
7 points
0
on: A Bear Case: My Predictions Regarding AI Progress
Why do you think Anthropic and OpenAI are making such bold predictions? (https://x.com/kimmonismus/status/1897628497427701961)
As I see it, one of the following is true:
1. They agree with you but want shape the narrative away from the truth to sway investors
2. They have mostly the same info as you but come to a different conclusion
3. They have evidence we don’t have which gives them confidence

satchlj

Willi­am­stown – ACX Mee­tups Every­where Spring 2025

Why don’t I do the project myself?

Williamstown – ACX Meetups Everywhere Spring 2025