Satron

Karma: 232

Satron Jan 22, 2025, 4:55 PM
1 point
0
in reply to: joshcohen’s comment on: The Field of AI Alignment: A Postmortem, and What To Do About It
It was pointed out, e.g. [here](https://web.archive.org/web/20170918044233/http://files.openphilanthropy.org/files/Grants/MIRI/consolidated_public_reviews.pdf) but unfortunately the mindset is self-reinforcing so there wasn’t much to be done.
Your link formatting got messed up here.

Satron Jan 21, 2025, 9:51 PM
5 points
0
on: How do you decide to phrase predictions you ask of others? (and how do you make your own?)
More generally when you’re asked a question like “How much do you expect [politician’s name] to win/lose by?” how do you go about generating your response? How do you generate your predictions?
My usual response would look something like this:
“Pre-scandal polls are showing Bodee leading by about 1-2%, but given the recent spying scandal and the strong emphasis of our society on ethical conduct, I expect him to lose by 3-4%.”
If I am speaking to a person who (on my expectations) is following the polls, I would try to explain why my prediction differs from them.

Satron Jan 20, 2025, 8:21 AM
1 point
0
in reply to: Remmelt’s comment on: What do you mean with ‘alignment is solvable in principle’?
But you seem to mean something broader with “possible worlds.” Something like “in theory, there is a physically possible arrangement of atoms/energy states that would result in an ‘aligned’ AGI, even if that arrangement of states might not be reachable from our current or even a past world.”
–> Am I interpreting you correctly?
Yup, that’s roughly what I meant. However, one caveat would be that I would change “physically possible” to “metaphysically/logically possible” because I don’t know if worlds with different physics could exist, whereas I am pretty sure that worlds with different metaphysical/logical laws couldn’t exist. By that, I mean stuff like the law of non-contradiction and “if a = b, then b = a.”
You saying this shows the ambiguity here of trying to understand what different people mean. One researcher can make a technical claim about the possibility/tractability of “alignment” that is similarly worded to a technical claim others made. Yet their meaning of “alignment” could be quite different.
It’s hard then to have a well-argued discussion, because you don’t know whether people are equivocating (ie. switching between different meanings of the term).
I think the main antidote against this is to ask the person you are speaking with to define the term if they are making claims in which equivocation is especially likely.
The way I deal with the wildly varying uses of the term “alignment” is to use a minimum definition that most of those six interpretations are consistent with
Yeah, that’s reasonable.

Satron Jan 20, 2025, 12:06 AM
10 points
0
in reply to: Siebe’s comment on: Jonathan Claybrough’s Shortform
Listeners would understandably infer that “these well-informed people apparently don’t really worry much, maybe I shouldn’t worry much either”.
I think this is countered to a great extent by all the well-informed people who worry a lot about AI risk. I think the “well-informed people apparently disagree on this topic, I better look into it myself” environment promotes inquiry and is generally good for truth-seeking.
More generally, I agree with @Neel Nanda, it seems somewhat doubtful that people listening to a very niche Epoch Podcast aren’t aware of all the smart people worried about AI risk.

Satron Jan 19, 2025, 3:18 PM
1 point
1
in reply to: Lorec’s comment on: What’s Wrong With the Simulation Argument?
Future humans in specific, will at least one of: [ die off early, run lots of high-fidelity simulations of our universe’s history [“ancestor-simulations”], decide not to run such simulations
Is there any specific reason the first option is “die off early” and not “be unable to run lots of high-fidelity simulations”? The latter encompasses the former as well as scenarios where future humans survive but for one reason or the other can’t run these simulations.
I think a more general argument, in my opinion, would look like this:
“Future humans will at least one of 1) be unable to run high-fidelity simulations or 2) be unwilling to run high-fidelity simulations or 3) run high-fidelity simulations.”

Satron Jan 18, 2025, 9:36 PM
3 points
0
in reply to: Satron’s comment on: What’s Wrong With the Simulation Argument?
Here is Richard Carrier’s response to avturchin’s comment (for the courtesy of those reading this thread later):
These do not respond to my article. Just re-stating what was refuted (that most civilizations will be irredeemably evil and wasteful) is not a rebuttal. It’s simply ignoring the argument.

Satron Jan 18, 2025, 2:03 PM
1 point
0
in reply to: avturchin’s comment on: What’s Wrong With the Simulation Argument?
I suggest sending this as a comment under his article if you haven’t already. I am similarly interested in his response.

Satron Jan 18, 2025, 10:00 AM
7 points
3
on: What’s Wrong With the Simulation Argument?
Richard Carrier has written an entire rebuttal of the argument on his blog. He always answers comments under his posts, so if you disagree with some part of the rebuttal, you can just leave a comment there explaining which claim makes no sense to you. Then, he will usually defend the claim in question or provide the necessary clarification.

Satron Jan 18, 2025, 9:52 AM
4 points
0
in reply to: Remmelt’s comment on: What do you mean with ‘alignment is solvable in principle’?
By “possible worlds,” I mean all worlds that are consistent with laws of logic, such as the law of non-contradiction.
For example, it might be the case that, for some reason, alignment would only have been solved if and only if Abraham Lincoln wasn’t assassinated in 1865. That means that humans in 2024 in our world (where Lincoln was assasinated in 1865) will not be able to solve alignment, despite it being solvable in principle.
My answer is kind of similar to @quila’s. I think that he means roughly the same thing by “space of possible mathematical things.”
I don’t think that my definition of alignment is particularly important here because I was mostly clarifying how I would interpret the sentence if a stranger said it. Alignment is a broad word, and I don’t really have the authority to interpret stranger’s words in a specific way without accidentally misrepresenting them.
For example, one article managed to find six distinct interpretations of the word:
P1: Avoiding takeover from emergent optimization in AI agents
P2: Ensuring that AI’s information processing (and/or reasoning) is intelligible to us
P3: Ensuring AIs are good at solving problems as specified (by user or designer)
P4: Ensuring AI systems enhance, and don’t erode, human agency
P5: Ensuring that advanced AI agents learn a human utility function
P6: Ensuring that AI systems lead to desirable systemic and long-term outcomes

Satron Jan 17, 2025, 5:41 PM
6 points
2
on: What do you mean with ‘alignment is solvable in principle’?
The claim “alignment is solvable in principle” means “there are possible worlds where alignment is solved.”
Consequently, the claim “alignment is unsolvable in principle” means “there are no possible worlds where alignment is solved.”

Satron Jan 15, 2025, 9:10 AM
1 point
0
on: Reducing sycophancy and improving honesty via activation steering
A new method for reducing sycophancy. Sycophantic behavior is present in quite a few AI threat models, so it’s an important area to work on.
The article not only uses activation steering to reduce sycophancy in AI models but also provides directions for future work.
Overall, this post is a valuable addition to the toolkit of people who wish to build safe advanced AI.

Satron Jan 14, 2025, 12:30 AM
5 points
1
in reply to: Aram Panasenco’s comment on: Cast it into the fire! Destroy it!
Providing for material needs is less than 0.0000001% of the range of powers and possibilities that an AGI/ASI offers.
Imagine a scenario where we are driving from Austin to Fort Worth. The range of outcomes where we arrive at our destination is perhaps less than 0.0000001% of the total range of outcomes. There are countless potential interruptions that might prevent us from arriving at Fort Worth: traffic accidents, vehicle breakdowns, family emergencies, sudden illness, extreme weather, or even highly improbable events like alien encounters. The universe of possible outcomes is vast, and arriving safely in Fort Worth represents just one specific path through this possibility space.
Yet despite this, we reasonably expect to complete such journeys under normal circumstances. We don’t let the theoretical multitude of failure modes prevent us from making the trip. Drives like this typically succeed when basic conditions are met—a functioning vehicle, decent weather, and an alert driver.
So, as Seth Herd correctly points out, it all ends up depending on whether we can manage to align AGI (and deal with other issues such as governance or economy). And that’s a very large topic with a very wide range of opinions.

Satron Jan 13, 2025, 4:29 PM
5 points
0
on: Measuring and Improving the Faithfulness of Model-Generated Reasoning
This article provides a concrete, object-level benchmark for measuring the faithfulness of CoT. In addition to that, a new method for improving CoT faithfulness is introduced (something that is mentioned in a lot of alignment plans).

The method is straightforward and relies on breaking questions into subquestions. Despite its simplicity, it is surprisingly effective.

In the future, I hope to see alignment plans relying on CoT faithfulness incorporate this method into their toolkit.

Satron Jan 10, 2025, 12:07 AM
5 points
0
on: Shallow review of live agendas in alignment & safety
An (intentionally) shallow collection of AI alignment agendas that different organizations are working on.

This is the post I come back to when I want to remind myself what agendas different organizations are pursuing.

Overall, it is a solid and comprehensive post that I found very useful.

Satron Jan 9, 2025, 11:56 PM
3 points
0
on: A Playbook for AI Risk Reduction (focused on misaligned AI)
A very helpful collection of measures humanity can take to reduce risk from advanced AI systems.

Overall, this post is both detailed and thorough, making it a good read for people who want to learn about potential AI threat models as well as promising countermeasures we can take to avoid them.

Satron Jan 8, 2025, 8:19 AM
1 point
0
in reply to: AynonymousPrsn123’s comment on: Is my distinctiveness evidence for being in a simulation?
I guess the word “reality” is kind of ambiguous, and maybe that’s why we’ve been disagreeing for so long.

For example, imagine a scenario where we have 1) a non-simulated base world (let’s say 10¹² observers in it) AND 2) a simulated world with 10¹¹ observers AND 3) a simulated world with 1 observer. All three worlds actually concretely exist. People from world #1 just decided to run two simulations (#2 and #3). Surely, in this scenario, as per SSA, I can say that I am a randomly selected observer from the set of all observers. As far as I see, this “set of all observers” would include 10¹² + 10¹¹ + 1 observer because all of these observers actually exist, and I could’ve been born as any one of them.

Edit 1: I noticed that you edited one of your replies to include this:

And note: one potential problem with your reasoning is that if we take it to it’s logical extreme, it would be 100% certain that we are living in a simulation with infinite invisible observers. Because infinity dominates all the finite possibilities.

I don’t actually think this is true. My reasoning only really says that we are most likely to exist in the world with the most observers as compared to other actual worlds, not other possible worlds.

The most you can get out of this is the fact that conditional on a simulation with infinite observers existing, we are most likely in that simulation. However, because of the weirdness of actual infinity, because of the abysmal computational costs (it’s one thing to simulate billions of observers and another thing to simulate an infinity of observers), and because of the fact that it is probably physically impossible, I put an incredibly low prior on the fact that a simulation with infinite observers actually exists. And if it doesn’t exist, then we are not in it.

Edit 2: You don’t even need to posit a 10¹¹ simulation for it to be unlikely that you are in an “only one observer” simulation. It is enough that the non-simulated world has multiple observers. To illustrate what I mean, imagine that a society in a non-simulated world with 10¹² observers decides to make a simulation with only 1 observer. The odds are overwhelming that you’d be among 10¹² mundane, non-distinct observers in the non-simulated world.

Satron 8 Jan 2025 0:23 UTC
1 point
0
in reply to: AynonymousPrsn123’s comment on: Is my distinctiveness evidence for being in a simulation?
The way I understand it, the main difference between SIA and SSA is the fact that in SIA “I” may fail to exist. To illustrate what I mean, I will have to refer to “souls” just because it’s the easiest thing I can come up with.

SSA: There are 10¹¹ + 1 observers and 10¹¹ + 1 souls. Each soul gets randomly assigned to an observer. One of the souls is you. The probability of you existing is 1. You cannot fail to exist.

SIA: There are 10¹¹ + 1 observers and a very large (much larger than 10¹¹ + 1) amount of souls. Let’s call this amount N. Each soul gets assigned to an observer. One of the souls is you. However, in this scenario, you may fail to exist. The probability of you existing is (10¹¹ + 1)/N

Satron 8 Jan 2025 0:10 UTC
1 point
0
in reply to: AynonymousPrsn123’s comment on: Is my distinctiveness evidence for being in a simulation?
But the thing is that, there is a matter of fact of whether there are other observers in our world if it is simulated. Either you are the only observer or there are other observers, but one of them is true. Not just potentially true, but actually true.

The same is true of my last paragraph in the original answer (although perhaps, I could’ve used a clearer wording). If, as a matter of fact there actually exist 10¹¹ + 1 observers, then you are more likely to be in 10¹¹ group as per SSA. We don’t know if there are actually 10¹¹ + 1 observers, but that is merely an epistemic gap.

Satron 7 Jan 2025 23:39 UTC
1 point
0
in reply to: AynonymousPrsn123’s comment on: Is my distinctiveness evidence for being in a simulation?
Can you please explain why my explaination requires SIA? From a quick Google search: “The Self Sampling Assumption (SSA) states that we should reason as if we’re a random sample from the set of actual existent observers”

My last paragraph in my original answer was talking about a scenario where simulators have actually simulated a) a world with 1 observers AND b) a world with 10¹¹ observer. So a set of “actual existent observers” includes 1 + 10¹¹ observers. You are randomly selected from that, giving you 1:10¹¹ odds of being in the world where you are the only observer. I don’t see where SIA is coming in play here.

Satron 7 Jan 2025 23:31 UTC
1 point
0
in reply to: AynonymousPrsn123’s comment on: Is my distinctiveness evidence for being in a simulation?

but the problem with your argument is that my original argument could also just consider only simulations in which I am the only observer. In which case Pr(I’m distinct | I’m in a simulation)=1, not 0.5. And since there’s obviously some prior probability of this simulation being true, my argument still follows.

But then this turns Pr(I’m in a simulation) into Pr(I’m in a simulation) + Pr(only simulations with one observer exist | simulations exist). It’s not enough that a simulation exists with only one observer. It needs to be so that simulations with multiple observers also don’t exist. For example, if there is just one simulation with a billion observers, it heavily skews the odds in favor of you not being in a simulation with just one observer.

And I am very much willing to say Pr(I’m in a simulation) + Pr(only simulations with one observer exist | simulations exist) is going to be lower than Pr(I’m distinct | I’m not in a simulation).

I now think my actual error is saying Pr(I’m distinct | I’m not in a simulation)=0.0001, when in reality this probability should be 1, since I am not a random sample of all humans (i.e., SSA is wrong), I am me. Is that clear?

That answer seems reasonable to me. However, I think that there is value in my answer as well: it works even if SSA (the “least favorable” assumption) is true.