It was pointed out, e.g. [here](https://web.archive.org/web/20170918044233/http://files.openphilanthropy.org/files/Grants/MIRI/consolidated_public_reviews.pdf) but unfortunately the mindset is self-reinforcing so there wasn’t much to be done.
Your link formatting got messed up here.
More generally when you’re asked a question like “How much do you expect [politician’s name] to win/lose by?” how do you go about generating your response? How do you generate your predictions?
My usual response would look something like this:
“Pre-scandal polls are showing Bodee leading by about 1-2%, but given the recent spying scandal and the strong emphasis of our society on ethical conduct, I expect him to lose by 3-4%.”
If I am speaking to a person who (on my expectations) is following the polls, I would try to explain why my prediction differs from them.
But you seem to mean something broader with “possible worlds.” Something like “in theory, there is a physically possible arrangement of atoms/energy states that would result in an ‘aligned’ AGI, even if that arrangement of states might not be reachable from our current or even a past world.”
–> Am I interpreting you correctly?
Yup, that’s roughly what I meant. However, one caveat would be that I would change “physically possible” to “metaphysically/logically possible” because I don’t know if worlds with different physics could exist, whereas I am pretty sure that worlds with different metaphysical/logical laws couldn’t exist. By that, I mean stuff like the law of non-contradiction and “if a = b, then b = a.”
You saying this shows the ambiguity here of trying to understand what different people mean. One researcher can make a technical claim about the possibility/tractability of “alignment” that is similarly worded to a technical claim others made. Yet their meaning of “alignment” could be quite different.
It’s hard then to have a well-argued discussion, because you don’t know whether people are equivocating (ie. switching between different meanings of the term).
I think the main antidote against this is to ask the person you are speaking with to define the term if they are making claims in which equivocation is especially likely.
The way I deal with the wildly varying uses of the term “alignment” is to use a minimum definition that most of those six interpretations are consistent with
Yeah, that’s reasonable.
Listeners would understandably infer that “these well-informed people apparently don’t really worry much, maybe I shouldn’t worry much either”.
I think this is countered to a great extent by all the well-informed people who worry a lot about AI risk. I think the “well-informed people apparently disagree on this topic, I better look into it myself” environment promotes inquiry and is generally good for truth-seeking.
More generally, I agree with @Neel Nanda, it seems somewhat doubtful that people listening to a very niche Epoch Podcast aren’t aware of all the smart people worried about AI risk.
Future humans in specific, will at least one of: [ die off early, run lots of high-fidelity simulations of our universe’s history [“ancestor-simulations”], decide not to run such simulations
Is there any specific reason the first option is “die off early” and not “be unable to run lots of high-fidelity simulations”? The latter encompasses the former as well as scenarios where future humans survive but for one reason or the other can’t run these simulations.
I think a more general argument, in my opinion, would look like this:
“Future humans will at least one of 1) be unable to run high-fidelity simulations or 2) be unwilling to run high-fidelity simulations or 3) run high-fidelity simulations.”
Here is Richard Carrier’s response to avturchin’s comment (for the courtesy of those reading this thread later):
These do not respond to my article. Just re-stating what was refuted (that most civilizations will be irredeemably evil and wasteful) is not a rebuttal. It’s simply ignoring the argument.
I suggest sending this as a comment under his article if you haven’t already. I am similarly interested in his response.
Richard Carrier has written an entire rebuttal of the argument on his blog. He always answers comments under his posts, so if you disagree with some part of the rebuttal, you can just leave a comment there explaining which claim makes no sense to you. Then, he will usually defend the claim in question or provide the necessary clarification.
By “possible worlds,” I mean all worlds that are consistent with laws of logic, such as the law of non-contradiction.
For example, it might be the case that, for some reason, alignment would only have been solved if and only if Abraham Lincoln wasn’t assassinated in 1865. That means that humans in 2024 in our world (where Lincoln was assasinated in 1865) will not be able to solve alignment, despite it being solvable in principle.
My answer is kind of similar to @quila’s. I think that he means roughly the same thing by “space of possible mathematical things.”
I don’t think that my definition of alignment is particularly important here because I was mostly clarifying how I would interpret the sentence if a stranger said it. Alignment is a broad word, and I don’t really have the authority to interpret stranger’s words in a specific way without accidentally misrepresenting them.
For example, one article managed to find six distinct interpretations of the word:
P1: Avoiding takeover from emergent optimization in AI agents
P2: Ensuring that AI’s information processing (and/or reasoning) is intelligible to us
P3: Ensuring AIs are good at solving problems as specified (by user or designer)
P4: Ensuring AI systems enhance, and don’t erode, human agency
P5: Ensuring that advanced AI agents learn a human utility function
P6: Ensuring that AI systems lead to desirable systemic and long-term outcomes
The claim “alignment is solvable in principle” means “there are possible worlds where alignment is solved.”
Consequently, the claim “alignment is unsolvable in principle” means “there are no possible worlds where alignment is solved.”
A new method for reducing sycophancy. Sycophantic behavior is present in quite a few AI threat models, so it’s an important area to work on.
The article not only uses activation steering to reduce sycophancy in AI models but also provides directions for future work.
Overall, this post is a valuable addition to the toolkit of people who wish to build safe advanced AI.
Providing for material needs is less than 0.0000001% of the range of powers and possibilities that an AGI/ASI offers.
Imagine a scenario where we are driving from Austin to Fort Worth. The range of outcomes where we arrive at our destination is perhaps less than 0.0000001% of the total range of outcomes. There are countless potential interruptions that might prevent us from arriving at Fort Worth: traffic accidents, vehicle breakdowns, family emergencies, sudden illness, extreme weather, or even highly improbable events like alien encounters. The universe of possible outcomes is vast, and arriving safely in Fort Worth represents just one specific path through this possibility space.
Yet despite this, we reasonably expect to complete such journeys under normal circumstances. We don’t let the theoretical multitude of failure modes prevent us from making the trip. Drives like this typically succeed when basic conditions are met—a functioning vehicle, decent weather, and an alert driver.
So, as Seth Herd correctly points out, it all ends up depending on whether we can manage to align AGI (and deal with other issues such as governance or economy). And that’s a very large topic with a very wide range of opinions.
This article provides a concrete, object-level benchmark for measuring the faithfulness of CoT. In addition to that, a new method for improving CoT faithfulness is introduced (something that is mentioned in a lot of alignment plans).
The method is straightforward and relies on breaking questions into subquestions. Despite its simplicity, it is surprisingly effective.
In the future, I hope to see alignment plans relying on CoT faithfulness incorporate this method into their toolkit.
An (intentionally) shallow collection of AI alignment agendas that different organizations are working on.
This is the post I come back to when I want to remind myself what agendas different organizations are pursuing.
Overall, it is a solid and comprehensive post that I found very useful.
A very helpful collection of measures humanity can take to reduce risk from advanced AI systems.
Overall, this post is both detailed and thorough, making it a good read for people who want to learn about potential AI threat models as well as promising countermeasures we can take to avoid them.
I guess the word “reality” is kind of ambiguous, and maybe that’s why we’ve been disagreeing for so long.
For example, imagine a scenario where we have 1) a non-simulated base world (let’s say 10¹² observers in it) AND 2) a simulated world with 10¹¹ observers AND 3) a simulated world with 1 observer. All three worlds actually concretely exist. People from world #1 just decided to run two simulations (#2 and #3). Surely, in this scenario, as per SSA, I can say that I am a randomly selected observer from the set of all observers. As far as I see, this “set of all observers” would include 10¹² + 10¹¹ + 1 observer because all of these observers actually exist, and I could’ve been born as any one of them.
Edit 1: I noticed that you edited one of your replies to include this:
And note: one potential problem with your reasoning is that if we take it to it’s logical extreme, it would be 100% certain that we are living in a simulation with infinite invisible observers. Because infinity dominates all the finite possibilities.
I don’t actually think this is true. My reasoning only really says that we are most likely to exist in the world with the most observers as compared to other actual worlds, not other possible worlds.
The most you can get out of this is the fact that conditional on a simulation with infinite observers existing, we are most likely in that simulation. However, because of the weirdness of actual infinity, because of the abysmal computational costs (it’s one thing to simulate billions of observers and another thing to simulate an infinity of observers), and because of the fact that it is probably physically impossible, I put an incredibly low prior on the fact that a simulation with infinite observers actually exists. And if it doesn’t exist, then we are not in it.
Edit 2: You don’t even need to posit a 10¹¹ simulation for it to be unlikely that you are in an “only one observer” simulation. It is enough that the non-simulated world has multiple observers. To illustrate what I mean, imagine that a society in a non-simulated world with 10¹² observers decides to make a simulation with only 1 observer. The odds are overwhelming that you’d be among 10¹² mundane, non-distinct observers in the non-simulated world.
The way I understand it, the main difference between SIA and SSA is the fact that in SIA “I” may fail to exist. To illustrate what I mean, I will have to refer to “souls” just because it’s the easiest thing I can come up with.
SSA: There are 10¹¹ + 1 observers and 10¹¹ + 1 souls. Each soul gets randomly assigned to an observer. One of the souls is you. The probability of you existing is 1. You cannot fail to exist.
SIA: There are 10¹¹ + 1 observers and a very large (much larger than 10¹¹ + 1) amount of souls. Let’s call this amount N. Each soul gets assigned to an observer. One of the souls is you. However, in this scenario, you may fail to exist. The probability of you existing is (10¹¹ + 1)/N
But the thing is that, there is a matter of fact of whether there are other observers in our world if it is simulated. Either you are the only observer or there are other observers, but one of them is true. Not just potentially true, but actually true.
The same is true of my last paragraph in the original answer (although perhaps, I could’ve used a clearer wording). If, as a matter of fact there actually exist 10¹¹ + 1 observers, then you are more likely to be in 10¹¹ group as per SSA. We don’t know if there are actually 10¹¹ + 1 observers, but that is merely an epistemic gap.
Can you please explain why my explaination requires SIA? From a quick Google search: “The Self Sampling Assumption (SSA) states that we should reason as if we’re a random sample from the set of actual existent observers”
My last paragraph in my original answer was talking about a scenario where simulators have actually simulated a) a world with 1 observers AND b) a world with 10¹¹ observer. So a set of “actual existent observers” includes 1 + 10¹¹ observers. You are randomly selected from that, giving you 1:10¹¹ odds of being in the world where you are the only observer. I don’t see where SIA is coming in play here.
This is almost exactly my story. I used to think that qualia was just another example of anti-physicalist magical thinking.
Currently, I am actually pretty firmly in Camp #2
As someone who used to be (but no longer is) an illusionist about both free will and qualia, I can assure you that this was not the case (at least with me). Nothing really changed, besides me getting a deeper understanding of the position of the other camp.