It seems to me that viewing a late Great Filter to be worse news than an early Great Filter is another instance of the confusion and irrationality of SSA/SIA-style anthropic reasoning and subjective anticipation. If you anticipate anything, believing that the great filter is more likely to lie in the future means you have to anticipate a higher probability of experiencing doom.
Let’s take this further: is there any reason, besides our obsession with subjective anticipation, to discuss whether a late great filter is ‘good’ or ‘bad’ news, over and above policy implications? Why would an idealized agent evaluate the utility of counterfactuals it knows it can’t realize?
That is a good question, and one that I should have asked and tried to answer before I wrote this post. Why do we divide possible news into “good” and “bad”, and “hope” for good news? Does that serve some useful cognitive function, and if so, how?
Without having good answers to these questions, my claim that a late great filter should not be considered bad news may just reflect confusion about the purpose of calling something “bad news”.
About the cognitive function of “hope”: it makes evolutionary sense to become all active and bothered when a big pile of utility hinges on a single uncertain event in the near future, because that makes you frantically try to influence that event. If you don’t know how to influence it (as in the case of a lottery), oh well, evolution doesn’t care.
Evolution might care. That is, systems that expend a lot of attention on systems they can’t influence might do worse than systems that instead focus their attention on systems they can influence. But yes, either there weren’t any of the second kind of system around to compete with our ancestors, or there were and they lost out for some other reason, or there were and it turns out that it’s a bad design for our ancestral environment.
is there any reason, besides our obsession with subjective anticipation, to discuss whether a late great filter is ‘good’ or ‘bad’ news,
no, I don’t think so. But if you strip subjective anticipation off a human, you might lose a lot of our preferences. We are who we are, so we care about subjective anticipation.
We are who we are, but who we are is not fixed. What we care about seem to depend on what arguments we listen to or think up, and in what order. (See my Shut Up and Divide post for an example of this.)
While a ideally rational agent (according to our current best conception of ideal rationality) would seek to preserve its values regardless of what they are, some humans (including me, for example) actively seek out arguments that might change what they care about. Such “value-seeking” behavior doesn’t seem irrational to me, even though I don’t know how to account for it in terms of rationality.
And while it seems impossible for a human to completely give up subjective anticipation, it does seem possible to care less about it.
Such “value-seeking” behavior doesn’t seem irrational to me, even though I don’t know how to account for it in terms of rationality.
I would say it is part of checking for reflective consistency. Ideally, there shouldn’t be arguments that change your (terminal) values, so if there are, you want to so you can figure out what is wrong and how to fix it.
I don’t think that explanation makes sense. Suppose an AI thinks it might have a security hole in its network stack, so that if someone sends it a certain packet, it would become that person’s slave. It would try to fix that security hole, without actually seeking to have such a packet sent to itself.
We humans know that there are arguments out there that can change our values, but instead of hardening our minds against them, some of us actually try to have such arguments sent to us.
We humans know that there are arguments out there that can change our values, but instead of hardening our minds against them, some of us actually try to have such arguments sent to us.
In the deontological view of values this is puzzling, but in the consequentialist view it isn’t: we welcome arguments that can change our instrumental values, but not our terminal values (A.K.A. happiness/pleasure/eudaimonia/etc.). In fact I contend that it doesn’t even make sense to talk about changing our terminal values.
My explanation is that the human mind is something like a coalition of different sub-agents, many of which are more like animals or insects than rational agents. In any given context, they will pull the overall strategy in different directions. The overall result is an agent with context dependent preferences, i.e. irrational behavior. Many people just live with this.
Some people, however, try to develop a “life philosophy” that shapes the disparate urges of the different mental subcomponents into an overall strategy, that reflects a consistent overall policy.
A moral “argument” might be a hypothetical that attempts to put your mind into a new configuration of relative power of subagents, so that you can re-assess the overall deal.
My explanation is that the human mind is something like a coalition of different sub-agents, many of which are more like animals or insects than rational agents. In any given context, they will pull the overall strategy in different directions. The overall result is an agent with context dependent preferences, i.e. irrational behavior.
Congratulations, you just reinvented [a portion of] PCT. ;-)
[Clarification: PCT models the mind as a massive array of simple control circuits that act to correct errors in isolated perceptions, with consciousness acting as a conflict-resolver to manage things when two controllers send conflicting commands to the same sub-controller. At a fairly high level, a controller might be responsible for a complex value: like correcting hits to self-esteem, or compensating for failings in one’s aesthetic appreciation of one’s work. Such high-level controllers would thus appear somewhat anthropomorphically agent-like, despite simply being something that detects a discrepancy between a target and an actual value, and sets subgoals in an attempt to rectify the detected discrepancy. Anything that we consider of value potentially has an independent “agent” (simple controller) responsible for it in this way, but the hierarchy of control does not necessarily correspond to how we would abstractly prefer to rank our values—which is where the potential for irrationaity and other failings lies.]
Let’s take this further: is there any reason, besides our obsession with subjective anticipation, to discuss whether a late great filter is ‘good’ or ‘bad’ news, over and above policy implications? Why would an idealized agent evaluate the utility of counterfactuals it knows it can’t realize?
That is a good question, and one that I should have asked and tried to answer before I wrote this post. Why do we divide possible news into “good” and “bad”, and “hope” for good news? Does that serve some useful cognitive function, and if so, how?
Without having good answers to these questions, my claim that a late great filter should not be considered bad news may just reflect confusion about the purpose of calling something “bad news”.
About the cognitive function of “hope”: it makes evolutionary sense to become all active and bothered when a big pile of utility hinges on a single uncertain event in the near future, because that makes you frantically try to influence that event. If you don’t know how to influence it (as in the case of a lottery), oh well, evolution doesn’t care.
Evolution might care. That is, systems that expend a lot of attention on systems they can’t influence might do worse than systems that instead focus their attention on systems they can influence. But yes, either there weren’t any of the second kind of system around to compete with our ancestors, or there were and they lost out for some other reason, or there were and it turns out that it’s a bad design for our ancestral environment.
If we expect a late great filter, we may need to strive significantly harder to avoid it. The policy implications are staggering.
no, I don’t think so. But if you strip subjective anticipation off a human, you might lose a lot of our preferences. We are who we are, so we care about subjective anticipation.
We are who we are, but who we are is not fixed. What we care about seem to depend on what arguments we listen to or think up, and in what order. (See my Shut Up and Divide post for an example of this.)
While a ideally rational agent (according to our current best conception of ideal rationality) would seek to preserve its values regardless of what they are, some humans (including me, for example) actively seek out arguments that might change what they care about. Such “value-seeking” behavior doesn’t seem irrational to me, even though I don’t know how to account for it in terms of rationality.
And while it seems impossible for a human to completely give up subjective anticipation, it does seem possible to care less about it.
I would say it is part of checking for reflective consistency. Ideally, there shouldn’t be arguments that change your (terminal) values, so if there are, you want to so you can figure out what is wrong and how to fix it.
I don’t think that explanation makes sense. Suppose an AI thinks it might have a security hole in its network stack, so that if someone sends it a certain packet, it would become that person’s slave. It would try to fix that security hole, without actually seeking to have such a packet sent to itself.
We humans know that there are arguments out there that can change our values, but instead of hardening our minds against them, some of us actually try to have such arguments sent to us.
In the deontological view of values this is puzzling, but in the consequentialist view it isn’t: we welcome arguments that can change our instrumental values, but not our terminal values (A.K.A. happiness/pleasure/eudaimonia/etc.). In fact I contend that it doesn’t even make sense to talk about changing our terminal values.
It is indeed a puzzling phenomenon.
My explanation is that the human mind is something like a coalition of different sub-agents, many of which are more like animals or insects than rational agents. In any given context, they will pull the overall strategy in different directions. The overall result is an agent with context dependent preferences, i.e. irrational behavior. Many people just live with this.
Some people, however, try to develop a “life philosophy” that shapes the disparate urges of the different mental subcomponents into an overall strategy, that reflects a consistent overall policy.
A moral “argument” might be a hypothetical that attempts to put your mind into a new configuration of relative power of subagents, so that you can re-assess the overall deal.
Congratulations, you just reinvented [a portion of] PCT. ;-)
[Clarification: PCT models the mind as a massive array of simple control circuits that act to correct errors in isolated perceptions, with consciousness acting as a conflict-resolver to manage things when two controllers send conflicting commands to the same sub-controller. At a fairly high level, a controller might be responsible for a complex value: like correcting hits to self-esteem, or compensating for failings in one’s aesthetic appreciation of one’s work. Such high-level controllers would thus appear somewhat anthropomorphically agent-like, despite simply being something that detects a discrepancy between a target and an actual value, and sets subgoals in an attempt to rectify the detected discrepancy. Anything that we consider of value potentially has an independent “agent” (simple controller) responsible for it in this way, but the hierarchy of control does not necessarily correspond to how we would abstractly prefer to rank our values—which is where the potential for irrationaity and other failings lies.]
It does seem that something in this region has to be correct.