I used to eat a lot of chicken and eggs before I read Peter Singer. After that, I went cold turkey (pardon the expression).
Utilitarian
Karma: 172
Some really creative ideas, ChristianKl. :)
Even with what you describe, humans wouldn’t become extinct, barring other outcomes like really bad nuclear war or whatever.
However, since the AI wouldn’t be destroyed, it could bide its time. Maybe it could ally with some people and give them tech/power in exchange for carrying out its bidding. They could help build the robots, etc. that would be needed to actually wipe out humanity.
Obviously there’s a lot of conjunction here. I’m not claiming this scenario specifically is likely. But it helps to stimulate the imagination to work out an existence proof for the extinction risk from AGI.
It’s not at all clear that a AGI will be human-like, anyone than humans are dog-like.
Ok, bad wording on my part. I meant “more generally intelligent.”
How do you fight the AGI past that point?
I was imagining people would destroy their computers, except the ones not connected to the Internet. However, if the AGI is hiding itself, it could go a long way before people realized what was going on.
Interesting scenarios. Thanks!
As we begin seeing robots/computers that are more human-like, people will take the possibility of AGIs getting out of control more seriously. These things will be major news stories worldwide, people will hold natural-security summits about them, etc. I would assume the US military is already looking into this topic at least a little bit behind closed doors.
There will probably be lots of not-quite-superhuman AIs / AGIs that cause havoc along the road to the first superhuman ones. Yes, it’s possible that FOOM will take us from roughly a level like where we are now to superhuman AGI in a matter of days, but this scenario seems relatively unlikely to me, so any leverage you want to make on it has to be multiplied by that small probability of it happening.
--
BTW, I’m curious to hear more about the mechanics of your scenario. The AGI hacks itself onto every (Internet-connected) computer in the world. Then what? Presumably this wouldn’t cause extinction, just a lot of chaos and maybe years’ worth of setback to the economy? Maybe it would increase chances of nuclear war, especially if the AGI could infect nuclear-warhead-related computer systems.
This could be an example of the non-extinction-level AGI disasters that I was referring to. Let me know if there are more ways in which it might cause total extinction, though.
This is a good point. :) I added an additional objection to the piece.
As an empirical matter, extinction risk isn’t being funded as much as you suggest it should be if almost everyone has some incentives to invest in the issue.
There’s a lot of “extinction risk” work that’s not necessarily labeled as such: Biosecurity, anti-nuclear proliferation, general efforts to prevent international hostility by nation states, general efforts to reduce violence in society and alleviate mental illnesses, etc. We don’t necessarily see huge investments in AI safety yet, but this will probably change in time, as we begin to see more AIs that get out of control and cause problems on a local scale. 99+% of catastrophic risks are not extinction risks, so as the catastrophes begin happening and affecting more people, governments will invest more in safeguards than they do now. The same can be said for nanotech.
In any event, even if budgets for extinction-risk reduction are pretty low, you also have to look at how much money can buy. Reducing risks is inherently difficult, because so much is out of our hands. It seems relatively easier to win over hearts and minds to utilitronium (especially at the margin right now, by collecting the low-hanging fruit of people who could be persuaded but aren’t yet). And because so few people are pushing for utilitronium, it seems far easier to achieve a 1% increase in support for utilitronium than a 1% decrease in the likelihood of extinction.
Thanks, Luke. See also this follow-up discussion to Ord’s essay.
As you suggest with your “some” qualifier, my essay that benthamite shared doesn’t make any assumptions about negative utilitarianism. I merely inserted parentheticals about my own views into it to avoid giving the impression that I’m personally a positive-leaning utilitarian.
Thanks, Jabberslythe! You got it mostly correct. :)
The one thing I would add is that I personally think people don’t usually take suffering seriously enough—at least not really severe suffering like torture or being eaten alive. Indeed, many people may never have experienced something that bad. So I put high importance on preventing experiences like these relative to other things.
Interesting story. Yes, I think our intuitions about what kinds of computations we want to care about are easily bent and twisted depending on the situation at hand. In analogy with Dennett’s “intentional stance,” humans have a “compassionate stance” that we apply to some physical operations and don’t apply to others. It’s not too hard to manipulate these intuitions by thought experiments. So, yes, I do fear that other people may differ (perhaps quite a bit) in their views about what kinds of computations are suffering that we should avoid.
I bet there are a lot more people who care about animals’ feelings and who care a lot more, than those who care about the aesthetics of brutality in nature.
Well, at the moment, there are hundreds of environmental-preservation organizations and basically no organizations dedicated to reducing wild-animal suffering. Environmentalism as a cause is much more mainstream than animal welfare. Just like the chickens that go into people’s nuggets, animals suffering in nature “are out of sight, and the connection between [preserving pristine habitats] and animals living terrible lives elsewhere is hard to visualize.”
It’s encouraging that more LessWrongers are veg than average, although I think 12.4% is pretty typical for elite universities and the like as well. (But maybe that underscores your point.)
The biggest peculiarity of Brian Tomasik’s utility function, that is least likely to ever be shared by the majority of humanity, is probably not that he cares about animals (even that he cares about insects) but that he cares so much more about suffering than happiness and other good things.
An example post. I care a lot about suffering, a little about happiness, and none about other things.
The exchange rate in your utility function between good things and bad things is pretty relevent to whether you should prefer CEV or paperclipping (And what the changes in the probabilities of each even based on actions you might take would have to be in order justify them) and whether you think lab universes would be a good thing.
Yep!
This is what you value, what you chose.
Yes. We want utilitarianism. You want CEV. It’s not clear where to go from there.
Not the hamster’s one.
FWIW, hamsters probably exhibit fairness sensibility too. At least rats do.
Do you think the typical person advocating ecological balance has evaluated how the tradeoffs would change given future technology?
Good point. Probably not, and for some, their views would change with new technological options. For others (environmentalist types especially), they would probably retain their old views.
That said, the future-technology sword cuts both ways: Because most people aren’t considering post-human tech, they’re not thinking of (what some see as) the potential astronomical benefits from human survival. If 10^10 humans were only going to live at most another 1-2 billion years on Earth, their happiness could never outweigh the suffering of the 10^18 insects living on Earth at the same time. So if people aren’t thinking about space colonization, why do they care so much about preserving humanity anyway? Two possible reasons are because they’re speciesist and care more about humans or because they value things other than happiness and suffering. I think both are true here, and both are potentially problematic for CEV values.
Though if people without such strong intuitions are likely to become more rational, this would not be strong evidence.
Yeah, that would be my concern. These days, “being rational” tends to select for people who have other characteristics, including being more utilitarian in inclination. Interesting idea about seeing how deep ecologists’ views would change upon becoming more rational.
The suffering is bad, but there are other values to consider here, that the scenario includes in far greater quantities.
We have different intuitions about how bad suffering is. My pain:pleasure exchange rate is higher than that of most people, and this means I think the expected suffering that would result from a Singularity isn’t worth the potential for lots of happiness.
Thanks, JGWeissman. There are certainly some deep ecologists, like presumably Hettinger himself, who have thought long and hard about the scale of wild-animal suffering and still support preservation of ecology as is. When I talk with ecologists or environmentalists, almost always their reply is something like, “Yes, there’s a lot of suffering, but it’s okay because it’s natural for them.” One example:
As I sit here, thinking about the landscape of fear, I watch a small bird at my bird feeder. It spends more time looking around than it does eating. I try to imagine the world from its point of view — the startles, the alarms, the rustle of wings, the paw of the cat. And although I wish it well, I wouldn’t like its predators to disappear.
You can see many more examples here. A growing number of people have been convinced that wild-animal suffering should be reduced where feasible, but I think this is still a minority view. If more people thought about it harder, probably there would be more support, but ecological preservation is also a very strong intuition for some people. It’s easy not to realize this when we’re in our own bubbles of utilitarian-minded rationalists. :)
Spreading life far and wide is less widespread as a value, but it’s popular enough that the Panspermia Society is one of a few groups that feels this way. I also have a very smart friend who happens to share this goal, even though he acknowledges this would create a lot of suffering.
As far as insects, it’s not obvious that post-humans would care enough to undertake the approximation of their brains that you mention, because maybe it would make the simulation more complicated (=> expensive) or reduce its fidelity. There’s an analogy with factory farming today: Sure, we could prevent animal suffering, but it’s more costly. Still, yes, we can hope that post-humans would give enough weight to insect suffering to avoid this. And I agree insects may very well not be sentient, though if they are, the numbers of suffering minds would be astronomical.
The work on nonperson predicates and computational hazards is great—I’m glad you guys are doing that!
Thanks, Benito. Do we know that we shouldn’t have a lot of chicken feed? My point in asking this is just that we’re baking in a lot of the answer by choosing which minds we extrapolate in the first place. Now, I have no problem baking in answers—I want to bake in my answers—but I’m just highlighting that it’s not obvious that the set of human minds is the right one to extrapolate.
BTW, I think the “brain reward pathways” between humans and chickens aren’t that different. Maybe you were thinking about the particular, concrete stimuli that are found to be rewarding rather than the general architecture.
Why not include primates, dolphins, rats, chickens, etc. into the ethics?
Future humans may not care enough about animal suffering relative to other things, or may not regard suffering as being as bad as I do. As noted in the post, there are people who want to spread biological life as much as possible throughout the galaxy. Deep ecologists may actively want to preserve wild-animal suffering (Ned Hettinger: “Respecting nature means respecting the ways in which nature trades values, and such respect includes painful killings for the purpose of life support.”) Future humans might run ancestor sims that happen to include astronomical numbers of sentient insects, most of which die (possibly painfully) shortly after birth. In general, humans have motivations to simulate minds similar to theirs, which means potentially a lot more suffering along for the ride.
Preventing suffering is what I care about, and I’m going to try to convince other people to care about it. One way to do that is to invent plausible thought experiments / intuition pumps for why it matters so much. If I do, that might help with evangelism, but it’s not the (original) reason why I care about it. I care about it because of experience with suffering in my own life, feeling strong empathy when seeing it in others, and feeling that preventing suffering is overridingly important due to various other factors in my development.
My understanding is that CEA exists in order to simplify the paperwork of multiple projects. For example, Effective Animal Activism is not its own charity; instead, you donate to CEA and transfer the money to EAA. As bryjnar said, there’s not really any overhead in doing this. Using CEA as an umbrella much simpler than trying to get 501(c)(3) status for EAA on its own, which would be painstaking process.
I appreciate personal anecdote. Sometimes I think anecdotes are the most valuable parts of an essay. It all depends on the style and the preferences of the audience. I don’t criticize HPMOR on the grounds that it focuses too much on Harry and not enough on rationality concepts...
Three friends independently pointed me to Overcoming Bias in fall/winter 2006.
Jonah, I agree with what you say at least in principle, even if you would claim I don’t follow it in practice. A big advantage of being Bayesian is that you retain probability mass on all the options rather than picking just one. (I recall many times being dismayed with hacky approximations like MAP that let you get rid of the less likely options. Similarly when people conflate the Solomonoff probability of a bitstring with the shortest program that outputs it, even though I guess in that case, the shortest program necessarily has at least as much probability as all the others can combined.)
My main comment on your post is that it’s hard to keep track of all of these things computationally. Probably you should try, but it can get messy. It’s also possible that in keeping track of too many details, you introduce more errors than if you had kept the analysis simple. On many questions in physics, ecology, etc., there’s a single factor that dominates all the rest. Maybe this is less true in human domains because rational agents tend to produce efficiencies due to eating up the free lunches.
So, I’m in favor of this approach if you can do it and make it work, but don’t let the best be the enemy of the good. Focus on the strong arguments first, and only if you have the bandwidth go on to think about the weak ones too.