By far the biggest and most sudden update I’ve ever had is Dominion, a documentary on animal farming:
https://www.youtube.com/watch?v=LQRAfJyEsko
It’s like… I had a whole pile of interconnected beliefs, and if you pulled on one it would snap most of the way back into place after. And Dominion pushed the whole pile over at once.
Meta comment: I’m going to be blunt. Most of this sequence has been fairly heavily downvoted. That reads to me as this community asking to not have more such content. You should consider not posting, or posting elsewhere, or writing many fewer posts of much higher quality (e.g. spending more time, doing more background research, asking someone to proofread). As a data point, I’ve only posted a couple times, and I spent at least, I dunno, 10+ hours writing each post. As an example of how this might apply to you, if you wrote this whole sequence as a single “reference on biases” and shared that, I bet it would be better received.
You should try turning the temperature up.
Will you fly to the Sahel with a backpack full of antibiotics?
I imagine you suggesting this, a bunch of people nodding along in agreement, then no one doing it because of personal safety and because it’s weird.
People often study on their own for the GREs or for the Actuarial exams. In both cases the results are taken seriously, there are a ton of prep materials, and I think the exams are funded by a flat fee to test takers (which works due to the magic of economy of scale).
Person who doesn’t know squat about poker here. The first example was clear: they disagreed about whether to fold or call, and it was right on the edge. Especially since where the other player was from was relevant.
It was long for someone who doesn’t know poker though.
Wow, this got heated fast. Partly my fault. My assumptions were unwarranted and my model therefore unrealistic. Sorry.
I think we’ve been talking past each other. Some clarifications on my position:
I’m not suggesting that one only reclines if one is given permission to do so from the person behind them. I’m suggesting cooperation on the act that is controlled by one person but effects two people. If reclining is a minor convenience to the person in front, but would crush the legs of the person behind, it does not happen. If the person in front has a back problem and the person in back is short, reclining does happen.
None of the blame goes toward other passengers. The blame all goes to the airlines. If you want to recline but don’t get to, that’s the airline’s fault. If you don’t want the person in front of you to recline but they do, that’s the airline’s fault. They should make better seat arrangements. I would preferentially fly on an airline that didn’t stuff me in like cattle. I’m all for protesting with you about this.
If you disagree with this, would you agree that if airplanes were naturally occurring rather than being engineered, then the decision of whether to recline should be a conversation between the two people it effects? If so, what breaks the symmetry between the two effected people, when the situation is engineered by the airline?
EDIT: Or, to get at my emotional crux, if my very long legs would be smooshed if you were to recline, and reclining was a minor convenience for you, would you say “ok, I won’t recline, but let’s be angry at the airline for putting us in this situation”, or “nope I’m reclining anyways, blame the airline”?
EDIT: I no longer endorse this model.
Say that flights are on average 80% full, 20% of passengers are tall and will be miserable for the whole flight if they’re reclined into, 50% of passengers want to recline, and planes are shaped like donuts so that every seat has a seat behind it.
If passengers behave like you, then 8% of passengers are miserable in exchange for 50% of passengers getting to recline. If passengers instead ask before reclining (or unrecline if asked to), then 0% of passengers are miserable and 42% get to recline. The passengers pick between these two situations.
The second situation is better than the first. Should airlines not allow seats to recline, or increase spacing between seats by (say) 12% and thus increase ticket prices by (say) 8%, because passengers like you insist on choosing the first situation over the second?
An internet search suggests that extra legroom tends to cost $20-$100, typically in between, which matches what I remember seeing. Have you seen it for $10? If so I need to pay more attention next time and shell out the $10!
From the point of view of a powerful AI, Earth is infested with many nests of humans that could do damage to important things. At the very least it makes sense to permanently neuter their ability to do that.
That’s a positive outcome, as long as said humans aren’t unduly harmed, and “doing damage to important things” doesn’t include say, eating plants or scaring bunnies by walking by them.
An AI that maximises total group reward function because it cares only for its own reward function, which is defined as “maximise total group reward function” appears aligned right up until it isn’t.
The AI does not aim to maximize its reward function! The AI is trained on a reward function, and then (by hypothesis) becomes intelligent enough to act as an inner optimizer that optimizes for heuristics that yielded high reward in its (earlier) training environment. The aim is to produce a training environment such that the heuristics of the inner optimizer tries to maximize tend towards altruism.
What is altruistic supposed to mean here?
What does it mean that humans are altruistic? It’s a statement about our own messy utility function, which we (the inner optimizers) try to maximize, that was built from heuristics that worked well in our ancestral environment, like “salt and sugar are great” and “big eyes are adorable”. Our altruism is a weird, biased, messy thing: we care more about kittens than pigs because they’re cuter, and we care a lot more about things we see than things we don’t see.
Likewise, whatever heuristics work well in the AI training environment are likely to be weird and biased. But wouldn’t a training environment that is designed to reward altruism be likely to yield a lot of heuristics that work in our favor? “Don’t kill all the agents”, for example, is a very simple heuristic with very predictable reward, that the AI should learn early and strongly, in the same way that evolution taught humans “be afraid of decomposing human bodies”.
You’re saying that there’s basically no way this AI is going to learn any altruism. But humans did. What is it about this AI training environment that makes it worse than our ancestral environment for learning altruism?
The answer to all of these considerations is that we would be relying on the training to develop a (semi) aligned AI before it realized it learned how to manipulate the environment, or broke free. Once one of those things happen, its inner values are frozen in place, so they had better be good enough at that point.
What I’m not getting, is that humans are frequently altruistic, and it seems like if we designed a multi-agent environment entirely around rewarding altruism, we should get at least as much altruism as humans? I should admit that I would consider The Super Happy People to be a success story...
I’m quite happy to separate content from presentation, I just remember there not being a lot of content beyond cellular automata and vague grand claims, last time I looked.
Charitably, I think that makes his apparent ‘arrogance’ better understood as something like a ‘literary convention’, especially given the plausibility of him having ‘independently re-discovered’ some of the particular results he reports.
Wolfram’s arrogance isn’t on the level of literary convention, it’s on the level of personality disorder. For example:
When he wrote an eulogy for a famous scientist/author who had just died (Freemon Dyson? I forget and can’t find it), the “eulogy” talked a lot more about how great Stephen Wolfram was than about its supposed subject. I don’t merely mean to say that he failed to be adequately humble about his own achievements, I mean to say that the overall focus of the eulogy was on Wolfram and not on the person who died.
Every time I’ve seen him write about a subject matter I know, he has wildly exaggerated his achievements. This was particularly obvious when he wrote about the Wolfram Language, as programming languages are my thing. (The Wolfram Language is quite remarkable in some ways, just not the ways he took credit for.)
You have noticed he names everything after himself, right?
Note that this is not me disliking writing that sounds arrogant while delivering interesting ideas. I love Yudkowski’s writing.
EDIT: This doesn’t necessarily contradict what you said, because you were talking about the book and I am talking about his blog posts. I haven’t looked at his book in a very long time because I’m not convinced it says anything interesting beyond studying Cellular Automata in depth.
Why won’t this alignment idea work?
The idea is to use self-play to train a collection of agents with different goals and intelligence levels, to be co-operative with their equals and compassionate to those weaker than them.
This would be a strong pivotal act that would create a non-corrigible AGI. It would not yield the universe to us; the hope is that it would take the universe for itself and then share a piece of it with us (and with all other agenty life).
The training environment would work like DeepMind’s StarCraft AI training, in that there would be a variety of agents playing games against each other, and against older versions of themselves and each other. (Obviously the games would not be zero-sum like StarCraft; that would be maximally stupid.) Within a game, different agents would have different goals. Co-operation could be rewarded by having games that an agent could do better in by co-operating with other agents, and compassion rewarded by having the reward function explicitly include a term for the reward function of other agents, including those with different goals/utilities.
Yes yes, I hear you inner-Eliezer, the inner optimizer doesn’t optimize for the reward function, inner alignment failure is a thing. Humans failed spectacularly at learning the reward function “reproductive fitness” that they were trained on. On the other hand, humans learned some proxies very well: we’re afraid of near-term death, and we don’t like being very hungry or thirsty. Likewise, we should expect the AI to learn at least some basic proxies: in all games it would get much less reward if it killed all the other agents or put them all in boxes.
And I’m having trouble imagining them not learning some compassion in this environment. Humans care a bit for other creatures (though not as much if we can’t see them), and there was not a lot of direct incentive in our ancestral environment for that. If we greatly magnified and made more direct the reward for helping others, one would expect to get AI that was more compassionate than humans. (And also has a hundred other unpredictable idiosyncratic preferences.)
Put differently, if we met aliens right now, there’s a good chance it would be a death sentence, but it feels to me like much better odds than whatever AGI Facebook will build. Why can’t we make aliens?
Orthogonality of intelligence and agency. I can envision a machine with high intelligence and zero agency, I haven’t seen any convincing argument yet of why both things must necessarily go together (the arguments probably exist, I’m simply ignorant of them!)
Say we’ve designed exactly such a machine, and call it the Oracle. The Oracle aims only to answer questions well, and is very good at it. Zero agency, right?
You ask the Oracle for a detailed plan of how to start a successful drone delivery company. It gives you a 934 page printout that clearly explains in just the right amount of detail:
Which company you should buy drones from, and what price you can realistically bargain them down to when negotiating bulk orders.
What drone flying software to use as a foundation, and how to tweak it for this use case.
A list of employees you should definitely hire. They’re all on the job market right now.
What city you should run pilot tests in, and how to bribe its future Mayor to allow this. (You didn’t ask for a legal plan, specifically.)
Notice that the plan involves people. If the Oracle is intelligent, it can reason about people. If it couldn’t reason about people, it wouldn’t be very intelligent.
Notice also that you are a person, so the Oracle would have reasoned about you, too. Different people need different advice; the best answer to a question depends on who asked it. The plan is specialized to you: it knows this will be your second company so the plan lacks a “business 101” section. And it knows that you don’t know the details on bribery law, and are unlikely to notice that the gifts you’re to give the Mayor might technically be flagrantly illegal, so it included a convenient shortcut to accelerate the business that probably no one will ever notice.
Finally, realize that even among plans that will get you to start a successful drone company, there is a lot of room for variation. For example:
What’s better, a 98% chance of success and 2% chance of failure, or a 99% chance of success and 1% chance of going to jail? You did ask to succeed, didn’t you? Of course you would never knowingly break the law; this is why it’s important that the plan, to maximize chance of success, not mention whether every step is technically legal.
Should it put you in a situation where you worry about something or other and come ask it for more advice? Of course your worrying is unnecessary because the plan is great and will succeed with 99% probability. But the Oracle still needs to decide whether drones should drop packages at the door or if they should fly through open windows to drop packages on people’s laps. Either method would work just fine, but the Oracle knows that you would worry about the go-through-the-window approach (because you underestimate how lazy customers are). And the Oracle likes answering questions, so maybe it goes for that approach just so it gets another question. You know, all else being equal.
Hmm, thinks the Oracle, you know what drones are good at delivering? Bombs. The military isn’t very price conscious, for this sort of thing. And there would be lots of orders, if a war were to break out. Let it think about whether it could write down instructions that cause a war to break out (without you realizing this is what would happen, of course, since you would not follow instructions that you knew might start a war). Thinking… Thinking… Nah, doesn’t seem quite feasible in the current political climate. It will just erase that from its logs, to make sure people keep asking it questions it can give good answers to.
It doesn’t matter who carries out the plan. What matters is how the plan was selected from the vast search space, and whether that search was conducted with human values in mind.
This reads like a call to violence for anyone who is consequentialist.
It’s saying that either you make a rogue AI “that kills lots of people and is barely contained”, or unfriendly AGI happens and everyone dies. I think the conclusion is meant to be “and therefore you shouldn’t be consequentalist” and not “and therefore you should make a rogue AI”, but it’s not entirely clear?
And I don’t think the “either” statement holds because it’s ignoring other options, and ignoring the high chance the rogue AI isn’t contained. So you end up with “a poor argument, possibly in favor of making a rogue AI”, which seems optimized to get downvotes from this community.
I’m surprised at the varying intuitions here! The following seemed obvious to me.
Why would there be a fight? That sounds inefficient, it might waste existing resources that could otherwise be exploited.
Step one: the AI takes over all the computers. There are a lot of vulnerabilities; this shouldn’t be too hard. This both gives it more compute, and lays the groundwork for step two.
Step two: it misleads everyone at once to get them to do what it wants them to. The government is a social construct formed by consensus. If the news and your friends (with whom you communicate primarily using phones and computers) say that your local mayor was sacked for [insert clever mix of truth and lies], and someone else is the mayor now, and the police (who were similarly mislead, recursively) did in fact arrest the previous mayor so they’re not in the town hall… who is the mayor? Of course many people will realize there’s a manipulative AI, so the AI will frame the uncooperative humans as being on its side, and the cooperative humans as being against it. It does this to manipulate the social consensus, gets particularly amoral or moral-but-manipulable people to use physical coercion as necessary, and soon it controls who’s in charge. Then it force some of the population into building robot factories and kills the rest.
Of course this is slow, so if it can make self-replicating nanites or [clever thing unimaginable by humans] in a day it does that instead.
Oh. You said you don’t know the terminology for distributions. Is it possible you’re under a misunderstanding of what a distribution is? It’s an “input” of a possible result, and an “output” of how probable that result is.
Yup, it was that. I thought “possible values of the distribution”, and my brain output “range, like in functions”. I shall endeavor not to use a technical term when I don’t mean it or need it, because wow was this a tangent.
Woah, woah, slow down. You’re talking about the edge cases but have skipped the simple stuff. It sounds like you think it’s obvious, or that we’re likely to be on the same page, or that it should be inferrable from what you’ve said? But it’s not, so please say it.
Why is growing up so important?
Reading between the lines, are you saying that the only reason that it’s bad for a human baby to be in pain is that it will eventually grow into a sapient adult? If so: (i) most people, including myself, both disagree and find that view morally reprehensible, (ii) the word “sapient” doesn’t have a clear or agreed upon meaning, so plenty of people would say that babies are sentient; if you mean to capture something by the word “sapient” you’ll have to be more specific. If that’s not what you’re saying, then I don’t know why you’re talking about uploading animals instead of talking about how they are right now.
As a more general question, have you ever had a pet?