Thanks! Glad you got good stuff out of it.
I won’t edit the post, due to markdown and latex issues, but thanks for pointing out the typos.
Stuart_Armstrong(Stuart Armstrong)
Karma: 15,890
AI, learn to be conservative, then learn to be less so: reducing side-effects, learning preserved features, and going beyond conservatism
Sigmoids behaving badly: arXiv paper
Immobile AI makes a move: anti-wireheading, ontology change, and model splintering
Reward splintering as reverse of interpretability
What are biases, anyway? Multiple type signatures
The multiplication example is good, and I should have thought about it and worked it into the post.
I have only very limited access to GPT-3; it would be interesting if others played around with my instructions, making them easier for humans to follow, while still checking that GPT-3 failed.
What does GPT-3 understand? Symbol grounding and Chinese rooms
Reward splintering for AI design
Here are a few examples of model splintering in the past:
The concept of honour; which includes concepts such as: “nobility of soul, magnanimity, and a scorn of meanness” [...] personal integrity [...] reputation [...] fame [...] privileges of rank or birth [...] respect [...] consequence of power [...] chastity”. That is a grab-bag of different concepts, but in various times and social situations, “honour” was seen as single, clear concept.
Gender. We’re now in a period where people are questioning and redefining gender, but gender has been splintering for a long time. In middle class Victorian England, gender would define so much about a person (dress style, acceptable public attitudes, genitals, right to vote, right to own property if married, whether they would work or not, etc...). In other times (and in other classes of society, and other locations), gender is far less informative.
Consider a Croat, communist, Yugoslav nationalist in the 1980s. They would be clear in their identity, which would be just one thing. Then the 1990s come along, and all these aspects come into conflict with each other.
Here are a few that might happen in the future; the first two could result from technological change, while the last could come from social change:
A human subspecies created who want to be left alone without interactions with others, but who are lonely and unhappy when solitary. This splinters preferences and happiness (more than they are today), and changes the standard assumptions about personal freedom and
A brain, or parts of a human brain, that loop forever through feelings of “I am am happy” and “I want this moment to repeat forever”. This splinters happiness-and-preferences from identity.
We have various ages of consent and responsibility; but, by age 21, most people are taken to be free to make decisions, are held responsible for their actions, and are seen to have a certain level of understanding about their world. With personalised education, varying subcultures, and more precise psychological measurements, we might end up in a world where “maturity” splinters into lots of pieces, with people having different levels of autonomy, responsibility, and freedom in different domains—and these might not be particularly connected with their age.
A very good point.
I’d add the caveat that a key issue in a job is not the just the content, but who you interact in. eg a graduate student job in a lab can be very interesting even if the work is mindless, because of the people you get to interact with.
Bayesianism versus conservatism versus Goodhart
This is a variant of my old question:
There is a button at your table. If you press it, it will give you absolute power. Do you press it?
More people answer no. Followed by:
Hitler is sitting at the same table, and is looking at the button. Now do you press it?
Underlying model of an imperfect morphism
Nope, that’s not the model. Your initial expected population is . After the anthropic update, your probabilities of being in the boxes are , , and (roughly , , and ). The expected population, however is . That’s an expected population update of 3.27 times.
Note that, in this instance, the expected population update and the probability update are roughly equivalent, but that need not be the case. Eg if your prior odds are about the population being , , or , then the expected population is roughly , the anthropic-updated odds are , and the updated expected population is roughly . So the probability boost to the larger population is roughly (, but the boost to the expected population is roughly .
Anthropic updates do not increase the probability of life in general; they increase the probability of you existing specifically (which, since you’ve observed many other humans and heard about a lot more, is roughly the same as the probability of any current human existing), and this might have indirect effects on life in general.
So they does not distinguish between “simple life is very hard, but getting from that to human-level life is very easy” and “simple life is very easy, but getting from that to human-level life is very hard”. So panspermia remains at its prior, relative to other theories of the same type (see here).
However, panspermia gets a boost from the universe seeming empty, as some versions of panspermia would make humans unexpectedly early (since panspermia needs more time to get going); this means that these theories avoid the penalty from the universe seeming empty, a much larger effect than the anthropic update (see here).
Yep.
Yep. Though I’ve found that, in most situations, the observations “we don’t see anyone” has a much stronger effect than the anthropic update. It’s not always exactly comparable, as anthropic updates are “multiply by and renormalise”, while observing no-one is “multiply by and renormalise”—but generally I find the second effect to be much stronger.
Hey there!
I haven’t been working much on population ethics (I’m more wanting to automate the construction of values from human preferences so that an AI could extract a whole messy theory from it).
My main thought on these issues is to set up a stronger divergence between killing someone and not bringing them into existence. For example, we could restrict preference-satisfaction to existing beings (and future existing beings). So if they don’t want to be killed, that counts as a negative if we do that, even if we replace them with someone happier.
This has degenerate solutions too—it incentivises producing beings that are very easy to satisfy and that don’t mind being killed. But note that “create beings that score max on this utility scale, even if they aren’t conscious or human” is a failure mode for average and total utilitarianism as well, so this isn’t a new problem.