Richard, I’m looking at the margins. The FAI is convinced that it’s humanity’s only protection against UFAIs. If UFAIs can wipe out humanity, surely the FAI is justified in killing a million or so people to protect itself, or perhaps even to make sure it’s capable of defeating UFAIs which have not yet been invented and whose abilities can only be estimated.
NancyLebovitz
Karma: 32,891
If you once tell a lie, the truth is ever after your enemy.
That isn’t true.
I’ve told lies when I was a kid. If I got caught I gave up rather than doing an epistomological attack.
Richard Kennaway: “I feel that X.” Every sentence of this form is false, because X is an assertion about the world, not a feeling. Someone saying “I feel that X” in fact believes X, but calling it a feeling instead of a belief protects it from refutation. Try replying “No you don’t”, and watch the explosion. “How dare you try to tell me what I’m feeling!”
If I say I feel something, I’m talking about an emotion. I don’t intend it to be an objective statement about the world, and I’m not offended if someone says it doesn’t apply to everyone else.
To Richard Kennaway:
Your original point, which I didn’t read carefully enough:
“I feel that X.” Every sentence of this form is false, because X is an assertion about the world, not a feeling. Someone saying “I feel that X” in fact believes X, but calling it a feeling instead of a belief protects it from refutation. Try replying “No you don’t”, and watch the explosion. “How dare you try to tell me what I’m feeling!”
“No, you don’t” sounds like a chancy move under the circumstances. Have you tried “How sure are you about X?” and if so, what happens?
More generally, statements usually imply more than one claim. If you negate a whole statement, you may think that which underlying claim you’re disagreeing with is obvious, but if the person you’re talking to thinks you’re negating a different claim, it’s very easy to end up talking past each other and probably getting angry at each other’s obtuseness.
My reply: If I say I feel something, I’m talking about an emotion.
You again: That prohibits you from saying “I feel that X”. No emotion is spoken of in saying “I feel that the Riemann hypothesis is true”, or “I feel that a sequel to The Hobbit should never be made”, or “I feel that there is no God but Jaynes and Eliezer (may he live forever) is His prophet”, or in any other sentence of that form. “I feel” and “that X” cannot be put together and make a sensible sentence.
If someone finds themselves about to say “I feel that X”, they should try saying “I believe that X” instead, and notice how it feels to say that. It will feel different. The difference is fear.”
It sounds to me as though you’ve run into a community (perhaps representative of the majority of English speakers) with bad habits. I, and the people I prefer to hang out with, would be able to split “I feel that x” into a statement about emotions or intuitions and a statement about the perceived facts which give rise to the emotions or intuitions.
I believe that “I believe that a sequel to The Hobbit should never be made” is emotionally based. Why would someone say such a thing unless they believed that the sequel would be so bad that they’d hate it?
Here’s something I wrote recently about the clash between trying to express the feeling that strong emotions indicate the truth and universality of their premises and the fact that real world is more complicated.
The universe isn’t set up to reward virtue.
I believe that ethics are an effort to improve the odds of good outcomes. So it’s not that the universe is set up to reward ethics, it’s that ethics are set up to follow the universe.
The challenge is that what we’re taught is good is a mixture of generally useful rules, rules which are more useful to the people in charge than to the people who aren’t, and mere mistakes.
Nothing is lost; the universe is honest, Time, like the sea, gives all back in the end, But only in its own way, on its own conditions: Empires as grains of sand, forests as coal, Mountains as pebbles. Be still, be still, I say; You were never the water, only a wave; Not substance, but a form substance assumed.
Elder Olson, 1968
Learning Methods might be a relevant system. It’s based on the idea that emotional and physical pain are information, and it’s important to override the impulse to shut them down so that you can use them as detailed signals.
I think the weakness of R[xx] systems is that they’re only working one move at a time. Any position suggests a number of strategies to a good player. Being stuck with a randomized system that tends towards plausible moves means that a move that builds in one direction could be weakened in the next round by a move that’s tending towards another good strategy.
Could R[xx] be improved (in theory—I think the calculations would be unmanageable) by considering more than one move at a time? Is there some number of moves where you don’t exactly have an R[xx] anymore because the program is more like simulating the way the real chess player thinks?
pdf23ds, thanks. I wasn’t sure whether the whole strategic situation could be implied in each position. Have people been able to learn anything about chess strategy by studying how programs play? Do program vs. program games make sense to people?
I wonder if there’s an implication that intelligence is being used to mean things that require effortful conscious thought for us.
Imagine a species that has very quick minds but less spacial sense than we do. They can catch and throw accurately, but only by thinking as they do it. They would see baseball as much more evidence of intelligence than we do.
Or a species with much more innate mathematical and logical ability than we have—they might put geometry on the same level that we put crows’ ability to count.
Is a beehive evidence of intelligence?
How about an international financial system?
I think you’re right that the subjects in the experiment simply don’t think of the 100% blue strategy, and I wonder if there’s any way to find out why it’s so unaesthetic that it doesn’t cross people’s minds.
My tentative theory is that conformity is a good strategy for dealing with people if you don’t have a definite reason for doing something else, and that the subjects are modeling the universe (or at least the random sequence) as conscious.
Introspecting, I think that choosing 100% blue also feels like choosing to be wrong some of the time, so some loss aversion kicks in, while doing a 70⁄30 strategy feels like trying to be right every time.
“Even a human” might just be a fair insult.
Tentative, but what about cases where you don’t trust your own judgement and suspect that you need to be shaken out of your habits?
I’d be interested in an essay about “the nonobvious difficulty of doing math”.
Wright either didn’t know or chose to ignore the thinking that led to Asimov’s Three Laws. While the laws themselves (that robots must keep humans from coming to harm, obey human orders, and preserve themselves, in that order of priority) are impossible to codify, the underlying insight that we make knives with hilts is sound. Science fiction has a dystopian/idiot inventor streak because that makes it easier to get the plot going.
From another angle, part of sf is amplifying aspects of the real world. We can wreck our lives in a moment of passion or bad judgement, or by following a bad idea repeatedly.
Having to figure out the neuroscience by yourself is not an especially good protection against mistakes. Knowing how to make a change is different from and easier than knowing how to debug a change.
I don’t think prohibiting textbooks is necessary or sufficient to give people the pleasure of making major discoveries. Some people are content to solve puzzles, but others don’t just want being right, they want to be right about something new. My feeling is that the world is always going to be more complex than what we know about it. I’m hoping that improved tools, including improved cognition, will mean that we’ll never run out of new things, including new general principles, to discover.
I agree with Psy-Kosh that advice should and would be available, and also something like therapy if you suspect that you’ve deeply miscalibrated yourself. However, there is going to more than one system of advice and of therapy because there isn’t going to be agreement on what constitutes an improvement.
Excuse me if it’s been covered here, but in an environment like that deciding, not just what you want, but what changes turn you into not-you is a hard problem.
In re lying when you’re trying to set up a research and invention organization: It seems to me that it would make recruiting difficult. The public impression of what you’re doing is going to be your lies, which makes it even harder to get the truth of what you’re doing to the people you want to work with. And even the discrepancy between your public and private versions doesn’t appear in some embarrassing form on the internet, you’re going to tend to attract sneaky people and repel candid people, and this will probably make it harder to have an organization which does what you want.
I should have read Michael Vassar’s original post in this thread more carefully.
I suspect that people’s fear of becoming more rational has at least as much to do with the perceived consequences of being more honest with themselves about what they’re doing as it does with the fear of having to tell the truth to other people.
Part of being thrown into a better future being scary is that (as you implied) some behaviors that you think of as completely normal will be strongly disapproved of and some behaviors that you think are revolting or pointless will be required.
The only way I can see “no public science” as workable would be if people’s ability to do experiments and math was greatly enhanced, both of which strike me as good things.
http://www.systems-thinking.org/columbo/columbo.htm
Complex System Rules of Thumb
Everything is connected to everything else.
You can never do just one thing.
There is no “away.”
There ain’t no such thing as a free lunch.
Nature knows best.
If ain’t what you don’t know that hurts you; it’s what you DO know that ain’t so.
“Obvious solutions” do more harm than good.
Look for high leverage points.
Nothing grows forever.
Don’t fight positive feedback; support negative feedback instead.
Don’t try to control the players, just change the rules.
Don’t make rules that can’t be enforced.
There are no simple solutions.
Good intentions are not enough.
High morality depends on accurate prophecy.
If you can’t make people self-sufficient, your aid does more harm than good.
There are no final answers.
Every solution creates new problems.
Loose systems are often better.
Don’t be fooled by system cycles.
Remember the Golden Mean.
Beware the empty compromise.
Don’t be a boiled frog.
Watch our for thresholds.
Competition is often cooperation in disguise.
Bad boundaries make bad governments.
Beware the Tragedy of the Commons.
Foresight always wins in the long run.
Probably of interest: The Happiness Myth by Jennifer Michael Hecht. The title’s not especially accurate. It’s actually about what people have generally said through history about what leads to happiness, and how much of what we believe now on the subject is true.
Science fiction fandom makes me happy. Tear it into two separate pieces, and the social network is seriously damaged.
Without going into details, I have some issues about romantic relationships—it’s conceivable that a boreana could make me happy (and I’m curious about what you imagine a boreana to be like), but I would consider that to be direct adjustment of my mind, or as nearly so as to not be different.
More generally, people tend to have friends and family members of the other sex. A twenty-year minimum separation is going to be rough, even if you’ve got “perfect” romantic partners.
If I were in charge of shaping utopia, I’d start with a gigantic survey of what people want, and then see how much of it can be harmonized. That would at least be a problem hard enough to be interesting for an AI.
If that’s not feasible, I agree that some incremental approach is needed.
Alternatively, how about a mildly friendly AI that just protects us from hostile AIs and major threats to the existence of human race? I realize that the human race will be somewhat hard to define, but that’s just as much of a problem for the “I just want to make you happy” AI.
It seems to me that an FAI would still be in an evolutionary situation. It’s at least going to need a goal of self-preservation [1] and it might well have a goal of increasing its abilities in order to be more effectively Friendly.
This implies it will have to somehow deal with the possibility that it might overestimate its own value compared to the humans it’s trying to help.
[1] What constitutes the self for an AI is left as a problem for the student.