Thanks for taking the effort to write this up. I was definitely wrong.
Rafael Harth
Karma: 3,293
An important part of HPMoR is that a substantial fraction of Harry’s earlier behavior is later revealed to be unhelpful and prideful and to have overall made the world substantially worse.
This was not my impression. I recall rationality mistakes he makes, and somewhere near the end, an overall takeaway that’s somewhat like “you generally won’t apply rationality properly unless you’ve also gotten practical experience”. But all of those indicate that his rationality wasn’t enough. I don’t recall any revelation indicating that he was going into the wrong direction, was too arrogant, or anything of the sort.
Maybe the one exception is that he was too rude to Dumbledore specifically, but this seems too narrow to warrant that overall description.
Could be wrong, of course. Do you have examples?
Does anyone happen to know the maximum number of inputs to any element in a digital computer at the level of electronic gates? E.g., there are of course AND and OR gates, so the number is at least 2. But you can also set up an electronic circuit such that it implements, e.g., an AND with 3 inputs (though this is never required since you can make a 3-input AND out of two 2-input ANDs, and similarly for any other gate with 3+ inputs). Hence the question, what is the highest number of inputs that does physically exist in a typical CPU?
This question made sense to me for the first four bullet points, then at “item 1 - item2” and “item 3″ and then “into this one” I got really confused. You may want to reword it.
One comment: taken literally, this description
(note: I use the phrase “net happiness” to mean the sum of all happiness in the future minus the sum of all suffering in the future)
completely eliminates second-order effects (and I answered accordingly). However, I suspect many people will consider second-order effects anyway. If something is sufficiently strange, I think explicitly stating it, even in unambiguous language, it is often not enough; you have to make it super-extra clear.
Seems misleading to me because
-
ImE, the premise is untrue. My Windows crashes much more rarely than it used to, my PC boots much faster, most websites I use work more smoothly than they used to, etc.
-
I suspect observations about regular software aren’t all that relevant for Machine Learning.
-
Ok, you’re probably right. The one thing that confuses me about it is that I tend to think people’s reports of their degree of suffering are reliable, but maybe that’s not true.
Or maybe the course created exceptions.
If you’re talking amount of possible superintelligent AIs, then yeah, definitely. (I don’t think it’s likely to have a large number all physically instantiated.)
I tend to consider Roko’s basilisk to be an information hazard and wasn’t thinking or saying anything specific to that. I was only making the general point that any argument about future AI must take into account that the distribution is extremely skewed. It’s possible that the conclusion you’re trying to reach works anyway.
If you build an AI to produce paperclips, then you’ve built it to add value to the economy, and that presumably works until it kills everyone—that’s what I meant. Like, an AI that’s build to do some economically useful task but then destroys the world because it optimizes too far. That’s still a very strong restriction on the total mind-space; most minds can’t do useful things like building paperclips.
(Note that “Paperclip” in “Paperclip maximizer” is a standin for anything that doesn’t have moral status, so it could also be a car maximizer or whatever. But even literal paperclips have economic value.)
Any post or article that speculates what future AIs will be like is about the probability distribution of AIs, it’s just usually not framed in that way.
I think you’re making an error by equivocating between two separate meanings of “impossible to predict”:
you can make absolutely no statements about the thing
you cant predict the thing narrowly
The latter is arguably true, but the former is obviously false. For example, one thing we can confidently predict about AI is that it will add value to the economy (until it perhaps kills everyone). This already shifts the probability distribution massively away from the uniform distribution.
If you actually pulled out a mind at random, it’s extremely likely that it wouldn’t do anything useful.
Like I think most probability mass comes from a tiny fraction of mind space, probably well below of the entire thing, but the space is so large that even this fraction is massive. But it means many-gods style arguments don’t work automatically.
This sounds like enlightenment to me. Enlightenment is the absence of suffering, not the absence of pain or bodily reactions. If both reports are true, then the person was reacting normally in terms of observable symptoms but not experiencing any suffering as a result.
But AI isn’t pulled randomly out of mindspace. Your argument needs to be about the probability distribution of future AIs to work.
(We may not be able to align AI, but you can’t jump from that to “and therefore it’s like a randomly sampled mind”.)
I’m confused about equivalence between “god” and “AI”. I know the many gods objection in the context of Pascal’s Wager. Are you talking about AI or about gods? Or is the idea that you expect a superintelligent AI in the future which will have the relevant properties of a god?
My initial reaction is that the analogy doesn’t work because AI is unlikely to care at all about what you’re doing.
The problem seems under-specified to me. What’s Paul’s utility function?
I think this is how I do make decisions in social contexts where I’m not running on instinct. I’m less convinced that expected social value is its own category. Seems more like applying regular expected value to a problem that happens to be social
Also, nitpick:
Our estimated probabilities of a person doing X after we do Y can be wildly off.
It doesn’t make too much sense to say that you assign the wrong probabilities. (What are the right probabilities?) It’s more that your probability distribution has a lot of uncertainty.[1]
And (2) is also not really specific to social contexts.
- ↩︎
“entropy” would be the corresponding mathematical property
I think most people are motivated by prestige to a significant degree, and they can produce valuable content anyway.
My advice for writing good posts in the future would be: if you have something to say,[1] write it down as simply as possible. One sentence. At most two. Once you have that, you can make it longer to communicate it. (And then proofread for grammar.) But if you can’t write it as 1-2 sentences, the reason is probably that you don’t really know what you’re saying, and then you shouldn’t publish the result.
- ↩︎
Also, maybe you don’t have anything to say! The universe isn’t fair, you’re not just going to have insights when you would like to write. That’s not how it works. You need to actually start with an insight.
- ↩︎
Private message. If you click on a user’s name, you find an option to message them. (Just click “Message”.)
I recommend checking out https://superhappiness.xyz/
Partial, I think. I mean, it’s all with respect to one variable.
(Looking at this post now, it really should have included concrete numbers in addition to the symbols, which would also have answered the question. Idk if I’ll get around to fixing it...)