gjm
Karma: 14,260
NewTop
It might. It’s pretty long. I don’t suppose you’d like to be more specific?
(If the answer is no, you wouldn’t, because all you mean is that it seems like the sort of discussion that might have useful things in it, then fair enough of course.)
Even if you are known to be the sort of person who will blow the whistle on blackmailers it is still possible that someone will try to blackmail you. How could that possibly involve a logical inconsistency? (People do stupid things all the time.)
You say
So he’s using a counterexample that’s predicated on a logical inconsistency and could not happen. If a decision theory fails in situations that couldn’t really happen, that’s actually not a problem.
but that counterexample isn’t predicated on a logical inconsistency. An FDT agent can still get blackmailed, by someone who doesn’t know it’s an FDT agent or who doesn’t care that the outcome will be bad or who is lousy at predicting the behaviour of FDT agents.
None. Just giving impressions. I didn’t do anything like 25% either :-).
I’m not particularly interested in bounties and therefore am not trying to randomize anything. I’ve worked backwards from the end of the (alphabetically-ordered) list and find the following:
World Cleanup Day: Too early to tell.
Women’s suffrage movement: Unqualified success.
Women’s liberation movement: Big success. Movement (or something like it) is still around, but its present-day balance of good versus harm is controversial (or should I say “problematic”?).
Women Against War: actually two movements. One in the 1950s was anti-Vietnam-War; it doesn’t appear to have had much success, but the war did eventually end. The more recent one is, I think, too recent to evaluate.
Wikimedia: Still around and going strong. Not clear to me to what extent “social movement” is a good description of it, though.
Western Cape Anti-Eviction Campaign: Seems like it still exists and is still campaigning against evictions in South Africa. Appears to have inspired similar campaigns elsewhere in the world. I don’t know how much actual success it’s had.
White Wednesdays: this seems to be one person’s campaign rather than an actual movement, and at only ~18mo old too young to assess for this sort of problem.
Voluntary Human Extinction Movement: clearly hasn’t _succeeded_ since the human race is still here and its population is still growing; my impression is that VHEMT hasn’t had any impact to speak of, but I haven’t heard of it suffering anything Eternal-September-like.
Via Campesina: Still seems to be around and pursuing something like its original goals. I have no idea how much difference it’s actually making (still less whether any difference is actually beneficial, but that’s not really the point here).
Veganism: Still around, still pursuing original goals; I think roughly constant over time.
Yes, some categories are more useful than others for understanding the universe. Or for various other purposes. Categories more useful for one thing are not always more useful for another. (E.g., Scott’s example of hypothetical-Solomon’s Ministry of Dag and Ministry of Behemah; hypothetical-Solomon would not have been well served by trying to have a Ministry of Mammals instead.)
The fact that some categories are more useful than others doesn’t stop it being true that “the categories were made for man”. It just means that our choices of categories aren’t 100% arbitrary. And that’s OK, because Scott is not claiming that they are.
I’m not convinced that relativity is really a problem: it looks to me like you can probably deal with it as follows. Instead of asking about the state of the universe at time T and making T one parameter in the optimization, ask about the state of the universe within a spacetime region including O (where O is a starting-point somewhere around where the AI is to start operating) where now that region is a parameter in the optimization. Then instead of , use times some measure of the size of that region. (You might use something like total computation done within the region but that might be hard to define and as OP suggests it might not penalize everything you care about.) You might actually want to use the size of the boundary rather than of the region itself in your regularization term, to discourage gerrymandering. (Which might also make some sense in terms of physics because something something holographic principle something something, but that’s handwavy motivation at best.)
Of course, optimizing over the exact extent of a more-or-less-arbitrary region of spacetime is much more work than optimizing over a single scalar parameter. But in the context we’re looking at, you’re already optimizing over an absurdly large space: that of all possible courses of action the AI could take.
OK! I’ll look forward to those future posts.
(I’m a big surreal number fan, despite the skeptical tone of my comments here, and I will be extremely interested to see what you’re proposing.)
So far as anyone knows, no actual processes in the actual world are accurately described by surreal numbers. If not, then I suggest the same goes for the “nearest possible worlds” in which, say, it is possible for Mr Trump to be faced with the sort of situation described under the heading “Trumped”. But you can have, in a universe very much like ours, an endless succession of events of order-type . If the surreal numbers are not well suited to describing such situations, so much the worse for the surreal numbers.
And when you say “I don’t think the ordinary notion of sequence makes sense”, what it looks like to me is that you have looked at the ordinary notion of sequence, made the entirely arbitrary choice that you are only prepared to understand it in terms of surreal numbers, and indeed not only that but made the further arbitrary choice that you are only prepared to understand it if there turns out to be a uniquely defined surreal number that is the length of such a sequence, observed that there is not such a surreal number, and then said not “Oh, whoops, looks like I took a wrong turn in trying to model this situation” but “Bah, the thing I’m trying to model doesn’t fit my preconceptions of what the model should look like, therefore the thing is wrong”. You can’t do that! Models exist to serve the things they model, not the other way around.
It’s as if I’d just learned about the ordinals, decided that all infinite things needed to be described in terms of the ordinals, was asked something about a countably infinite set, observed that such a set is the same size as but also the same size as and , and said “I don’t think the notion of countably infinite set makes sense”. It makes perfectly good sense, I just (hypothetically) picked a bad way to think about it: ordinals are not the right tool for measuring the size of a (not-necessarily-well-ordered) set. And likewise, surreal numbers are not the right tool for measuring the length of a sequence.
Don’t get me wrong; I love the surreal numbers, as an object of mathematical study. The theory is gorgeous. But you can’t claim that the surreal numbers let you resolve all these paradoxes, when what they actually allow you to do is to replace the paradoxical situations with other entirely different situations and then deal with those, while rejecting the original situations merely because your way of trying to model them doesn’t work out neatly.
If the number of days is, specifically, , then the days are numbered 0, 1, 2, …, with precisely the (ordinary, finite) non-negative integers occurring. They are all smaller than . The number isn’t the limit or the least upper bound of those finite integers, merely the simplest thing bigger than them all.
If you are tempted to say “No! What I mean by calling the number of days is precisely that the days are numbered by all the omnific integers below .” then you lose the ability to represent a situation in which Trump suffers this indignity on a sequence of days with exactly the order-type of the first infinite ordinal , and that seems like a pretty serious bullet to bite. In particular, I think you can’t call this a solution to the “Trumped” paradox, because my reading of it—even as you tell it! -- is that it is all about a sequence of days whose order-type is .
Rather a lot of these paradoxes are about situations that involve limiting processes of a sort that doesn’t seem like a good fit for surreal numbers (at least so far as I understand the current state of the art when it comes to limiting processes in the surreal numbers, which may not be very far).
Though spelling was pretty flexible in and after Shakespeare’s time! He doesn’t seem to have used “Shakespear” himself (it looks as if “Shakspere” may have been his usual spelling) but that was the usual spelling in the late 18th century :-).
Was the name “Event Horizon” chosen specifically in reference to what happens when you pack things as densely into a small space as you can?
A bunch of us have commented to say how much we hate the comment-abridging “feature”. It would be good to hear from the other side.
Hear ye, hear ye! If you (1) think comment-abridgement is an improvement and (2) aren’t one of the admins responsible for implementing it (whose reasons I think we already know), please comment below and tell us why. Thanks!
I’ve already been whingeing at Ben in PMs about the comment-abridging system; Ben, if you’re reading this, feel free to ignore it since I promised I’d shut up about it :-). Anyway, this is just to go on record as really disliking this “feature”, which I think encourages superficial skimming over careful reading, makes it a big pain to engage fully with what others have said (because you have to expand lots of things separately), and just generally feels to me as if it does nothing I can see why I should actually want.
I agree with Raemon that any choice about how comments behaviour will shape what sort of conversations are had. I think this particular choice will encourage the wrong sort of conversations. And it makes LW harder to read.
There are various different culinary measures called a cup, but if filled with water they all give you between 1⁄5 and 1⁄4 of a kilogram. Actual cups-for-drinking-from can give wildly different amounts but a kilogram would correspond to a litre of water, and vessels that large usually have names other than “cup”.
For me right now (Firefox on Windows 10) neither those mysterious extra symbols nor any placeholders for them appear.
The actual stream of bytes looks something like this: Completeness%3A%20%E2%88%80x%2C%20y%20%E2%88%88%20X%2C%20either%20x%20%E2%89%BC%20y%20or%20y%20%E2%89%BC%5C%5Cu0016%20x
The result of “unquoting” this is: Completeness% [forall]x, y [element] X, either x [leq] y or y [leq]\\u0016 x where the things in square brackets represent single Unicode characters each represented by three UTF-8-encoded octets.
This stuff is all inside a <script type=“text/inject-data”> element, which seems to be some Meteor thing and I don’t know what gets done with it—but presumably it’s being processed by something that interprets backslashed Unicode escapes. \u0016 is an old ASCII control character (yes, the ASCII control characters have Unicode code points assigned to them), the one called SYN. I have absolutely no idea what is the “correct” behaviour for a web browser asked to display a SYN character.
You must have larger cups than me.
I think the analogy to the round earth is unhelpful, for essentially the reasons Said Achmiz’s comment alludes to. More relevant would be an infinite plane (where you can always move west and there is a consistent global notion of west-ness).
In fairness, “bureaucracy” is an unusually difficult word to spell correctly (though splitting it as bureau/-cracy, the former being an English word and the latter the suffix in democracy, autocracy, etc., helps) and Christian is not a native anglophone.
The paper discusses two specific blackmail scenarios. One (“XOR Blackmail”) is a weirdly contrived situation and I don’t think any of what wo says is talking about it. The other (“Mechanical Blackmail”) is a sort of stylized version of real blackmail scenarios, and does assume that the blackmailer is a perfect predictor. The paper’s discussion of Mechanical Blackmail then considers the case where the blackmailer is an imperfect (but still very reliable) predictor, and says that there too an FDT agent should refuse to pay.
wo’s discussion of blackmail doesn’t directly address either of the specific scenarios discussed in the FDT paper. The first blackmail scenario wo discusses (before saying much about FDT) is a generic case of blackmail (the participants being labelled “Donald” and “Stormy”, perhaps suggesting that we are not supposed to imagine either of them as any sort of perfectly rational perfect predictor...). Then, when specifically discussing FDT, wo considers a slightly different scenario, which again is clearly not meant to involve perfect prediction, simulation, etc., because he says things like ” FDT says you should not pay because, if you were the kind of person who doesn’t pay, you likely wouldn’t have been blackmailed” and “FDT agents who are known as FDT agents have a lower chance of getting blackmailed” (emphasis mine, in both cases).
So. The paper considers a formalized scenario in which the blackmailer’s decision is made on the basis of a perfectly accurate prediction of the victim, but then is at pains to say that it all works just the same if the prediction is imperfect. The blog considers only imperfect-prediction scenarios. Real-world blackmail, of course, never involves anything close to the sort of perfect prediction that would make it take a logical inconsistency for an FDT agent to get blackmailed.
So taking the paper and the blogpost to be talking only about provably-perfect-prediction scenarios seems to me to require (1) reading the paper oddly selectively, (2) interpreting the blogpost very differently from me, and (3) not caring about situations that could ever occur in the actual world, even though wo is clearly concerned with real-world applicability and the paper makes at least some gestures towards such applicability.
For the avoidance of doubt, I don’t think there’s necessarily anything wrong with being interested primarily in such scenarios: the best path to understanding how a theory works in practice might well go via highly simplified scenarios. But it seems to me simply untrue that the paper, still more the blogpost, considers (or should have considered) only such scenarios when discussing blackmail.