The nuclear famine paper I analyzed earlier lied about this
I think it pays to have a higher bar for calling something a “lie”. The paper does explicitly state their assumption that stores last only one year. You provide good evidence that this is a bad assumption, and it’s possible that it was made in bad faith (i.e. the authors maybe knew it was a bad assumption), but I think calling it a lie based on current evidence causes more heat than light.
Decision theories are not about what kind of agent you want to be. There is no one on god’s green earth who disputes that the types of agents who one box are better off on average. Decision theory is about providing a theory of what is rational.
Taboo the word “rational”—what real-world effects are you are actually trying to accomplish? Because if FDT makes me better off on average and CDT allows me to write “look how rational I am” in my diary, then there’s a clear winner here.
I don’t doubt that in the heat of battle, we’re going to be asking some version of “is nuking women and children cheaper than firebombing them”. But discussing the ethical questions outside of battle directly feeds into this calculation: because humanity has decided that nuking civilians is immoral, even a commander with no moral compass knows that nuking civilians may have reputational/legal impacts on themselves and their country, and may cause the enemy to retaliate in kind where it would not have otherwise: in other words, the “cost” of using the nuke has gone up, hopefully to the point that they’ll decide alternatives are “cheaper”.
I think one key reason this can’t be done is the Copenhagen Interpretation of Ethics—once you’ve interacted with the problem by allowing people to settle in your (former) land, people will now blame you for anything that happens, including the fact that they don’t have all your country’s nice citizenship benefits (even though they didn’t have those benefits back home either).
An unrelated issue is that the incentives might be built wrong: if failure means absorption into the host country, why bother succeeding?
One key challenge will be both a) being nonpartisan, and also b) appearing nonpartisan. If you’re not actually aiming for (a), then for honesty’s sake you should not aim for (b) either. But if you are actually aiming for (a) as you claim, then I would avoid including “stopping Donald Trump” in your stated goals as that hurts your (b). I know from personal experience that there are at least some people who’d support a nonpartisan version of your effort but also voted for Trump as (what they saw as) the lesser of two lizards; there is no reason to alienate those people like this.
EA is an optimization of altruism with “suboptimal” human tendencies like morality and empathy stripped from it
I so utterly disagree with this statement. Indeed I think one can almost summarize EA as the version of altruism that’s been optimized for morality, instead of solely empathy (though it includes that too!).
All I can see on this page are the (sub)section headers but no other content??
Just wanted to thank you for trying this!
Minor note: would you be willing to change the name? two extremely similar projects with extremely similar names is asking for trouble.
Note that an observer must always see that all previous prophets of existential doom were wrong. (Yay anthropics. I’m not sure if or how much this should change our reasoning though.)
re “No Long-Run Average Frequency” and “Not Useful to Decision Making”: You say that there is no way to assign a probability to “I am L”, and consequently no “valid strategy” for problems that rely on that information. Consider the following two games:
Game 1: You have been fissioned once. You may say ‘I am L’ and get paid 1000$ if correct, or ‘I am not L’ and get paid 999$ if correct. Game 2: You have been fissioned twice (with names LL, LR, RL, RR). You may say ‘I am LR’ and get paid 1000$ if correct, or ‘I am not LR’ and get paid 999$ if correct.
What move would you personally actually make in each of these games, and why?
This is what I’d do:
I’d pick ‘I am L’ in the first game and ‘I am not LR’ in the second
I’d justify that by writing down the inequalities “0.5 * 1000 > 0.5 * 999” and “0.25 * 1000 < 0.75 * 999″
I’d use the word “probabilities” to refer to those numbers above that have decimal points
If you disagree on the first or second point (i.e. you would make different moves in the games, or you would justify your moves using different math), I’d love to hear your alternatives. If you disagree only on the third point, then it seems like a disagreement purely over definitions; you are welcome to call those numbers bleggs or something instead if you prefer, but once the games get more complicated and the math gets harder and you need help manipulating your bleggs, I think you’ll find perfectly usable advice in a probability textbook.
The title is definitely true: humans are irrational in many ways, and an AGI will definitely know that. But I think the conclusions from this are largely off-base.
the framing of “it must conclude we are inferior” feels like an unnecessarily human way of looking at it? There are a bunch of factual judgements anyone analyzing humans can make, some that have to do with ways we’re irrational and some that don’t (can we be trusted to keep to bargains if one bothered to make them? maybe not. Are we made of atoms which could be used for other things? yes); once you’ve established these kinds of individual questions, I’m not sure what you gain by summing it up with the term ‘inferior’, or why you think that the rationality bit in particular is the bit worth focusing on when doing that summation.
there is nothing inherently contradictory in a “perfectly rational” agent serving the goals of us “irrational” agents. When I decide that my irrational schemes require me to do some arithmetic, the calculator I pick up does not refuse to function on pure math just because it’s being used for something “irrational”. (Now whether we can get a powerful AGI to do stuff for us is another matter; “possible” doesn’t mean we’ll actually achieve it.)
How valid is it to assume that (approximately) everyone who got the heliocentrism question wrong got it wrong by “guessing”? If 18% got it wrong, then your model says that there’s 36% who had no clue and half guessed right, but at the other extreme there’s a model that everyone ‘knows’ the answer, but 18% ‘know’ the wrong answer. I’m not sure which is scarier − 36% clueless or 18% die-hard geocentrists—but I don’t think we have enough information here to tell where on that spectrum it is. (In particular, if “I don’t know” was an option and only 3% selected it, then I think this is some evidence against the extreme end of 36% clueless?)
(This is of course just my understanding of his model, but) yes. The analogy he uses is that while you cannot predict Stockfish’s next move in chess, you can predict for ‘certain’ that it will win the game. I think the components of the model are roughly:
it is ‘certain’ that, given the fierce competition and the number of players and the incentives involved, somebody will build an AGI before we’ve solved alignment.
it is ‘certain’ that if one builds an AGI without solving alignment first, one gets basically a random draw from mindspace.
it is ‘certain’ that a random draw from mindspace doesn’t care about humans
it is ‘certain’ that, like Stockfish, this random draw AGI will ‘win the game’, and that since it doesn’t care about humans, a won gameboard does not have any humans on it (because those humans were made of atoms which could be used for whatever it does care about)
I originally downvoted because I couldn’t figure out the point. The title definitely contributed to that, though not for the reason you suggest: rather, the title made it sound like this fact about a fictional world was the point, in the same way that you would expect to come away from a post titled “Rowling: Wizarding Britain has a GDP of 1 million Galleons” pretty convinced about a fictional GDP but with possibly no new insights on the real world. I think the new title makes it a bit clearer what one might expect to get out of this: it’s more like “here’s one (admittedly fictional) story of how alignment was handled; could we do something similar in our world?”. I’m curious if that’s the intended point? If so then all the parts analyzing how well it worked for them (e.g. the stuff about cryonics and 97%) still don’t seem that relevant: “here’s an idea from fiction you might consider in reality” and “here’s an idea from fiction which worked out for them you might consider in reality” provide me almost identical information, since the valuable core worth potentially adopting is the idea, but all discussion of whether it worked in fiction tells me extremely little about whether it will work in reality (especially if, as in this case, very little detail is given on why it worked in fiction).
AI in general is meant to do what we cannot do—else we would have done it, already, without AI—but to “make things better”, that we could very readily have done, already.
Strong disagree, for two related reasons:
AI in general is meant to do what we cannot do
Why must AI only be used for things we cannot do at all? e.g. the internet has made it much easier to converse with friends and family around the world; we could already do that beforehand in inferior or expensive ways (voice-only calls, prohibitively-frequent travel) but now we can also do it with easy video calling, which I think pretty clearly counts as “make things better”. It would be silly to say “the internet is meant to do what we cannot do, and we could very readily have already been calling home more often without the internet, so the internet can’t make things better”.
to “make things better”, that we could very readily have done… we certainly had many opportunities to create such a civilization… [etc]
Imagine a trade between two strangers in a setting without atomic swaps. What is the “upside” of a trusted third-party escrow agent? After all, whichever trader must hand over their goods first could ‘very readily’ just decide to trust the second trader instead of mistrusting them. And then that second trader could ‘very readily’ honestly hand over their goods afterward instead of running away with everything. Clearly since they didn’t come up with that obvious solution, they don’t really want to trade, and so this new (social) technology of escrow that would allow them to do so isn’t actually providing a real “upside”.
Possibly people overall are just better calibrated than I thought.
Note that the relevant reference class is not “people overall”; at the risk of overfitting, I’d say it should be something closer to “people who are mathematically literate, habitually make tons of predictions, and are at least aware of the concept of calibration”. It is far less surprising (though still surprising, I think) that a member of this group is this well calibrated.
The problem with this argument is that it’s implicitly assuming that all curves are lines, i.e. very roughly you argue that politicians on Twitter is bad, politicians not on Twitter is good, and therefore removing one politician from Twitter is a step in the bad->good direction so it’s good. But not all curves are lines—e.g. if Twitter banned half of all politicians but chose only the conservative half, I claim that’s worse than each of the endpoints of banning nobody and banning all politicians. (Well, except for the fact that with such an outrageous show of bias they might lose some of the hold they have over the world, which would be good.) So you have to argue more specifically that removing Trump and only Trump is still good. Might be doable, but I don’t think you did it—in fact you specifically replace that narrow question with the general one (“A better question than “why are we banning Trump from Twitter?” is instead “why are we encouraging high-profile politicians to engage with potential behavioral addictions?””)
committing to an earning-to-give path would be a bet on this situation being the new normal.
Is that true, or would it just be the much more reasonable bet that this situation ever occurs again? because at least in theory, a dedicated earning-to-give person could just invest money during the funding-flush times and then donate it the next time we’re funding-constrained?
Very cool! One nitpick:
The probability of having as unbalanced a string as #121 in a sample of 62 random strings is 12%… Given that balance between zeros and ones is one of the most basic things to look at, Round 2 participants got unlucky… Contestants justifiably gave probabilities near 0 for both strings.
This feels like a failure to apply Bayes. Let E = “#121 is 3.1 or more stddevs away from evenly split” and H = “#121 is ‘real’”. Then it is true that real randomness doesn’t do 3.1 stddevs very often, so P(E|H) is roughly 1:400 odds. But “given that balance between zeros and ones is one of the most basic things to look at”, human participants probably also aren’t going to do 3.1 stddevs very often, so P(E|!H) is also very low! How low? Based on the best dataset I have for how humans actually behave on this task (i.e. your dataset, so admittedly this method would not have actually been usable by contest participants), the odds a participant would create a string as unbalanced as #121 (i.e. 56 or fewer of whichever digit was rarer) is 0:62!! But there was 1 participant with each number of rare-digits between 57 and 60, so I’m going to irresponsibly eyeball that curve and say that P(E|!H) is probably somewhere between 1:50 and 1:200 odds. With our 1:400 from earlier, this means an odds ratio of between 2:1 and 8:1 in favor of #121 being human-generated; without more evidence to go on, the correct answer would have thus been between 11% and 33%. “Probabilities near 0″ are not justified, unless you’re incorporating other evidence as well, or you think both that the correct answer is nearer the 11% end of my range and also that 11% counts as “near 0”. (Could one guess P(E|!H) without seeing your dataset? Obviously I’m spoiled on the answers now so I don’t know, but I think I might have guessed that “one or two” of 62 participants would do something that extreme, which would correspond to an answer of ~10%.)
Unrelatedly, your post was extra fun for me because I happened to try something along the same lines (but much less well executed) as my middle school science fair project—I had people do Round 1 only (and only ~30 bits, not 150), and then just did some analysis myself to show ways they differed on average from random (primarily just looking at run lengths). Unfortunately I did not put any effort into getting motivated participants—I gathered data in part by getting teachers to force a bunch of uninterested kids to play along, and unsurprisingly ended up with some garbage responses like all 0s or strictly alternating 0s and 1s (and presumably most of the others, while not total garbage, did not represent best effort).
I don’t think you have any object-level disagreement with most people who say AI is “about to” shake things up. You just view “the tech is already such that this could theoretically be happening today” as sufficient justification to say things are being shaken up in the present tense, while others are using that same justification for the words “about to”, and are presumably waiting for real-world effects (significant shifts in commission art pricing, artists losing their jobs, any major project shifting their art budget to use this tech, etc.) before we say the shake up is actually happening.