AnnaSalamon

Karma: 17,484

AnnaSalamon 26 Nov 2007 8:00 UTC
7 points
on: The Hidden Complexity of Wishes
“You cannot predict, in advance, which of your values will be needed to judge the path through time that the genie takes.… The only safe genie is a genie that shares all your judgment criteria.”

Is a genie that does share all my judgment criteria necessarily safe?

Maybe my question is ill-formed; I am not sure what “safe” could mean besides “a predictable maximizer of my judgment criteria”. But I am concerned that human judgment under ordinary circumstances increases some sort of Beauty/Value/Coolness which would not be increased if that same human judgment was used to search over a less restricted set of possibilities.

The world is full of cases where selecting for A automatically increases B when you are searching over a restricted set of possibilities but does not increase B when those restrictions are lifted. Overfitting is a classic example. In cases of overfitting, if we search only over a restricted set of few-parameter models, models that do well on the training set will automatically do well on the generalization set, but if we allow more parameters the correlation disappears.

Modern marketing / product development can search over a larger set of alternatives than we used to have access to. In many cases human judgments correlate with less when used on modern manufactured goods than when used on the smaller set of goods that was formerly available. Judgments of tastiness used to correlate with health but now do not. Judgments of “this is a limited resource which I should grab quickly” used to indicate resources which we really should grab quickly but now do not (because of manufactured “limited time offer only” signs and the like).

Genies or AGI’s would search over an even larger space of possibilities than contemporary marketing searches over. In this larger space, many of the traditional correlates of human judgment will disappear. That is: in today’s restricted search spaces, outcomes which are ranked highly according to human judgment criteria tend also to have various other properties P1, P2, … Pk. In an AGI’s search space, outcomes which are ranked highly according to human judgment criteria will not have properties P1… Pk.

I am worried that properties P1...Pk are somehow valuable. That is, I am worried that in this world human judgments pick out outcomes that are somehow valuable and that human judgments’ ability to do this resides, not in our judgment criteria alone (which would be uploaded into our imagined genie) but in the conjunction of our judgment criteria with the restricted set of possibilities that has so far been available to us.

AnnaSalamon 27 Nov 2007 1:02 UTC
2 points
on: The Hidden Complexity of Wishes
“In what sense can [properties P1...Pk] be valuable, if they are not valued by human judgment criteria (even if not consciously most of the time)?”

I don’t know. It might be that the only sense in which something can be valuable is to look valuable according to human judgment criteria (when thoroughly implemented, and well informed, and all that). If so, my concern is ill-formed or irrelevant.

On the other hand, it seems possible that human judgments of value are an imperfect approximation of what is valuable in some other (external?) sense. Imagine for example if we met multiple alien races and all of them said “I see what you’re getting at with this ‘value/goodness/beauty/truth’ thing, but you are misunderstanding it a bit; in a few thousand years, you will modify your root judgment criteria in such-and-such a way.” In that case I would wonder whether my current judgment criteria were not best understood as an approximation of this other set of criteria and whether it was not value according to this other set of criteria that I should be aiming for.

If human judgment criteria are an approximation of some other kind of value, they would probably cease to approximate that other kind of value when used to search over the large space of genie-accessible possibilities.

By way of analogy, scientists’ criteria for judging scientific truth/relevance/etc. seem to be changing usefully over time, and it may be that scientists’ criteria at different times can be viewed as successive approximations of some other (external?) truth-criteria. Galilean physicists had one way of determining what to believe, Newtonians another, and contemporary physicists yet another. In the restricted set of situations considered by Galilean physicists, Galilean methods yield approximately the same predictions as the methods of contemporary physicists. In the larger space of genie-accessible situations, they do not.

AnnaSalamon 18 May 2008 3:52 UTC
18 points
on: Do Scientists Already Know This Stuff?
My impression:

(1) Yes, there are all kinds of good points in Eliezer’s posts that I was not taught in my science coursework or internships and that others are also not taught. Eliezer’s last few posts caused me to raise my (low) estimate of the probability that Eliezer and others can pull off the breakthroughs needed for FAI.

(2) No, the “science” that I and many others were taught in research apprenticeships is not exhausted by the “hypothesis, experiment, conclusion” scientific method that Eliezer has been discussing. It includes plenty of details about what do and do not constitute legitimate research questions and fruitful conjectures within a particular subfield. These details are supplied mainly by example and do not transfer well between different scientific subfields, unlike the general techniques Eliezer is after.

That is: “legitimate science”, as it is often taught, involves sticking to a narrow set of known mechanisms and to hypotheses that sound like previously successful hypotheses. Legitimate science includes “stuff similar to these established examples and nothing else”. It also recommends that an individual only propose hypotheses in subfields where he has been thoroughly steeped in both the formal results and the culture/traditions. This is a good enough notion of science to: (i) prevent many hypotheses along the lines of Eliezer_18′s, (ii) to label Penrose’s theories of consciousness unscientific, and (iii) to label detailed predictions about 2050 unscientific. (Which, indeed, is how many scientists I know regard both Penrose and futurists.) Unfortunately, this sort of scientific education does not show people how to do revolutionary science, nor does it allow scientists to distinguish between detailed stories about 2050 and simpler statements like “AI stands a good chance of eventually destroying the world by one means or another”. (The latter is branded “unscientific” in the same way the detailed sci-fi stories are branded “unscientific”: both are not made from the toolkit of known examples and mechanisms.)

(3) Like Z. M. Davis and others, I fear rhetorical disaster. Z. M. points out that railing against Mainstream Science is a frequent indicator of crackpottery. I’d like to generalize the principle: people get offended and impute lousy motives when someone talks overmuch about how he alone possesses unique knowledge and powers. Talking about how Bayescraft is completely different from everything else anyone has ever thought/taught, or even sounding like you’re doing so, suggests ego and risks causing offense. Especially if your competitors are at all caricatured or misrepresented.

AnnaSalamon 18 May 2008 5:27 UTC
7 points
on: Do Scientists Already Know This Stuff?
Eliezer,

I’m not suggesting you fail to pass on the warning. I’m suggesting you make sure the warning is placed in an accurate, non-caricatured picture of the scientific traditions you are criticizing.

For example, you talk about Science “not being strict enough”. The notion of “scientific” that I described in my comments (which is one of several competing, half-articulated modes of science in which students are sometimes trained) is in some ways too strict; it correctly throws out Penrose and detailed 2050 predictions, and it unfortunately also throws useful, simple hypotheses like “Most conceivable AIs would, if created, destroy the world.”

More accurate, less singular models of science would make your points easier to digest in some ways (because more accurate). More accurate models of science would also make your points less crackpot-sounding, but you may be right that I should ignore that aspect.

AnnaSalamon 23 May 2008 22:22 UTC
3 points
on: My Childhood Role Model
We might get a very different intelligence scale if we graph: (1) “how much mind-design effort is required to build that intelligence” than if we graph (2) “how much [complex problem solving / technology / etc.] that intelligence can accomplish”.

On scale (1), it is obvious that the chimp-human distance is immensely larger than the human-human distance. (The genetic differences took immensely longer to evolve).

On scale (2), it is less obvious. Especially if we compared linear rather than log differences in what Einstein, the village idiot, and the chimp can accomplish in the way of technological innovation.

AnnaSalamon 25 Jul 2008 22:30 UTC
2 points
on: Math is Subjunctively Objective
“But my dear sir, if the fact of 2 + 3 = 5 exists somewhere outside your brain… then where is it?”

The truth-condition for “There are five sheep in the meadow” concerns the state of the meadow.

My guess is that the truth condition for “2 + 3 = 5” concerns the (more complex, but unproblematically material) set of facts you present: the facts that e.g.: It’s easy to find sheep for which two sheep and three sheep make five sheep It’s fairly easy to build calculators that model what happens with the sheep It’s fairly easy to evolve brains that model what happens with the calculators and the sheep. It’s fairly easy to find “formal mathematical models” that can run on these evolved brains, that model what’s going on with the sheep and the rocks and various other systems, with axioms and rules of inference that can be briefly described in English.

We have good reason to claim that “2+3 = 5” has an existence outside your mind. We have such reason because, as you point out, we see many different material systems that “correlate with each other and successfully predict one another”.

But… do we have any reason to claim that “2+3=5” has an existence outside of these correlations between material systems? My guess is “no”. My guess is that we should say that “2+3=5” is about these correlations. Once we say this, we can go ahead and investigate these correlations the way we’d investigate other aspects of material systems: we can try to spell out just what systems do “correlate with each other and successfully predict one another” in the ways we summarize with addition, and then what systems correlate with one another in the ways we summarize with Euclidean geometry, and then look for the meta-level pattern that unites the two sets of correlations.

These questions about the correlations are interesting and partially unsolved. But my guess is that they aren’t gaps in our understanding of what math is about, just gaps in our understanding of the correlated material processes that math is about. The lines of questioning needed to explain these correlations are different from the lines of questioning that tend to be invoked when someone asks “where” math is.

AnnaSalamon 28 Jul 2008 20:54 UTC
0 points
on: Setting Up Metaethics
Seconding Doug, IL, Tiiba, and Z. M Davis re: Caledonian.

AnnaSalamon 24 Sep 2008 21:27 UTC
2 points
on: Horrible LHC Inconsistency
@michael vassar

For the probability of a die coming up “6” eight times out of nine, I get about 1 in 200 thousand, not 1 in 2 million. If the die coming up anything (e.g., “1″ or “3”) eight times out of nine would have been similarly notable, I get 1 in 37 thousand.

Why do you override the urge to see dark lords of the martrix in these sorts of coincidences? Calculations of how many such coincidences one would expect, given confirmation bias etc.? Belief that coincidence-detectable dark lords of the matrix are sufficiently unlikely that such calculations aren’t worth making? A desire not to look or be crazy?

AnnaSalamon 29 Sep 2008 12:14 UTC
0 points
on: Friedman’s “Prediction vs. Explanation”
Formalizing Vijay’s answer here:

The short answer is that you should put more of your probability mass on T1′s prediction because experts vary, and an expert’s past performance is at least somewhat predictive of his future performance.

We need to assume that all else is symmetrical: you had equal priors over the results of the next experiment before you heard the scientists’ theories; the scientists were of equal apparent caliber; P( the first twenty experimental results | T1 ) = P( the first twenty experimental results | T2); neither theorist influenced the process by which the next experiment was chosen; etc.

Suppose we have a bag of experts, each of which contains a function for generating theories from data. We draw a first expert from our bag at random and show him data points 1-10; expert 1 generates theory T1. We draw a second expert from our bag at random and show him data points 1-20: expert 2 generates theory T2.

Given the manner in which real human experts vary (some know more than others about a given domain; some aim for accuracy where others aim to support their own political factions; etc.), it is reasonable to suppose that some experts have priors that are well aligned with the problem at hand (or behave as if they do) while others have priors that are poorly aligned. Expert 1 distinguished himself by accurately predicting the results of experiments 11-20 from the results of experiments 1-10; many predictive processes would not have done so well. Expert 2 has only shown an ability to find some theory that is consistent with the results of experiments 1-20; many predictive processes put a non-zero prior on some such theory that would not have given the results of experiments 11-20 “most expected” status based only on the results from experiments 1-10. We should therefore expect better future performance from Expert 1, all else equal.

The problem at hand is complicated slightly in that we are judging, not experts, but theories, and the two experts generated their theories at different times from different amounts of information. If Expert 1 would have assigned a probability < 1 to results 11-20 (despite producing a theory that predicted those results), Expert 2 is working from more information than Expert 1, which gives Expert 2 at least a slight advantage. Still, given the details of human variability and the fact that Expert 1 did predict results 11-20, I would expect the former consideration to outweigh the latter.

AnnaSalamon 29 Sep 2008 13:30 UTC
0 points
on: Friedman’s “Prediction vs. Explanation”
Peter, your point that we have different beliefs about the theories prior to looking at them is helpful. AFAICT theories don’t screen off theorists, though. My belief that the college baseball team will score at least one point in every game (“theory A”), including the next one (“experiment 21″), may reasonably be increased by a local baseball expert telling me so and by evidence about his expertise. This holds even if I independently know something about baseball.

As to the effect of “number of parameters” on the theories’ probabilities, would you bet equally on the two theories if you were told that they contained an identical number of parameters? I wouldn’t, given the asymmetric information contained in the two experts vouching for the theories.

Tim, I agree that if you remove the distinct scientists and have the hypotheses produced instead by a single process (drawn from the same bucket), you should prefer whichever prediction has the highest prior probability. Do you mean that the prior probability is equal to the prediction’s simplicity or just that simplicity is a good rule of thumb in assigning prior probabilities? If we have some domain knowledge I don’t see why simplicity should correspond exactly to our priors; even Solomonoff Inducers move away from their initial notion of simplicity with increased data. (You’ve studied that math and I haven’t; is there a non-trivial updated-from-data notion of “simplicity” that has identical ordinal structure to an updated Solomonoff Inducer’s prior?)

Tyrrell, I like your solution a lot. A disagreement anyhow: as you say, if experts 1 and 2 are good probability theorists, T1 will contain the most likely predictions given the experimental results according to Expert 1 and T2 likewise according to Expert 2. Still, if the experts have different starting knowledge and at least one cannot see the other’s predictions, I don’t see anything that surprising in their “highest probability predictions given the data” calculations disagreeing with one another. This part isn’t in disagreement with you, but it also relevant that if the space of outcomes is small or if experiments 1-20 are part of some local regime that experiment 21 is not(e.g. physics at macroscopic scales, or housing prices before the bubble broke), it may not be surprising to see two theories that agree on a large body of data and diverge elsewhere. Theories that agree in one regime and disagree in others seem relatively common.

Alex, Bertil, and others, I may be wrong, but I think we should taboo “overfitting” and “ad hoc” for this problem and substitute mechanistic, probability-theory-based explanations for where phenomena like “overfitting” come from.

AnnaSalamon 29 Sep 2008 18:55 UTC
0 points
on: Friedman’s “Prediction vs. Explanation”
Hi Tyrrell,

Let T1_21 and T2_21 be the two theories’ predictions for the twenty-first experiment.

As you note, if all else is equal, our prior beliefs about P(T1_21) and P(T2_21) -- the odds we would’ve accepted on bets before hearing T1s and T2′s predictions—are relevant to the probability we should assign after hearing T1′s and T2′s predictions. It takes more evidence to justify a high-precision or otherwise low-prior-probability prediction. (Of course, by the same token, high precision and otherwise low-prior predictions are often more useful.)

The precision (or more exactly, the prior probability) of the predictions T1 and T2 assign to the first twenty experimental results are also relevant. The precision of these tested predictions, however, pulls in the opposite direction: if theory T1 made extremely precise, low-prior-probability predictions and got them right , this should more strongly increase our prior probability that T1′s set of predictions is entirely true. You can formalize this with Bayes’ theorem. [However, the obvious formalization only shows how probability of the conjunction of all of T1′s predictions increases; you need a model of how T1 and T2 were generated to know how indicative each theory’s track record is of its future predictive accuracy, or how much your beliefs about P(T1_21) specifically should increase. If you replace the scientists with random coin-flip machines, and your prior probability for each event is (1/2), T1′s past success shouldn’t increase your P(T1_21) belief at all.]

As to whether there is a single “best” metric for evaluating theories, you are right that for any one expert, with one set of starting (prior) beliefs about the world and one set of data with which to update those beliefs, there will be exactly one best (Bayes’-score-maximizing) probability to assign to events T1_21 and T2_21. However, if the two experts are working from non-identical background information (e.g., if one has background knowledge the other lacks), there is no reason to suppose the two experts’ probabilities will match even if both are perfect Bayesians. If you want to stick with the Solomonoff formalism, we can make the same point there: a given Solomonoff inducer will indeed have exactly one best (probabilistic) prediction for the next experiment. However, two different Solomonoff inducers, working from two different UTM’s and associated priors (or updating to two different sets of observations) may disagree. There is no known way to construct a perfectly canonical notion of “simplicity”, “prior probability” or “best” in your sense.

If you want to respond but are afraid of the “recent comments” limit, perhaps email me? We’re both friends of Jennifer Mueller’s (I think. I’m assuming you’re the Tyrrell McAllister she knows?), so between that and our Overcoming Bias intersection I’ve been meaning to try talking to you sometime. annasalamon at gmail dot com.

Also, have you read A Technical Explanation ? It’s brilliant on many of these points.

AnnaSalamon 2 Oct 2008 11:14 UTC
39 points
on: Awww, a Zebra
l, Aww, Ben_Wraith:

While I appreciate the effort toward optimal decision-making, surely there is some way to contribute without invading Eliezer’s personal life?

Can you picture anyone doing peak creative work while trying to justify every ounce of their resource use? To others or to themselves? Eliezer presumably knows he doesn’t need to do that, but… threads like this can’t help his or others’ morale. And morale is a precious resource.

Group efforts in general, and particularly philanthropic efforts, devolve too easily into shows of self-sacrifice. After all, sacrifice takes less effort in many ways, and it looks like trying . If we want to create a positive singularity, we’ll need to make our project fun, we’ll need to make the actual useful work attractive, we’ll need to get people aim for achievement (not for an appearance of “using all their resources”), and we’ll to make it something that real people want to join and don’t burn out at.

It isn’t only Eliezer who can help, by the way. If nothing else, you can help the effort get money; some of those willing and able able to do FAI research are spending their time raising money, right now, for lack of other ways to get money. If you find a way to gather money for the effort, more research will be done and the chances of a positive singularity will improve. There are other possibilities for helping, too. If you’re concerned about the future, perhaps take a look at what you have, who you know, and what you might do? Creating a positive singularity can be a lot of fun.

AnnaSalamon 6 Oct 2008 18:13 UTC
0 points
on: Bay Area Meetup for Singularity Summit
I’m game any time and live in Sunnyvale. Depending on who is in town when, we may want to consider two meet-ups to accommodate different schedules (e.g., one Thurs night and one Sat. night). Last time I went to an OB meet-up, in Manhattan, there were more interesting people than there was time to get to know all of them.

AnnaSalamon 12 Oct 2008 1:06 UTC
1 point
on: Crisis of Faith
Eliezer, what do you mean here? Do you mean:

(A1) Individuals in the reference class really are always better off with the truth, with sufficient probability that the alternative does not bear investigating;

(A2) Humans are so unreliable as judges of what we would and would not benefit from being deceived about that the heuristic “we’re always better off with the truth” is more accurate than the available alternatives;

(B) Individuals must adopt the Noble Might-be-truth “I’m always better off with the truth” to have a chance at the Crisis of Faith technique?

AnnaSalamon 16 Dec 2008 9:17 UTC
4 points
on: Not Taking Over the World
Wei,

Is it any safer to think ourselves about how to extend our adaptation-executer preferences than to program an AI to figure out what conclusions we would come to, if we did think a long time?

I’m thinking here of studies I half-remember about people preferring lottery tickets whose numbers they made up to randomly chosen lottery tickets, and about people thinking themselves safer if they have the steering wheel than if equally competent drivers have the steering wheel. (I only half-remember the studies; don’t trust the details.) Do you think a bias like that is involved in your preference for doing the thinking ourselves, or is there reason to expect a better outcome?

AnnaSalamon 14 Jan 2009 13:42 UTC
3 points
on: Building Weirdtopia
Roko, Instead of Shakespeare, perhaps resolve to get a sense of the knowledge, rationality, and general motivations and outlook of a larger set of people? Knowing who makes up our economy, governments, medical systems, and research institutions is as important as knowing how a lightbulb works for making good decisions in the world.

I’m far from perfect on this one; if anyone has suggestions for illuminating conversations to have with strangers, do share.

AnnaSalamon 6 Feb 2009 13:54 UTC
26 points
on: Epilogue: Atonement (8/8)
Aleksei, I don’t know what you think about the current existential risks situation, but that situation changed me in the direction of your comment. I used to think that to have a good impact on the world, you had to be an intrinsically good person. I used to think that the day to day manner in which I treated the people around me, the details of my motives and self-knowledge, etc. just naturally served as an indicator for the positive impact I did or didn’t have on global goodness.

(It was a dumb thing to think, maintained by an elaborate network of rationalizations that I thought of as virtuous, much the way many people think of their political “beliefs”/clothes as virtuous. My beliefs were also maintained by not bothering to take an actually careful look either at global catastrophic risks or even at the details of e.g. global poverty. But my impression is that it’s fairly common to just suppose that our intuitive moral self-evaluations (or others’ evaluations of how good of people we are) map tolerably well onto actual good consequences.)

Anyhow: now, it looks to me as though most of those “good people”, living intrinsically worthwhile lives, aren’t contributing squat to global goodness compared to what they could contribute if they spent even a small fraction of their time/money on a serious attempt to shut up and multiply. The network of moral intuitions I grew up in is… not exactly worthless; it does help with intrinsically worthwhile lives, and, more to the point, with the details of how to actually build the kinds of reasonable human relationships that you need for parts of the “shut up and multiply”-motivated efforts to work… but, for most people, it’s basically not very connected to how much good they do or don’t do in the world. If you like, this is good news: for a ridiculously small sum of effort (e.g., a $500 donation to SIAI; the earning power of seven ten-thousandths of your life if you earn the US minimum wage), you can do more expected-good than perhaps 99.9% of Earth’s population. (You may be able to do still more expected-good by taking that time and thinking carefully about what most impacts global goodness and whether anyone’s doing it.)

AnnaSalamon 6 Feb 2009 14:58 UTC
19 points
on: Epilogue: Atonement (8/8)
“though probably most of my effort was on relatively useless rather ordinary do-gooder projects”

Aleksei, “ordinary do-gooder projects” are relatively useless. That is, they are multiple orders of magnitude less efficient at global expected-goodness production than well thought out efforts to reduce existential risks. If you somehow ignore existential risks, “ordinary do-gooder projects” are even orders of magnitude less efficient than the better charities working on current human welfare, as analyzed by Givewell or by the Copenhagen Consensus.

Enjoy your life, and don’t feel guilty if you don’t feel guilty; but if you do want to increase the odds that anything of value survives in this corner of the universe, don’t focus on managing to give up more of your current pleasure. Focus on how efficiently you use whatever time/money/influence you are putting toward global goodness. Someone who spends seven ten-thousandths of their time earning money to donate to SIAI does ridiculously more good than someone who spends 90% of their time being a Good Person at an ordinary charity. (They have more time left-over to enjoy themselves, too.)

AnnaSalamon 6 Feb 2009 14:59 UTC
4 points
on: Epilogue: Atonement (8/8)
Never mind, I’m an idiot. I somehow read “relatively useless rather than ordinary”, even though Aleksei wrote “relatively useless rather ordinary”.

AnnaSalamon 12 Feb 2009 2:16 UTC
1 point
on: Cynicism in Ev-Psych (and Econ?)
Robin, I’m still unclear on what you are saying. Do you mean Smullyan’s use of informal language doesn’t actually make sense as an unbiased best guess as to how to get people to understand Smullyan’s logic puzzles (whatever his conscious motives)?