homunq

Karma: 480

homunq 6 Jul 2008 23:21 UTC
0 points
on: Living in Many Worlds
I think that there are deep philosophical implications for many-world theories, including but not limited to quantum many-world theories. If there are many worlds, presumably a large number of them must differ in their most obvious meta-characteristics. Some of these meta-characteristics that I observe are consequence, complexity, and difficulty (that is, across a wide array of phenomena, harmony is possible but not easy. There is no argument that will convince everyone, there is no FTL, there is a great filter...). Thus I can safely presume that inhabitants of the worlds which do not share these meta-characteristics are in some separate anthropic set. Thus, for beings in my anthropic set, I can take these characteristics as moral axioms. I do not argue that they are a source for all moral reasons, except through the difficult mediation of evolution; however, they are a moral bedrock. In other words, they underdetermine my morals, but they do determine them.

Did you ever read “A Fire Upon The Deep”? Obviously, it’s shameless space opera. But it’s a good metaphor for what I think our real situation is. The premise is that there is some kind of “IQ limit” that goes from 0 at the center of the galaxy to infinite outside it. The outside is the domain of the strong AIs, ineffable to human reason, and we are in the grey zone, where intelligence is possible but AI is not. I think that a situation something like this pertains, not over real space, but over the parameter space of multiple worlds. We ARE on the border of God’s Mandelbrot Set, and that there is something special about that. If we ever make it to AGI, for me, that is not a win condition (or a lose condition) but just a boundary condition: I cannot begin to evaluate the conditions of my actions here and now on the world beyond that boundary, so it is beyond my morals. The specialness of our position, and the fact that a world where we attain AGI is not in the same way special, is for me a consequence of the anthropic principle plus many worlds (as I said, quantum and otherwise.)

So many worlds for me is an argument that we should not be in any all-fired hurry to reach AGI, that moral actions within the context of the world-as-we-know-it are more important.

homunq 12 Nov 2008 17:35 UTC
0 points
on: Complexity and Intelligence
To me, this is all as silly as the Chinese Room. Naive intuition says that the Chinese Room doesn’t understand chinese in the way we do… because it doesn’t, running as it does on exponentially slower timescales. Similarly, I can close the box on my turing machine and it can, over mega-exponential timescales and using mega-exponential amounts of tape, get just as much smarter as it could by looking around and talking to people, but there are still several reasons why the latter strategy is superior.

Of course, I may just not understand what is supposed to be being demonstrated (or refuted) here. But if you’re arguing that Godel doesn’t outlaw singularities, then this is a massive non-sequitur. What you’re saying is that Godel doesn’t outlaw singularities which are r-e-a-l-l-y s-l-o-w, which seems to me to violate any useful definition of singularities. (OK, sure, it shows the wall is breached, but don’t expect it to convince any skeptics.)

homunq 12 Nov 2008 21:12 UTC
3 points
on: The Magnitude of His Own Folly
Let’s see if I can get perm-ignore on, on such an old post.

This whole line of thinking (press “on”, six million bodies fall) depends on a self-modifying AI being qualitatively different from a non-self-modifying one OR on self-modifying characteristics being the dominant strategy for achieving AI. In other words, there is a magic intelligence algorithm, which if implemented will lead to exponentially increasing intelligence, then you have to worry about the relative probability of that intelligence being in the Navel Gazing, Paperclips, and Friendly categories (and of course defining the categories) before you hit the switch on any candidate algorithm.

I think that intelligence is a very hard goal to hit, and that there is no self-contained, fast, non-iterative algorithm that rockets there. It is going to be much easier to build successive AIs with IQs of 10, 20, 30… (or 10, 20, 40...; the point remains) than to build a single AI which rockets itself off the scale. And in the process, we will need a lot of research in keeping the things stable, just to get up to 100, let alone to make it to 5000. We will also learn “what kind of animal” the unstable ones are, and what kind of mistake they tend to make. Progress will be slow—there is a long way from IQ100, when it starts to help us out, to IQ500, when it starts to be incomprehensible to humanity. In other words: it is not actually a whole lot easier to do it unfriendly-style than friendly-style.

That does not, of course, mean we should stop worrying about unfriendly risks. But it does mean that it is silly hubris to imagine that yelling “slow down until we figure this out” actually helps our chances of getting it right. We will stub our toe many times before we get the chance to blow our brains out, and anyone who is afraid to take a step because there might be a bullet out there underestimates the complexity and difficulty of the task we face.

homunq 13 Nov 2008 1:47 UTC
1 point
on: The Weighted Majority Algorithm
Sure, randomness is the lazy way out. But if computational resources are not infinite, laziness is sometimes the smart way out. And I said “infinite”—even transhuman quantum computers are finite.

homunq 14 Nov 2008 17:39 UTC
1 point
on: Selling Nonapples
vendor a: selling apples on wood carts isn’t making as much money as I hoped.

vendor b: maybe we should sell nonapples on nonwood carts.

a: that’s just silly. Which convinces me that we should continue selling non-nonapples on non-nonwood.

… ie, the opposite of stupidity is rarely intelligence, but the opposite of the opposite of stupidity never is.

Human intelligence arose out of a neurological Turing tarpit. It is reasonable to imagine that designing intelligence in less time will take tricks—ones which Mother Nature didn’t use—to get out of the tarpit. It is not reasonable to imagine that there is one magic trick which gets us all the way there. So saying “my trick is better than your trick” is interesting, but don’t mean that the final answer won’t need both or neither.

In fact, as soon as it comes to actual wagon design, “nonwood” can be a useful step. If you are building wagons, and there are 5 wagon stores in town, one of which is the “nonwood” store and has the generally lowest-quality—but still functioning—wagons, and you only have time to visit 2 stores for inspiration, which would you visit? Would you be happy to have the nonwood designs all in one store, or would you rather use some classification system (number of spokes) which split them across several stores? (That is, until metal is the new wood.) Any actual “nonwood” wagon is in fact made of something.

homunq 21 Feb 2009 1:08 UTC
1 point
on: Torture vs. Dust Specks
The question is, of course, silly. It is perfectly rational to decline to answer. I choose to try to answer.

It is also perfectly rational to say “it depends”. If you really think “a dust speck in 3^^^3 eyes” gives a uniquely defined probability distribution over different subsets of possibilityverse, you are being ridiculous. But let’s pretend it did—let’s pretend we had 3^^^^3 parallel Eleizers, standing on flat golden surfaces in 1G and one atmosphere, for just long enough to ask each other enough enough questions to define the problem properly. (I’m sorry, Eleizer, if by stating that possibility, I’ve increased the “true”ness of that part of the probabilityverse by ((3^^^3+1)/3^^^3) :) ).

You can or “I’ve thought about it, but I don’t trust my thought processes”. That is not my position.

My position is that this question does not, in fact, have an answer. I think that that fact is very important.

It’s not that the numbers are meaningless. 3^^^3 is a very exact number, and you can prove any number of things about it. A different question using ridiculous numbers—say, would you rather torture 4^^^4 people for 5 minutes or 3^^^3 of them for 50 years—has a single correct answer which is very clear (of course, the 3^^^3 ones; 4^^^4 >>> (3^^^3)^2). (Unless there were very bizarre extra conditions on the problem.)

It’s just that there is no universal moral utility function which inputs a probability distribution over a finite subset of the possibilityverse and outputs a number. It’s more like relativistic causality—substitute “better” for “after”. A is after B and B is a spacelike distance from C, but C can also be spacelike from A. The dust specks and the torture are incomparable, a spacelike distance.

I think that, philosophically, that makes a big difference. If you pilosophically can’t always go around morally comparing near-infinite sets, then it’s silly to try to approximate how you would behave if you could. Which means you consider the moral value of the consequences which you could possibly anticipate. So yeah, if you are working on AI, you are morally obligated to think about FAI, because that’s intentional action, and you would have to be a monster to say you didn’t care. But you don’t get to use FAI and the singularity to trump the here-and-now, because in many ways they’re just not comparable.

Which means, to me, for instance, that people can understand the singularity idea and believe it has a non-0 probability, and have abilities or resources that would be meaningful to the FAI effort, and still morally choose to simply live as “good people” in a more traditional sense (have a good life in which they make the people with whom they interact overall happier). It’s not just a lack of ability to trace the consequences; it’s also the possibility that the consequences of this or that outcome will be literally incomparable by any finite halting algorithm, whereas even our desperately-limited brains have decent approximations of algorithms for morally comparing the effect of, say, posting on OB versus washing the dishes.

Going to wash the dishes now.

homunq 21 Feb 2009 1:37 UTC
0 points
on: Torture vs. Dust Specks
Tim: You’re right—if you are a reasonably attractive and charismatic person. Otherwise, the question (from both sides) is worse than the dust speck.

(Asking people also puts you in the picture. You must like to spend eternity asking people a silly question, and learning all possible linguistic vocalizations in order to do so. There are many fewer vocalizations than possible languages, and many fewer possible human languages than 3^^^3. You will be spending more time going from one person of the SAME language to another, at 1 femtosecond per journey, than you would spend learning all possible human languages. That would be true even if the people were fully shuffled by language—just 1 femtosecond each for all the times when coincidence gives you two of the same language in a row. 3^^^3 is that big.)

homunq 8 May 2009 21:59 UTC
4 points
in reply to: Paul Crowley’s comment on: Why Our Kind Can’t Cooperate
I saw a study recently that said that the mere presence of a salad on the menu increases people’s consumption. I deeply doubt that fast food chains were surprised by that result.

From the nature of the study, it’s not even about convincing themselves they intend to buy a salad. By merely seriously having considered the option, they give themselves virtue points which offset the vice of more consumption.

homunq 14 May 2009 15:58 UTC
3 points
in reply to: taw’s comment on: The mind-killer
I myself would be disappointed if over half of LW put the probability of a single biological human (not an upload, not a reconstruction—an actual descendent with the appropriate number of ancestors alive today) alive in 100 years under 95%. I would consider that to be a gross instance of all kinds of biases. I’m not going to argue about scenarios, here, just point out that there any scenario which tends inevitably to wipe out humanity within one lifetime is totally unimaginable. That doesn’t mean implausible, but it does mean improbable.

Personally, I do not believe that any person, group of people, or human-built model to date can consistently predict the probability of defined classes of black-swan events (“something that’s never happened before which causes X” where X is a defined consequence such as humanity’s extinction) to within even an order of magnitude for p/(1-p). I doubt anybody can get even to within two orders of magnitude consistently. (I also doubt that this hypothesis of mine will be clearly decidable within the next 20 years, so I’m not particularly inclined to listen to philosophical arguments from people who’d like to discard it.)

What I’m saying is, we should stop trying to put numbers on this without big error bars. And I’ve yet to see anybody propose an intelligent way to deal with probabilities like 10^(-6 +/- 4); just meta-averaging it over the distribution of possible probabilities, to come up with something like 10^-3 seems to be discarding data and to lead to problems. However, that’s the kind of probability I’d put on this lemma. (“Earth made uninhabitable by normal cosmic event and rescue plans fail” would probably put a floor somewhere above 10^-22 per year.)

“The chance we’re all wrong about something totally unprecedented has got to be less than 99.9%” is total hubris. Yes, totally unprecedented things happen every day. But telling yourselves stories about AGI and foom does not make these stories likely.

This is not, by the way, an argument to ignore existential risk. Even at the 10^-6 (or, averaged over meta-probabilities, 10^-3) level which I estimated, it is clearly worth thinking about, given the consequences. But if you’re all getting that carried away, then Less Wrong should just be renamed More Wrong.

homunq 14 May 2009 16:11 UTC
0 points
in reply to: homunq’s comment on: The mind-killer
Oh, also, I’d accept that the risk of humanity being seriously hosed within 100 years, or extinct within 1000 years, is significant—say, 10^(-3 +/- 4) which meta-averages to something like 15%.

(“Seriously hosed” means gigadeath events, total enslavement, or the like. Note that we’re already moderately hosed and always have been, but that seriously hosed is still distinguishable.)

homunq 14 May 2009 16:32 UTC
0 points
in reply to: Nick_Tarleton’s comment on: The mind-killer
I think that the original poster was discounting low-probability non-anthropogenic risks (sun goes nova, war of the worlds) and counting as “unknown unknowns” any risk which is unimaginable (that is, involves significant new developments which would tend to limit the capacity of human (metaphorical) reasoning to assess the specific probability or consequences at this time; this includes all fooms, gray goos, etc.)

I would agree with the poster that a general attitude of readiness (that is, education, democracy, limits on overall social inequality, and precautionary attitudes to new technologies) is probably orders of magnitude more effective at dealing with such threats than any specific measures until a specific threat becomes clearer.

And I dispute the characterization that, if I’m correct about the poster’s attitudes, they’re “carefully hiding conclusions [they] disagree with”; a refusal to consider vague handwaving categories of possibility like gray goo in the same class as much-more-specific possibilities like nuclear holocaust may not be your attitude, but that does not make it dishonest.

homunq 14 May 2009 16:47 UTC
0 points
in reply to: jimmy’s comment on: The mind-killer
Large groups of smart people are frequently wrong about the future, and overwhelmingly so about the non-immediate future. 0.1% may be low but it’s not ridiculously so.

(Also “they’re right and you’re wrong” is redundant. This has nothing to do with any set of scenario probabilities being “right”. And any debate of “p=.9” “no, p=.1″ is essentially silly because it misunderstands both the meaning of probability as a function of knowledge and our ability to create models which give meaningfully-accurate probabilities.)

homunq 14 May 2009 17:19 UTC
2 points
in reply to: timtyler’s comment on: Survey Results
No contradiction between the two posts above. The second is, nonetheless, probably more useful in judging accuracy.

homunq 14 May 2009 17:48 UTC
1 point
in reply to: MichaelVassar’s comment on: Survey Results
Or believe that a creator god created the universe but does not exist. Depending on your definition of “exist” this could be meaningful.

homunq 14 May 2009 17:58 UTC
1 point
in reply to: Vladimir_Nesov’s comment on: The mind-killer
As you explain it, it’s not silly.

Do you have a link for a top-level post that puts this kind of caveat on probability assignments? Personally, I think that if most people here understood it that way, they’d use more qualified language when talking about subjective probability. I also think that developing and standardizing such qualified language would be a useful project.

homunq 14 May 2009 18:17 UTC
2 points
in reply to: Vladimir_Nesov’s comment on: The mind-killer
You clipped out “to within an order of magnitude”. I stated that my best-guess probability for human extinction within a century was 10^(-6 +/- 4). This is a huge confusion − 9 orders of magnitude on the probability—yet still means that I have over 80% confidence that the probability is under 10^-2. There is no contradiction here.

(It also means that, despite believing that extinction is probably one-in-a-million, I should treat it as more like one-in-a-thousand, because averaging over the meta-probability distribution naturally weights the high end. It would be a pity if this effect, of uncertainty inflating small probabilities, resulted in social feedback. When you hear me say “we should treat it as a .1% risk”, I am implicitly stating that all models I can credit give a significantly lower risk. If your best model’s risk-estimate is .01%, I am actually telling you that I think your model overestimates the risk.)

homunq 14 May 2009 19:21 UTC
2 points
in reply to: steven0461’s comment on: The mind-killer
“the probability” means something like the following: take a random selection of universe-histories starting with a state consistent with my/your observable past and proceeding 100 years forward, with no uncaused discontinuities in the laws of physics, to a compact portion of a wave function (that is “one quantum universe”, modulo quantum computers which are turned on). What portion of those universes satisfy the given end state?

Yes, I’m doing what I can to duck the measure problem of universes, sorry. And of course this is underdefined and unobservable. Yet it contains the basic elements: both knowledge and uncertainty about the current state of the universe, and definite laws of physics, assumed to independently exist, which strongly constrain the possible outcomes from a given initial state.

On a more practical level, it seems to be the case that, given enough information and study of a class of situations, post-hoc polynomial-computable models which use non-determinism to model the effects of details which have been abstracted out, can provide predictions about some salient aspects of that situation under certain constraints. For instance, the statement “42% of technological societies of intelligent biological agents with access to fissile materiels destroy themselves in a nuclear holocaust” could, subject to the definitions of terms that would be necessary to build a useful model, be a true or false statement.

Note that this allows for three completely different kinds of uncertainty: uncertainty about the appropriate model(s), uncertainty about the correct parameters for those model(s), and uncertainty inherent within a given model. In almost all questions involving predicting nonlinear interactions of intelligent agents, the first kind of uncertainty currently dominates. That is the kind of uncertainty I’m trying (and of course failing) to capture with the error bar in the exponent. Still, I think my failure, which at least acknowledges the overwhelming probability that I’m wrong (albeit in a limited sense) is better than a form of estimation that presents an estimate garnered from a clearly limited set of models as a final one.

In other words: I’m probably wrong. You’re probably wrong too. Since giving an estimate under 95% requires certain specific extrapolations, while almost any induction points to estimates over 95%, I would expect most rational people to arrive at an estimate over 95%, and would suspect any community with the reverse situation to be subject to biases (of which selection bias is the most innocuous). This suspicion would not apply when dealing with individuals.

homunq 14 May 2009 20:15 UTC
0 points
in reply to: steven0461’s comment on: The mind-killer
To get the right answer, you need to make a honest effort to construct a model which is an unbiased composite of evidence-based models. Metaphorical reasoning is permitted as weak evidence, but cannot be the only sort of evidence.

And you also need to be lucky. I mean, unless you have the resources to fully simulate universes, you can never know that you have the right answer. But the process above, iterated, will tend to improve your answer.

homunq 4 Aug 2009 17:22 UTC
−2 points
on: Beyond the Reach of God
I tend to resolve these issues with measure-problem hand-waving. Basically, since any possible universe exists (between quantum branching, inflationary multiverse, and simulated/purely mathematical existence), any collection of particles (such as me sitting here) exists with a practically uncountable set of futures and pasts, many of which make no sense (bolzman brains). The measure problem is, why is that “many” not actually “most”? The simplest answer is the anthropic one: because that kind of existence simply “doesn’t count”. So, there is some set of qualities of the universe as we know it that make it “count”, let’s call that set “consciousness”. And, personally, I think that this set includes not only the existence of optimizing agents (ourselves), but also the fact that these agents are fundamentally limited in something similar to the ways that we are. In other words, the very existence of some FAI which can keep all of your bad decisions (for any given definition of “bad”) from having consequences, means that “consciousness” as we know it has ended. Whatever exists on the other side of that barrier is simply incommensurable with my values here on this side. It’s “game over”. I can have perfect faith that my “me” will never see it completed—by definition, since then I’d no longer be a conscious “me” under my definition.

That means I am much more motivated to look for (weakly) “incremental” solutions to the problems I see with the world than for truly revolutionary ones like FAI or cryonics. (I regard the last “revolutionary” change to be the evolution of humanity itself—so “incremental” in this sense encompasses all human revolutions to date. The end of death would not be incremental in this sense.)

Sure, I can see where this is more of a justification for acting like a normal person than a rational exploration of fully coherent value space. Yet I can also argue that being meta-rational means justifying, not re-questioning, certain axioms of behavior.

Shorter me: “solving the whole world” leaves me cold, despite fun theory and all. So does ending death, or avoiding it personally. So me not signing up for cryo is perfectly rational.

While I acknowledge that this might not be the most complete and coherent possible set of values, I see no evidence that it’s specifically incoherent, and it is complete enough for me. The Singularity Institute set of values may be more complete and just as non-incoherent, but I suspect that mine are operationally superior, or at least less likely to be falsified, since they attain similar coherence with less of a divergence from evolved human behaviour.

homunq 6 Aug 2009 21:27 UTC
3 points
in reply to: Vladimir_Nesov’s comment on: The mind-killer
I have confidence that nobody here has secret information that makes human extinction much more likely—because almost no information which currently exists could have more than a marginal bearing on a result which, if likely, is a result of human (that is, intelligent) interaction. Therefore I have confidence that the difference in estimates is largely not due to information, but to models. I have confidence that inductive models—say, “how often does a random species survive any hundred year period, correcting for initial population” give answers over 95% which should be considered the default. Therefore, I have confidence that a community of people who generally give lower estimates is subject to some biases (such as narrative bias).

Doesn’t mean LW’s wrong and I’m right. But to believe that human extinction within a century is likely clearly puts LW in the minority of humanity in your beliefs—even in the minority of rational atheists. And the fact that there is substantial agreement within the LW community on this, when uncertainty is clearly so high that orders of magnitude of disagreement are possible, makes me suspect bias.

Also, I find it funny that people will argue passionately over estimates that differ in log(p/q) from −1 to +1 (~10% to ~90%), but couldn’t care less over the difference from say −9 to −7 (.0001% vs .000001%) or 7 to 9. This is in one sense the right attitude for people who think they can do something about it, but it ends up biasing numbers towards log(p/q)=0 [ie 50%], since you are more likely to get argument from somebody who has an estimate on the other side of 50% as yours is.