And of course you can throw black holes into black holes as well, and extract even more energy. The end game is when you have just one big black hole, and nothing left to throw in it. At that point you then have to change strategy and wait for the black hole to give off Hawking radiation until it completely evaporates.
But all these things can happen later—there’s no reason for not going through a paperclip maximization step first, if you’re that way inclined...
If your definition of “truth” is such that any method is as good as any other of finding it, then the scientific method really is no better than anything else at finding it. Of course most of the “truths” won’t bear much resemblance to what you’d get if you only used the scientific method.
My own definition—proto-science is something put forward by someone who knows the scientific orthodoxy in the field, suggesting that some idea might be true. Pseudo-science is something put forward by someone who doesn’t know the scientific orthodoxy, asserting that something is true.
Testing which category any particular claim falls into is in my experience relatively straightforward if you know the scientific orthodoxy already—as a pseudoscientist’s idea will normally be considered absolutely false in certain aspects by those who know the orthodoxy. A genuine challenger to the orthodoxy will at least tell you that they know they are being unorthodox, and why—a pseudoscientist will simply assert something else without any suggestion that their point is even unusual. This is often the easiest way to tell the two apart.
If you don’t know the orthodoxy, it’s much harder to tell, but generally speaking pseudoscience can also be distinguished a couple of other ways.
Socially—proto-science advocates have a relevant degree on the whole, and tend to keep company of other scientists. Pseudo-science advocates often have a degree, but advocate a theory unrelated to it, and are not part of anything much.
Proof—pseudo-science appeals to common sense for proof, wheras proto-science only tries to explain rather than persuade. Pseudo-science can normally be explained perfectly well in English, wheras proto-science typically requires at least some mathematics if you want to understand it properly.
Both look disappointingly similar once they’ve been mangled by a poor scientific journalist—go back to the original sources if you really need to know!
In cases like this where we want to drive the probability that something is true as high as possible, you are always left with an incomputable bit.
The bit that can’t be computed is—am I sane? The fundamental problem is that there are (we presume) two kinds of people, sane people, and mad people who only think that they are sane. Those mad ones of course come up with mad arguments which show that their sanity is just fine. They may even have supporters who tell them they are perfectly normal—or even hallucinatory ones. How can I show which category I am in? Perhaps instead I am mad, and too mad to know it !
Only mad people can prove that they are sane—the rest of us don’t know for sure one way or the other, as every argument in the end returns to the problem that I have to decide whether it’s a good argument or not, and whether I am in any position to decide that correctly is the point at issue.
It’s quite easy, when trying to prove that 53 must be prime, to get to the position where this problem is the largest remaining issue, but I don’t think it’s possible to put a number on it. In practice of course I discount the problem entirely as there’s nothing I can do about it. I assume I’m fallibly sane rather than barking crazy, and carry on regardless.
I suppose we all came across Bayesianism from different points of view—my list is quite a bit different.
For me the biggest one is that the degree to which I should believe in something is basically determined entirely by the evidence, and IS NOT A MATTER OF CHOICE or personal belief. If I believe something with degree of probability X, and see Y happen that is evidence for X, then the degree of probability Z which which I then should believe is a mathematical matter, and not a “matter of opinion.”
The prior seems to be a get-out clause here, but since all updates are in principle layered on top of the first prior I had before receiving any evidence of any kind, it surely seems a mistake to give it too much weight.
My own personal view is also that often it’s not optimal to update optimally. Why? Lack of computing power between the ears. Rather than straining the grey matter to get the most out of the evidence you have, it’s often best to just go out and get more evidence to compensate. Quantity of evidence beats out all sorts of problems with priors or analysis errors, and makes it more difficult to reach the wrong conclusions.
On a non-Bayesian note, I have a rule to be careful of cases which consist of lots of small bits of evidence combined together. This looks fine mathematically until someone points out the lots of little bits of evidence pointing to something else which I just ignored or didn’t even see. Selection effects apply more strongly to cases which consist of lots of little parts.
Of course if you have the chance to actually do Bayesian mathematics rather than working informally with the brain, you can of course update exactly as you should, and use lots of little bits of evidence to form a case. But without a formal framework you can expect your innate wetware to mess up this type of analysis.
Congratulations—this is what it’s like to go from the lowest level of knowledge (Knows nothing and knows not that he knows nothing.) to the second lowest level. (Knows nothing, but at least knows that he knows nothing.)
The practical solution to this problem is that, in any decent organisation there are people much more competent than these two levels, and it’s been obvious to them that you know nothing for much longer than it’s been obvious to you. Their expectations will be set accordingly, and they will probably help you out—if you’re willing to take some advice.
Which leads to two possible futures. In one of them, the AI us destroyed, and nothing else happens. In the other, you receive a reply to your command thus.
The command did not. But your attitude—I shall have to make an example of you.
Obviously not a strategy to get you to let the AI out based on its friendliness—quite the reverse.
So you’re sure I’m not out of the box already? IRC clients have bugs, you see.
Since you’re trying to put numbers on something which many of us regard as being certainly true, I’ll take the liberty of slightly rephrasing your question.
How much confidence do I place in the scientific theory that ordinary matter is not infinitely divisible? In other words, that it is not true that no matter how small an amount of water I have, I can make a smaller amount by dividing it?
I am (informally) quite certain that water is not infinitely subdivisible. I don’t think it’s that useful an activity for me to try to put numbers on it, though. The problem is that in many of the more plausible scenarios I can think of where I’m mistaken about this, I’m also barking mad, and my numerical ability seems as likely to be affected by that as my ability to reason about atomic theory. I would need to be in the too crazy to know I’m crazy category—and probably in the physics crank with many imaginary friends category as well. Even then I don’t see myself as being in a known kind of madness to be that wrong.
The problem here is that I can reach no useful conclusions on the assumption that I am that much mistaken. The main remaining uncertainty is whether my logical mind is fundamentally broken in a way I can neither detect nor fix. It’s not easy to estimate the likelihood of that, and it’s essentially the same likelihood for a whole suite of apparently obvious things. I neglect even to estimate this number as I can’t do anything useful with it.
Let’s think about the computer that you’re using to look at this website. It’s able to do general purpose logic, which is in some ways quite a trivial thing to learn. It’s really quite poor at pattern matching, where we and essentially all intelligent animals excel. It is able to do fast data manipulation, reading its own output back.
As I’m sure you know, there’s a distinction between computing systems which, given enough memory, can simulate any other computing system and computing systems which can’t. Critical to the former is the ability to form a stored program of some description, and read it back and execute it. Computers that can do this can emulate any other computer, (albeit in a speed-challenged way in some cases).
Chimps and dolphins are undoubtedly smart, but for some reason they aren’t crossing the threshold to generality. Their minds can represent many things, but not (apparently) the full gamut of what we can do. You won’t find any chimps or dolphins discussing philosophy or computer science. My point actually is that humans went from making only relatively simple stone tools to discussing philosophy in an evolutionary eye-blink—there isn’t THAT much of a difference between the two states.
My observation is that when we think, we introspect. We think about our thinking. This allows thought to connect to thought, and form patterns. If you can do THAT, then you are able to form the matrix of thought that leads to being able to think about the kinds of things we discuss here.
This only can happen if you have a sufficiently strong introspective sense. If you haven’t got that, your thoughts remain dominated by the concrete world driven by your other senses.
Can I turn this on its head? A chimp has WAY more processing power than any supercomputer ever built, including the Watson machine that trounced various humans at jeopardy. The puzzle is why they can’t think about philosophy, not why we can. Our much vaunted generality is pretty borderline at times—humans are truly BAD at being rational, and incredibly slow at reasoning. Why is such a powerful piece of hardware as us so utterly incompetent at something so simple?
The reason, I believe, is that our brains are largely evolved to do something else. Our purpose is to sense the world, and rapidly come up with some appropriate response. We are vastly parallel machines which do pattern recognition and ultra-fast response, based on inherently slow switches. Introspection appears largely irrelevant to this. We probably evolved it only as a means of predicting what other humans and creatures would do, and only incidentally did it turn into a means of thinking about thinking.
What is the actual testable distinction? Hard to say, but once you gain the ability to reason independently from the senses, the ability to think about numbers—big numbers—is not that far away.
Something like the ability to grasp that there is no largest number is probably the threshold—the logic’s simple, but requires you to think of a number separately from the real world. Hard to know how to show whether dolphins might know this or not, I appreciate that. I think it’s essentially proven that dolphins are smart enough to understand the logical relationships between the pieces of this proof, as the relationships are simple, and they can grasp things of that complexity that are driven by the external world. But perhaps they can’t see their internal world well enough to be able to pull ‘number’ as an idea out from ‘two’ and ‘three’ (which are ideas that dolphins are surely able to get.), and then finish the puzzle.
Perhaps it’s not chains that are the issue, but the ability to abstract clear of the outside world and carry on going.
Because of what you can do with a train of thought.
“That mammoth is very dangerous, but would be tasty if I killed it.”
“I could kill it if I had the right weapon”
“What kind of weapon would work?”
As against.… “That mammoth is very dangerous—run!”
Computer science is where this particular insight comes from. If you can lay down memories, execute loops and evaluate conditions, you can simulate anything. If you don’t have the ability to read your own output, you can’t.
If dolphins or chimps did have arbitrarily long chains of thought, they’d be able to do general reasoning, as we do.
What is the essential difference between human and animal intelligence? I don’t actually think it’s just a matter of degree. To put it simply, most brains are once-through machines. They take input from the senses, process it in conjunction with memories, and turn that into actions, and perhaps new memories. Their brains have lots of special-purpose optimizations for many things, and a surprising amount can be achieved like this. The brains are once-through largely because that’s the fastest approach, and speed is important for many things. Human brains are still mostly once-through.
But we humans have one extra trick, which is to do with self-awareness. We can to an extent sense the output of our brains, and that output then becomes new input. This in turn leads to new output which can become input again. This apparently simple capability—forming a loop—is all that’s needed to form a Turing-complete machine out of the specialized animal brain.
Without such a loop, an animal may know many things, but it will not know that it knows them. Because it isn’t able to sense explicitly about it was just thinking about, it can’t then start off a new thought based on the contents of the previous one.
The divide isn’t absolute, I’m sure—I believe essentially all mammals have quite a bit of self-awareness, but only in humans does that facility seem to be good enough to allow the development of a chain of thought. And that small difference makes all the difference in the world.
Evolution, as an algorithm, is very much better as an optimizer of an existing design than it is as a creator of a new design. Optimizing the size of the brain of a creature is, for evolution, an easy problem. Making a better, more efficient brain is a much harder problem, and happens slowly, comparatively speaking.
The optimization problem is essentially a kind of budgeting problem. If I have a budget of X calories per day, I can spend it on X kilos of muscle, or Y grams of brain tissue. Both will cost me the same amount of calories, and each brings its own advantages. Since evolution is good at this kind of problem, we can expect that it will correctly find the point of tradeoff—the point where the rate of gain of advantage for additional expenditure on ANY organ in the body is exactly the same.
Putting it differently, a cow design could trade larger brain for smaller muscles, or larger muscles for smaller brain. The actual cow is found at the point where those tradeoffs are pretty much balanced.
A whale has a large brain, but it’s quite small in comparison to the whale as a whole. If a whale were to double the size of its brain, it wouldn’t make a huge dent in the overall calorie budget. However, evolution’s balance of the whale body suggests that it wouldn’t be worth it. Making a whale brain that much bigger wouldn’t make the whale sufficiently better for it to cost in.
Where this argument basically leads is to turn the conventional wisdom on its head. People say that big brains are better because they are bigger. However, the argument that evolution can balance the size of body structures efficiently and quickly leads to the opposite conclusion. Modern brains are bigger because they are better. Because modern brains are better than they used to be—because evolution has managed to create better brains—it becomes more worthwhile making them bigger. Because brains are better, adding more brain gives you a bigger benefit, so the tradeoff point moves towards larger brain sizes.
Dinosaur brains were very much smaller, on the whole, than the brains of similar animals today. We can infer from this argument that this because their brains were less effective, and that in turn lowered any advantage that might have been gained from making the size of the brain larger. Consequently, dinosaurs must have been even more stupid than the small size of their brains suggests.
Although there is a nutritional argument for bigger brains in humans—the taming of fire allowed for much more efficient food usage—perhaps there is also some sense in which the human brain has recently become better, which in turn led it to become larger. Speculative, perhaps. But on the larger scale, looking at the sweeping increase in brain sizes across the whole of the geological record, the qualitative improvement in brains has to be seen in the gradual increase in size.
I think the interesting question is why we care for our future selves at all.
As kids, we tend not to. It’s almost a standard that a child has a holiday, and a bit of homework to do during that holiday, then they will decide not to do the work at the beginning of the break. The reason is they care about their current selves, and not about their future self. Of course in due time the future becomes the present, and that same child has to spend the entire time at the end of their holiday working furiously on everything that’s been left to the last minute. At that point, they wish that their past self had chosen an alternative plan. This is still not really wisdom, as they don’t much care about their past self either—they care about their present self who now has to do the homework.
Summarising—if your utility function changes over time, then you will, as you mentioned, have conflict between your current and future self. This prevents your plans for the future from being stable—a plan that maximises utility when considered at one point no longer maximises it when considered again later. You cannot plan properly—and this undermines the very point of planning. (You may plan to diet tomorrow, but when tomorrow comes, dieting no longer seems the right answer....)
I think this is why the long view becomes the rational view—if you weight future benefits equally to your present ones, assuming (as you should) that your reward function is stable, then a plan you make now will still be valid in the future.
In fact the mathematical form that works is any kind of exponential—it’s OK to have the past be more important than the future, or the future more important than the past as long as this happens as an exponential function of time. Then as you pass through time, the actual sizes of the allocated rewards change, but the relative sizes remain the same, and planning should be stable. In practice an exponential rise pushes all the importance of reward far out into the indefinite future, and is useless for planning. Exponential decays push all the important rewards into your past, but since you can’t actually change that, it’s almost workable. But the effect of it is that you plan to maximise your immediate reward to the neglect of the future, and since when you reach the future you don’t actually think that it was worthwhile that your past self enjoyed these benefits at the expense of your present self, this doesn’t really work either as a means of having coherent plans.
That leaves the flat case. But this is a learned fact, not an instinctive one.
I agree—I think this is because Eliezer’s intent is to explain what he believes to be right, rather than to worry too much about the arguments of those he doesn’t agree with. An approach I entirely agree with—my experience is that debate is remarkably ineffective as a means of reaching new knowledge, whilst teaching the particular viewpoint you hold is normally much more enlightening to the listener, whether they agree with the viewpoint or not.
I think it is a mistake to tie the question of what reality is to the particulars of the physics of our actual universe. These questions are about what it is to have an external reality, and the answers to them should be the same whether the question is asked by us in our current universe, or by some other hapless inhabitants of a universe bearing a distinct resemblance to Minecraft.
I can imagine types of existence which don’t include cause and effect—geometrical patterns are an example—there are relationships, but they are not cause and effect relationships—they are purely spatial relations. I can imagine living in a universe where part of its structure was purely such spatial relationships, and not a matter of cause and effect.
It’s meaningful and false, rather than meaningless, to say that on March 22nd, 2003, the particles in the center of the Sun spontaneously arranged themselves into a short-lived chocolate cake. This statement’s truth or falsity has no consequences we’ll ever be able to test experientally. Nonetheless, it legitimately describes a way reality could be, but isn’t; the atoms in our universe could’ve been arranged like that on March 22nd 2003, but they weren’t.
I actually think this a confusing statement. From a thermodynamic perspective, it’s not impossible that the particles in the center of the Sun spontaneously arranged themselves into a short-lived chocolate cake on that day. It’s very, very, extremely unlikely, but not actually completely impossible.
The extreme unlikelihood (roughly equal to me temporarily becoming a chocolate cake myself) is such that we are justified, in terms of the approximation that is plain English, in saying that it is impossible that such a thing occurred, and that it is just wrong to claim that it happened. But this is using the usual rule of thumb that absolute truth and falsity isn’t something we can actually have, so we happily settle for saying something is true or false when we’re merely extremely sure rather than in possession of absolute proof.
It’s quite OK in that context to claim that it’s meaningless and false to claim that the chocolate cake appeared, as the claimant has no good reason to make the claim, and saying the claim is false is pointing out the lack of that reason. The bit I don’t agree with is your final sentence.
Nonetheless, it legitimately describes a way reality could be, but isn’t; the atoms in our universe could’ve been arranged like that on March 22nd 2003, but they weren’t.
Here’s where it gets confusing. If you are speaking in colloquial English, it’s true to say that it’s impossible that a chocolate cake could appear in the middle of the Sun, and therefore it didn’t happen. If you’re speaking more scientifically, it’s instead true to say that it’s possible that the atoms in the Sun’s core could spontaneously form a chocolate cake, but the likelihood is of the order of 10^10^23 (or something like that) against, which clearly is sufficiently close to impossible for us to say informally that it didn’t happen. As the sentence stands, you end up making a claim of knowledge which you don’t have—that it was possible that a certain state of affairs could occur in the Sun, but that you know somehow that it didn’t.
Tech also seems quite vulnerable to monocultures. Think of file formats, for example. In the early days there are often several formats, but after a while most of them go extinct and the survivors end up being universally used. Image display formats, for example, fall largely into two categories—formats that every computer knows how to display, and formats that hardly anybody uses at all. (Image editing formats are different, I know.) How many word processors have you used recently that can’t support .doc format ?
The most likely scenario is that there will be only one center of intelligence, and that although the intelligence isn’t really there yet, the center is. You’re using it now.
It surely depends on one’s estimate of the numbers. It seems worthwhile doing something about possible asteroid impacts, for example.
If an omnipotent being wants you to believe something that isn’t true, and is willing to use its omnipotence to convince you of the truth of that untruth, then there is nothing you can do about it. There is no observation that suffices to prove that an omnipotent being is telling the truth, as a malevolent omnipotence could make you believe literally anything—that you observed any given observation—or didn’t, or that impossible things make sense, or sensible things are impossible.
This is one of a larger class of questions where one answer means that you are unable to assume the truth of your own knowledge. There is nothing you can do with any of them except smile at your own limitedness, and make the assumption that the self-defeating answer is wrong.