The dashed/inaccurate is more valuable than solid, but it’s not really the connection that’s the locus of inaccuracy.
If instead the self-model contained a diagram inside of it, then we could see that the connection self-model to planning is working fine, it’s the diagram inside self-model that is wrong.
I’m trying to imagine a scenario.
Possibly the decider knows that people sometimes make multiplicative errors, transposing numbers or misplacing decimals, and is confronted with a set of estimates hovering around, say, 0.05 (and that is plausible according to the decider’s prior) and a few estimates at estimated around 0.5 and 5.0. Would the correction effectively trim the outliers back to almost exactly 0.05 (because we can’t learn much information from an estimate that probably had at least one mistake in it), and the decider should go with the highest of the “plausible” numbers?
It seems to me like the conditional distributions that would lead to actually changing your decision are nearly as likely to be a source of error as a correction.
If you are willing to allow the optimal-against-this-pool strategy to run the entire tournament as a subroutine, I think it can guarantee a tie by cloning the eventual winner.
It may be able to do better than that, but it depends on the details of the pool whether there are points “lying on the ground” to be picked up; I suspect there are with this pool, but I don’t have a great example.
There are various distinguishing points between geocentrism and heliocentrism—one is that heliocentrism had fewer free parameters. More elaborately: Certain parameters of the epicycles had to be delicately balanced, and heliocentrism “predicted” that those parameters would be turn out to be balanced. This sense of prediction has nothing to do with previous-in-time, and everything to do with opportunities for falsification; if ever those parameters drifted from perfect balance, heliocentrism would be falsified. Heliocentrism was preferable (even though it was slightly less accurate at first) because it exposed itself to a more stringent test.
Overall, what I’m saying is that those two theories are asymmetric.
Consequentialism and deontologism can be encoded in terms of one another (please allow me to temporarily mischaracterize the discussion as if there were only two options consequentialism and deontologism). Both theories have “free parameters”; consequentialism has preferences over states, and deontologism has precepts (should-rules). By carefully setting the free parameters, you can turn one into the other. The deontologist can say “You should make decisions by considering their consequences according to this utility function”, and the consequentialist may say “I prefer world-lines where I obey these precepts”.
Given this background, advocacy of consequentialism over deontolgism is essentially saying that the correct entire theory (that is, including the free parameters as well) is best expressed using the language of preferences and states-of-the-world.
However, there are things that I value, including autonomy, freedom, self-determination, that are difficult to express as states-of-the-world. These can be expressed as preferences over world-lines or paths through configuration space, of course. However, this is an awkward encoding.
My argument against foundational consequentialism (as opposed to practical, day-to-day consequentialism) is that most of the things that I ultimately value are actually paths and properties of paths, and not states or properties of states at all. If the only reason to pick one over the other is the apt capture of the entire theory, then I prefer to discuss ethics in terms of “shoulds” and relegate consequentialism to a practical, day-to-day role.
Possibly we can set foundations aside (or agree as a group to be foundationally-pluralistic), and focus on very-slightly-more-practical questions?
One of the obvious extensions of this thought experiment is to posit a laser-powered blue goo that absorbs laser energy, and uses it to grow larger.
This thought experiment also reminds me: Omohundro’s arguments regarding likely uFAI behavior are based on the AI having goals of some sort—that is, something we would recognize as goals. It’s entirely possible that we wouldn’t perceive it as having goals at all, merely behavior.
As I understand it, the scenario is that you’re hearing a complicated argument, and you don’t fully grok or internalize it. As advised by “Making Your Explicit Reasoning Trustworthy”, you have decided not to believe it fully.
The problem comes in the second argument—should you take the advice of the person (or meme) that you at least somewhat mistrust in “correcting” for your mistrust? As you point out, if the person (or the meme) is self-serving, then the original proposal and the correction procedure will fit together neatly to cause a successful mugging.
I think concerns like this suggest that we ought to be using some sort of robust statistics—that is, the direction of the argument and the fact that the conclusion is extreme should influence our conclusions, but the magnitude cannot be allowed to influence our conclusions.
The link to Anna’s post in the footnotes is broken. Should be here.
You focus on visibly HAL-like or Skynet-like AI—the sort of thing that AI researchers produce as demos. However, we have large, smart, durable, existing entities (businesses and other computer+human teams) that are continuously getting smarter (and entrenching themselves deeper into our society) by automating their existing business practices.
I don’t advocate trying to stop business automation, or humans organizing themselves into better and better teams; I think that would be throwing the baby out with the bathwater. However, I do think “business as usual” or “the default future” is the threat that existential risks people should be imagining.
The vast majority of writing about these issues has a story of terrorists or scientists (who are wizards meddling with things man was not meant to know) accidentally creating paperclip-making machines. That isn’t thinking, that’s straight out of folklore; e.g. Why the Sea is Salt.
1) Yes, there are countably many constructions of various kinds of numbers. The construction can presumably be written down, and strings are finite-information-content entities. Yes, they’re normally understood to form a set-theoretic lattice—the integers are a subset of the gaussian integers, and the integers are a subset of the rationals, and both the gaussian and rationals are a subset of the complex plane.
However, the reals are not in any well-defined sense “the” limit of that lattice—you could create a contrived argument that they are, but you could also create an argument that the natural limit is something else, either stopping sooner, or continuing further to include infinities and infinitiesimals or (salient to mathematicians) the complex plane.
Defenders of the reals as a natural concept will use the phrase “the complete ordered field”, but when you examine the definition of “complete” they are referencing, it uses a significant amount of set theory (and an ugly Dedekind cuts construction) to include everything that it wants to include, and exclude many other things that might seem to be included.
2) Yes. I think the reals are a history-laden concept; they were built in fear of set-theoretic and calculus paradoxes, and without awareness of the computational approach—information theory and Godel’s incompleteness. They are useful primarily in the way that C++ is useful to C++ programmers—as a thicket or swamp of complexity that defends the residents from attack. Any mathematician doing useful work in a statistical, calculus-related, or topological field who casually uses the reals will need someone else, a numerical analyst or computer scientist, to laboriously go through their work and take out the reals, replacing them with a computable (and countable) alternative notion—different notions for different results. Often, this effort is neglected, and people use IEEE floats where the mathematician said “real”, and get ridiculous results—or worse, dangerously reasonable results.
3) You’re right that the finite amount of sensor data will only say it is consistent with this interval. As you point out, if there are an uncountable set within that interval, then it’s entirely possible for there to be no single value that is a maximum of the probability distribution function. (That’s an excellent example of some of the ridiculous circumlocutions that come from using uncountable sets, when you actually want the system to come up with one or a few best hypotheses, each of which is stateable.)
Pi is always a finite-information entity. Everything nameable is. It doesn’t become an infinitely large in information content just because you consider it as an element of the reals.
Yes—if you use a probability distribution over the rationals as your prior, and the actual value is irrational, then you can get bad results. I think this is a serious problem, and we should think hard about what bayesian updating with misspecified models looks like (I know Cosma Shalizi has done some work on this), so that we have some idea what failure looks like. We should also think carefully about what we would consider to be a reasonable hypothesis, one that we might eventually come to rest on.
However, it’s a false fork to argue “We wouldn’t use the rationals therefore we should use the reals”. As I’ve been trying to say, the reals are a particular, large, complicated, and deeply historical construction, and we should not expect to encounter them “out in the world”.
Andrej Bauer has implemented actual real number arithmetic (not IEEE nonsense, or “computable reals” which are interesting, but not reals). Peano integers, in his (Ocaml-based) language, RZ, would probably be five or ten lines. (Commutative groups are 13 lines). In contrast, building from commutative groups up to defining reals as sequences of nested intervals takes five pages; see the appendix: http://math.andrej.com/wp-content/uploads/2007/04/rzreals.pdf
Regarding “reminding you of things you didn’t think of”, I think Cosma Shalizi and Andrew Gelman have convincingly argued that Bayesian philosophy/methodology is flawed—we don’t just pick a prior, collect data, do an update, and believe the results. If we were magical uncomputable beasties (Solomonoff induction), possibly that is what we would do. In the real world, there are other steps, including examining the data, including the residual errors, to see if it suggests hypotheses that weren’t included in the original prior. http://www.stat.columbia.edu/~gelman/research/unpublished/philosophy.pdf
Perhaps you’ve confused the “pi has a decimal expansion goes on forever without seeming pattern” with “a generic real number has a decimal expansion that goes on forever without pattern”? Pi does have a finite representation, “Pi”. We use it all the time, and it’s just as precise as “2″ or “0.5”.
Specifically, you could start with the rationals, and complete it by including all solutions to differential equations. Pi would be included, and many other numbers, but you’d still only have a countable set—because every number would have one or more shortest definitions—finite information content.
If you had a probability distribution over such a set, it would naturally favor hypotheses with short definitions. If it started out including pi as a possibility, and you gathered sufficient sensor data consistent with pi (a finite amount), the probability distribution would give pi as the best hypothesis. This is reasonable behavior. You have to do non-obvious mucking around with your prior to get that sort of reasonableness with standard real-number probabilities.
As others have pointed out, any specific countable system of numbers (such as the “solutions to differential equations” that I mentioned) is susceptible to diagonalization—but I see no reason to “complete” the process, as if being susceptible to diagonalization were a terrible flaw above all others. All the entities that you’re actually manipulating (finite precision data and finite, stateable hypotheses like “exactly 2” and “exactly pi”) are finite-information-content, and “completing” the reals against diagonalization makes essentially all the reals infinite-information-content—a cure in my mind far worse than the disease.
If you start selecting things at random, then you need a probability distribution. Many routinely used probability distributions over the natural numbers give you a nontrivial chance of being able to fit the number on your computer.
There are, of course, corresponding probability distributions over the reals (take a probability distribution over the natural numbers and give zero probability to anything else). However, the routinely used probability distributions on the reals give zero probability to the output being a natural number, a rational number, describable with a finite algebraic equation, or in fact, being able to fit the number on your computer.
One of the problems with real numbers is that if someone trying to do Bayesian analysis of a sensor that reads 2.000..., or 3.14159… using one of these real number distributions as their prior, cannot conclude that the quantity measured probably is 2 or pi, no matter how many digits of precision the sensor goes out to.
“The set of busy beaver numbers” communicates, points to, refers to, picks out, a concept in natural language that we can both talk about. It has finite information content (a finite sequence of ascii characters suffices to communicate it). An analogous sentence in a sufficiently formal language would still have finite information content.
Note that a description, even if it is finite, is not necessarily in the form that you might desire. Transforming the description “the sequence of the first ten natural numbers” into the format “{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}” is easy, but the analogous transformation of “the first 10 busy beaver numbers” is very difficult if not impossible.
As nshepperd points out, an element of a countable set necessarily has finite information content (you can pick it out by providing an integer—“the fifth one” or whatever), while generic elements of uncountable sets cannot be picked out with finite information.
Yes, that’s my understanding as well.
Proof theory for second-order logic seems to be problematic, and I have a formalist stance towards mathematics in general, which leads me to suspect that the standard definitions of second-order logic are somehow smuggling in uncountable infinities, rather than justifying them.
But I admit second-order logic is not something I’ve studied in depth.
No, there’s nothing substantive beyond that.
My understanding is this thread was started, and to some extent kept rolling, by an unrelated thread, where I was behaving extremely hostile to EY, and several people went through all my back posts, looking for things to downvote. Patrick found something.
Yes, if you have to embrace the same messy compromises, then I am mistaken. My belief (which is founded on studying logic and mathematics in college, and then software development after college) is that better foundations, with effort, show through to better implementations.
Yes, that’s my position. In fact, if you had hardware support for an intuitionistic / constructivist representation (intervals, perhaps), my bet would be that the circuits would be simpler than the floating-point hardware implementing the IEEE standard now.
Are you suggesting that modeling real numbers with floating point is a good practice?
Yes, it is a standard practice, and it may be the best compromise available for a programmer or a team on a limited time budget, but the enormous differences between real numbers and floating point numbers mean that everything that was true upstream in math-land regarding real numbers, becomes suspect and have to be re-checked, or transformed into corresponding not-quite-the-same theorems and proofs.
If we (downstream consumers of mathematics) could get mathematicians to use realistic primitives, then we could put a lot of numerical analysts out of work (free up their labor to do something more valuable).
There’s a valuable difference between two different kinds of counterarguments (that I encountered in Drescher’s Good and Real, but I presume it’s widely known):
To correct someone, it is not sufficient to offer an argument for the opposite conclusion. At that point you merely have an apparent paradox—two arguments for opposite conclusions. You must also point out where in their argument they went wrong.
If everyone followed this policy, it could break certain circular or repetitious disputes which are trapped in a cycle of: A offers argument for X, B offers argument for not-X, A repeats their argument more clearly or more loudly, B repeats their argument in turn, and so on.