Johnicholas

Karma: 2,008

Johnicholas Oct 23, 2011, 3:48 PM
10 points
on: Better Disagreement
The first few levels of the hierarchy, DH0, DH1, DH2, DH3, are not useless, if you’re concerned with getting to the truth efficiently. Pointing out that the author has no credentials in the area they are speaking in would allow people to limit their attention to credentialed speakers (due to bounded rationality). The senator example is similar—if you can’t listen to everything and have to prune someone, choosing to forgo listening to obviously biased speakers might be a reasonable choice.

DH2 and DH3 are sometimes valuable as well, because we can do some inference about the world, taking a piece of communication as evidence. For example, we may infer something about the author’s personality or beliefs from their tone; and then (bounded rationality again) decide whether to read further based on this (limited, probabilistic) estimate of the author.

Contradiction works similarly; it’s a performance, putting the reputation of the speaker behind the negated claim. If a third party hears a college freshman make a point, and the professor flatly contradicts it, then the third party may be able to decide who to believe, shortcutting detailed analysis (scarce processing again). In some cases, this kind of contradiction argument can even be supported solely on tone, without leaning on reputation. A scholarly-sounding contradiction to a casual claim may be sufficient to sway me in some situations—perhaps the stakes are low.

Johnicholas Oct 23, 2011, 3:30 PM
20 points
on: Better Disagreement
There’s a valuable difference between two different kinds of counterarguments (that I encountered in Drescher’s Good and Real, but I presume it’s widely known):

To correct someone, it is not sufficient to offer an argument for the opposite conclusion. At that point you merely have an apparent paradox—two arguments for opposite conclusions. You must also point out where in their argument they went wrong.

If everyone followed this policy, it could break certain circular or repetitious disputes which are trapped in a cycle of: A offers argument for X, B offers argument for not-X, A repeats their argument more clearly or more loudly, B repeats their argument in turn, and so on.

Johnicholas Oct 23, 2011, 12:06 PM
0 points
in reply to: kilobug’s comment on: The Protagonist Problem
The dashed/inaccurate is more valuable than solid, but it’s not really the connection that’s the locus of inaccuracy.

If instead the self-model contained a diagram inside of it, then we could see that the connection self-model to planning is working fine, it’s the diagram inside self-model that is wrong.

Johnicholas Sep 17, 2011, 1:51 PM
1 point
in reply to: JGWeissman’s comment on: The Optimizer’s Curse and How to Beat It
I’m trying to imagine a scenario.

Possibly the decider knows that people sometimes make multiplicative errors, transposing numbers or misplacing decimals, and is confronted with a set of estimates hovering around, say, 0.05 (and that is plausible according to the decider’s prior) and a few estimates at estimated around 0.5 and 5.0. Would the correction effectively trim the outliers back to almost exactly 0.05 (because we can’t learn much information from an estimate that probably had at least one mistake in it), and the decider should go with the highest of the “plausible” numbers?

It seems to me like the conditional distributions that would lead to actually changing your decision are nearly as likely to be a source of error as a correction.

Johnicholas Sep 5, 2011, 6:49 PM
8 points
in reply to: Will_Sawin’s comment on: Prisoner’s Dilemma Tournament Results
If you are willing to allow the optimal-against-this-pool strategy to run the entire tournament as a subroutine, I think it can guarantee a tie by cloning the eventual winner.

It may be able to do better than that, but it depends on the details of the pool whether there are points “lying on the ground” to be picked up; I suspect there are with this pool, but I don’t have a great example.

Johnicholas Sep 2, 2011, 10:30 AM
0 points
on: Is Rationality Teachable?
Charles Twardy wrote a paper (that I presented here). The impression I got was a bit more optimistic about the prospect of teaching critical thinking than from this survey.

Johnicholas Aug 31, 2011, 6:21 PM
−1 points
in reply to: lessdazed’s comment on: Consequentialism Need Not Be Nearsighted
There are various distinguishing points between geocentrism and heliocentrism—one is that heliocentrism had fewer free parameters. More elaborately: Certain parameters of the epicycles had to be delicately balanced, and heliocentrism “predicted” that those parameters would be turn out to be balanced. This sense of prediction has nothing to do with previous-in-time, and everything to do with opportunities for falsification; if ever those parameters drifted from perfect balance, heliocentrism would be falsified. Heliocentrism was preferable (even though it was slightly less accurate at first) because it exposed itself to a more stringent test.

Overall, what I’m saying is that those two theories are asymmetric.

Johnicholas Aug 31, 2011, 4:10 PM
1 point
on: Consequentialism Need Not Be Nearsighted
Consequentialism and deontologism can be encoded in terms of one another (please allow me to temporarily mischaracterize the discussion as if there were only two options consequentialism and deontologism). Both theories have “free parameters”; consequentialism has preferences over states, and deontologism has precepts (should-rules). By carefully setting the free parameters, you can turn one into the other. The deontologist can say “You should make decisions by considering their consequences according to this utility function”, and the consequentialist may say “I prefer world-lines where I obey these precepts”.

Given this background, advocacy of consequentialism over deontolgism is essentially saying that the correct entire theory (that is, including the free parameters as well) is best expressed using the language of preferences and states-of-the-world.

However, there are things that I value, including autonomy, freedom, self-determination, that are difficult to express as states-of-the-world. These can be expressed as preferences over world-lines or paths through configuration space, of course. However, this is an awkward encoding.

My argument against foundational consequentialism (as opposed to practical, day-to-day consequentialism) is that most of the things that I ultimately value are actually paths and properties of paths, and not states or properties of states at all. If the only reason to pick one over the other is the apt capture of the entire theory, then I prefer to discuss ethics in terms of “shoulds” and relegate consequentialism to a practical, day-to-day role.

Possibly we can set foundations aside (or agree as a group to be foundationally-pluralistic), and focus on very-slightly-more-practical questions?

Johnicholas Jul 4, 2011, 3:45 PM
5 points
on: The Blue-Minimizing Robot
One of the obvious extensions of this thought experiment is to posit a laser-powered blue goo that absorbs laser energy, and uses it to grow larger.

This thought experiment also reminds me: Omohundro’s arguments regarding likely uFAI behavior are based on the AI having goals of some sort—that is, something we would recognize as goals. It’s entirely possible that we wouldn’t perceive it as having goals at all, merely behavior.

Johnicholas Jun 14, 2011, 4:02 PM
1 point
on: Model Uncertainty, Pascalian Reasoning and Utilitarianism
As I understand it, the scenario is that you’re hearing a complicated argument, and you don’t fully grok or internalize it. As advised by “Making Your Explicit Reasoning Trustworthy”, you have decided not to believe it fully.

The problem comes in the second argument—should you take the advice of the person (or meme) that you at least somewhat mistrust in “correcting” for your mistrust? As you point out, if the person (or the meme) is self-serving, then the original proposal and the correction procedure will fit together neatly to cause a successful mugging.

I think concerns like this suggest that we ought to be using some sort of robust statistics—that is, the direction of the argument and the fact that the conclusion is extreme should influence our conclusions, but the magnitude cannot be allowed to influence our conclusions.

Johnicholas Jun 14, 2011, 3:46 PM
1 point
on: Model Uncertainty, Pascalian Reasoning and Utilitarianism
The link to Anna’s post in the footnotes is broken. Should be here.

Johnicholas Jun 9, 2011, 1:15 PM
−3 points
on: Safety Culture and the Marginal Effect of a Dollar
You focus on visibly HAL-like or Skynet-like AI—the sort of thing that AI researchers produce as demos. However, we have large, smart, durable, existing entities (businesses and other computer+human teams) that are continuously getting smarter (and entrenching themselves deeper into our society) by automating their existing business practices.

I don’t advocate trying to stop business automation, or humans organizing themselves into better and better teams; I think that would be throwing the baby out with the bathwater. However, I do think “business as usual” or “the default future” is the threat that existential risks people should be imagining.

The vast majority of writing about these issues has a story of terrorists or scientists (who are wizards meddling with things man was not meant to know) accidentally creating paperclip-making machines. That isn’t thinking, that’s straight out of folklore; e.g. Why the Sea is Salt.

Johnicholas Jun 4, 2011, 4:57 PM
5 points
in reply to: bogdanb’s comment on: The Pascal’s Wager Fallacy Fallacy
1) Yes, there are countably many constructions of various kinds of numbers. The construction can presumably be written down, and strings are finite-information-content entities. Yes, they’re normally understood to form a set-theoretic lattice—the integers are a subset of the gaussian integers, and the integers are a subset of the rationals, and both the gaussian and rationals are a subset of the complex plane.

However, the reals are not in any well-defined sense “the” limit of that lattice—you could create a contrived argument that they are, but you could also create an argument that the natural limit is something else, either stopping sooner, or continuing further to include infinities and infinitiesimals or (salient to mathematicians) the complex plane.

Defenders of the reals as a natural concept will use the phrase “the complete ordered field”, but when you examine the definition of “complete” they are referencing, it uses a significant amount of set theory (and an ugly Dedekind cuts construction) to include everything that it wants to include, and exclude many other things that might seem to be included.

2) Yes. I think the reals are a history-laden concept; they were built in fear of set-theoretic and calculus paradoxes, and without awareness of the computational approach—information theory and Godel’s incompleteness. They are useful primarily in the way that C++ is useful to C++ programmers—as a thicket or swamp of complexity that defends the residents from attack. Any mathematician doing useful work in a statistical, calculus-related, or topological field who casually uses the reals will need someone else, a numerical analyst or computer scientist, to laboriously go through their work and take out the reals, replacing them with a computable (and countable) alternative notion—different notions for different results. Often, this effort is neglected, and people use IEEE floats where the mathematician said “real”, and get ridiculous results—or worse, dangerously reasonable results.

3) You’re right that the finite amount of sensor data will only say it is consistent with this interval. As you point out, if there are an uncountable set within that interval, then it’s entirely possible for there to be no single value that is a maximum of the probability distribution function. (That’s an excellent example of some of the ridiculous circumlocutions that come from using uncountable sets, when you actually want the system to come up with one or a few best hypotheses, each of which is stateable.)

Pi is always a finite-information entity. Everything nameable is. It doesn’t become an infinitely large in information content just because you consider it as an element of the reals.

Yes—if you use a probability distribution over the rationals as your prior, and the actual value is irrational, then you can get bad results. I think this is a serious problem, and we should think hard about what bayesian updating with misspecified models looks like (I know Cosma Shalizi has done some work on this), so that we have some idea what failure looks like. We should also think carefully about what we would consider to be a reasonable hypothesis, one that we might eventually come to rest on.

However, it’s a false fork to argue “We wouldn’t use the rationals therefore we should use the reals”. As I’ve been trying to say, the reals are a particular, large, complicated, and deeply historical construction, and we should not expect to encounter them “out in the world”.

Andrej Bauer has implemented actual real number arithmetic (not IEEE nonsense, or “computable reals” which are interesting, but not reals). Peano integers, in his (Ocaml-based) language, RZ, would probably be five or ten lines. (Commutative groups are 13 lines). In contrast, building from commutative groups up to defining reals as sequences of nested intervals takes five pages; see the appendix: http://math.andrej.com/wp-content/uploads/2007/04/rzreals.pdf

Regarding “reminding you of things you didn’t think of”, I think Cosma Shalizi and Andrew Gelman have convincingly argued that Bayesian philosophy/methodology is flawed—we don’t just pick a prior, collect data, do an update, and believe the results. If we were magical uncomputable beasties (Solomonoff induction), possibly that is what we would do. In the real world, there are other steps, including examining the data, including the residual errors, to see if it suggests hypotheses that weren’t included in the original prior. http://www.stat.columbia.edu/~gelman/research/unpublished/philosophy.pdf

Johnicholas May 31, 2011, 11:20 AM
4 points
in reply to: bogdanb’s comment on: The Pascal’s Wager Fallacy Fallacy
Perhaps you’ve confused the “pi has a decimal expansion goes on forever without seeming pattern” with “a generic real number has a decimal expansion that goes on forever without pattern”? Pi does have a finite representation, “Pi”. We use it all the time, and it’s just as precise as “2″ or “0.5”.

Specifically, you could start with the rationals, and complete it by including all solutions to differential equations. Pi would be included, and many other numbers, but you’d still only have a countable set—because every number would have one or more shortest definitions—finite information content.

If you had a probability distribution over such a set, it would naturally favor hypotheses with short definitions. If it started out including pi as a possibility, and you gathered sufficient sensor data consistent with pi (a finite amount), the probability distribution would give pi as the best hypothesis. This is reasonable behavior. You have to do non-obvious mucking around with your prior to get that sort of reasonableness with standard real-number probabilities.

As others have pointed out, any specific countable system of numbers (such as the “solutions to differential equations” that I mentioned) is susceptible to diagonalization—but I see no reason to “complete” the process, as if being susceptible to diagonalization were a terrible flaw above all others. All the entities that you’re actually manipulating (finite precision data and finite, stateable hypotheses like “exactly 2” and “exactly pi”) are finite-information-content, and “completing” the reals against diagonalization makes essentially all the reals infinite-information-content—a cure in my mind far worse than the disease.

Johnicholas May 16, 2011, 10:58 AM
3 points
in reply to: bogdanb’s comment on: The Pascal’s Wager Fallacy Fallacy
If you start selecting things at random, then you need a probability distribution. Many routinely used probability distributions over the natural numbers give you a nontrivial chance of being able to fit the number on your computer.

There are, of course, corresponding probability distributions over the reals (take a probability distribution over the natural numbers and give zero probability to anything else). However, the routinely used probability distributions on the reals give zero probability to the output being a natural number, a rational number, describable with a finite algebraic equation, or in fact, being able to fit the number on your computer.

One of the problems with real numbers is that if someone trying to do Bayesian analysis of a sensor that reads 2.000..., or 3.14159… using one of these real number distributions as their prior, cannot conclude that the quantity measured probably is 2 or pi, no matter how many digits of precision the sensor goes out to.

Johnicholas May 3, 2011, 5:16 PM
0 points
in reply to: Patrick’s comment on: The Pascal’s Wager Fallacy Fallacy
“The set of busy beaver numbers” communicates, points to, refers to, picks out, a concept in natural language that we can both talk about. It has finite information content (a finite sequence of ascii characters suffices to communicate it). An analogous sentence in a sufficiently formal language would still have finite information content.

Note that a description, even if it is finite, is not necessarily in the form that you might desire. Transforming the description “the sequence of the first ten natural numbers” into the format “{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}” is easy, but the analogous transformation of “the first 10 busy beaver numbers” is very difficult if not impossible.

As nshepperd points out, an element of a countable set necessarily has finite information content (you can pick it out by providing an integer—“the fifth one” or whatever), while generic elements of uncountable sets cannot be picked out with finite information.

Johnicholas Apr 29, 2011, 11:11 AM
3 points
in reply to: Eliezer Yudkowsky’s comment on: The Pascal’s Wager Fallacy Fallacy
Yes, that’s my understanding as well.

Proof theory for second-order logic seems to be problematic, and I have a formalist stance towards mathematics in general, which leads me to suspect that the standard definitions of second-order logic are somehow smuggling in uncountable infinities, rather than justifying them.

But I admit second-order logic is not something I’ve studied in depth.

Johnicholas Apr 29, 2011, 10:36 AM
1 point
in reply to: Sniffnoy’s comment on: The Pascal’s Wager Fallacy Fallacy
No, there’s nothing substantive beyond that.

My understanding is this thread was started, and to some extent kept rolling, by an unrelated thread, where I was behaving extremely hostile to EY, and several people went through all my back posts, looking for things to downvote. Patrick found something.

Johnicholas Apr 29, 2011, 10:31 AM
1 point
in reply to: cousin_it’s comment on: The Pascal’s Wager Fallacy Fallacy
Yes, if you have to embrace the same messy compromises, then I am mistaken. My belief (which is founded on studying logic and mathematics in college, and then software development after college) is that better foundations, with effort, show through to better implementations.

Johnicholas Apr 28, 2011, 8:16 PM
1 point
in reply to: cousin_it’s comment on: The Pascal’s Wager Fallacy Fallacy
Yes, that’s my position. In fact, if you had hardware support for an intuitionistic / constructivist representation (intervals, perhaps), my bet would be that the circuits would be simpler than the floating-point hardware implementing the IEEE standard now.