Trialing for the machine learning living library position at MIRI and occasional volunteer instructor and mentor at CFAR.
Qiaochu_Yuan
A simple model of calibration training is that it helps you more honestly integrate whatever evidence is floating around in your brain pertaining to a subject. Whether a prediction is short-term or long-term ought to be less important than other aspects of the quality of that evidence. This model predicts that, for example, calibration training on short-term predictions about which one has very little evidence should improve calibration on long-term predictions about which one also has very little evidence.
And people regularly make both short- and long-term predictions on PredictionBook, so in 5 to 10 years…
Suppose the AI lives in a universe with Turing oracles. Give it one.
“Non-zero probability” doesn’t seem like quite the right word. If a parameter describing the way things could conceivably turn out to be can take, say, arbitrary real values, then we really want “non-zero probability density.” (It’s mathematically impossible to assign non-zero probability to each of uncountably many disjoint hypotheses because they can’t add to 1.)
The first answer that occurred to me was “enumerate all Turing machines” but I’m worried because it seems pretty straightforward to coherently think up a universe that can’t be described by a Turing machine (either because Turing machines aren’t capable of doing computations with infinite-precision real numbers or because they can’t solve the halting problem). More generally I’m worried that “coherently-thinkable” implies “not necessarily describable using math,” and that would make me sad.
I don’t think your first point solves the problem. If the universe is exponentially sensitive to initial conditions, then even arbitrarily small inaccuracies in initial conditions make any simulation exponentially worse with time.
Suppose the AI lives in a universe where infinitely many computations can be performed in finite time...
(I’m being mildly facetious here, but in the interest of casting the “coherently-thinkable” net widely.)
Several mathematicians I know (and, I would guess, a sizable population of physicists as well) regard Feynman sums-over-histories as mathematical abstractions only. From this perspective they don’t describe processes that are actually happening out-there-in-the-world, they’re just mathematically convenient and maybe also intuitively useful. (I haven’t thought about whether or how this position can be reconciled with what I think is the standard LW position on many-worlds.)
If you ever plan on talking about your hypothesis, you need to be able to describe it in a language with a finite alphabet (such as English or a programming language). There are only countably many things you can say in a language with a finite alphabet, so there are only countably many hypotheses you can even talk about (unambiguously).
This means that if there are constants floating around which can have arbitrary real values, then you can’t talk about all but countably many of those values. (What you can do instead is, for example, specify them to arbitrary but finite precision.)
For a fixed amount of time. What if you wanted to simulate a universe that runs forever?
I’m not sure I understand the illustration. In particular, I don’t understand what “want” means if it doesn’t mean having a world-model over world-states and counting gliders.
When I read the beginning of this post I asked myself, “if people don’t have utility functions, why haven’t LWers gotten rich by constructing Dutch books against people?”
I answered myself, “in practice, most people will probably ignore clever-looking bets because they’ll suspect that they’re being tricked. One way to avoid Dutch books is to avoid bets in general.”
Me too, but almost all of it would be concentrated at “sums over histories are inaccurate descriptions of what happens.” Sums-over-histories are conceptually unsatisfying to me in that they use the classical concept of a history in order to describe quantum phenomena. My vague intuition is that a truer theory of physics would be more “inherently quantum.”
Thanks! I’m trying to branch out into writing things on the internet that aren’t just math. Hopefully it won’t come back to bite me in 20 years…
Point.
If I lived in such a universe, then it seems like I could potentially entertain uncountably many disjoint hypotheses about something, all of which I could potentially write down and potentially distinguish from one another. But I wouldn’t be able to assign more than countably many of them nonzero probability (because otherwise they couldn’t add to 1) as long as I stuck to real numbers. So it seems like I would have to revisit that particular hypothesis in Cox’s theorem…
The examples involving killing people seem like good examples, but the others seem like they could be predicated on disagreements about semantics rather than, say, disagreements about anticipated experiences (or utility functions, I guess). Words would need to be tabooed before I would trust those examples.
Well, for starters, an AI living in a universe where infinitely many computations can be performed in finite time can verify the responses a Turing oracle gives it. So it can determine that it lives in a universe with Turing oracles (in fact it can itself be a Turing oracle), which is not what an AI living in this universe would determine (as far as I know).
When you discuss the calibration results, could you mention that the surveyors were told what constituted a correct answer? I didn’t take the survey and it isn’t obvious from reading this post. Also, could you include a plug for PredictionBook around there? You’ve included lots of other helpful plugs.
I don’t understand what you mean by “matter.” People don’t care about hair color because hair color is not very predictive of other traits that people care about, but this doesn’t seem to be true of race.
Yes, let’s replace “computations” with “actions,” I guess.
That depends on what you mean by “strongly.” I would tentatively posit that even if race isn’t strongly predictive in an absolute sense of other traits that people care about, it is relatively predictive compared to other traits that are easy to unambiguously learn about a person. For example, if I wanted to predict the performance of a high school student on standardized tests, I think race would be a better predictor than height or weight, and I don’t know enough to confidently say whether it would a better predictor than income level.
I’ve recently begun to suspect that a possibly substantial amount of what gets labeled “racism” is just using race as weak Bayesian evidence in the spirit of http://lesswrong.com/lw/aq2/fallacies_as_weak_bayesian_evidence/ (edit: and then subsequently failing to distinguish between the probability of a statement being true having increased and the statement becoming true).
Hello! I’m a first-year graduate student in pure mathematics at UC Berkeley. I’ve been reading LW posts for awhile but have only recently started reading (and wanting to occasionally add to) the comments. I’m interested in learning how to better achieve my goals, learning how to choose better goals, and “raising the sanity waterline” generally. I have recently offered to volunteer for CFAR and may be an instructor at SPARC 2013.