# abramdemski(Abram Demski)

Karma: 10,788
• What’s the point of a loan if you need 100% collateral, and the collateral isn’t something like a house that you can put to good use while using as collateral?

If I can use bitcoin as collateral to get some Doge, hoping to make enough money with the Doge to get my bitcoin back later… couldn’t I just sell my bitcoin to buy the Doge, again hoping to make enough money to get my bitcoin back later?

• A bettor who can make an infinite number of expected profitable bets is going to outperform one who can only make a finite number of bets.

Agreed

(any number between 1 and 0 exclusive)^infinity=0

Agreed

i.e. for an infinite series of bets, the probability of ruin with naive EV maximization is 1.

Agreed

So, expected value is actually −1x your bet size.

Not so!

I think you’ll start to see where you’re wrong pretty fast if you examine finite examples.

Let’s consider a max-expectation bettor on a double-or-nothing bet with an 80% probability of paying out.

My expected value per dollar in this bet is $1.60, whereas the expected value of a dollar in my pocket is$1. So I maximize expected value by putting all my money in. If I start with $100, my expected value after 1 round is$160. The expected value of playing this way for two rounds is $100x1.6x1.6 =$256. In general, the expected value of this strategy is 100 .

The Kelly strategy puts 60% of its money down, instead. So in expectation, the Kelly strategy multiplies the money by .

So after one round, the Kelly bettor has $136 in expectation. After two rounds, about$185. In general, the Kelly strategy gets an expected value of .

So, after a large number of rounds, the all-in strategy will very significantly exceed the Kelly strategy in expected value.

I suspect you will object that I’m ignoring the probability of ruin, which is very close to 1 after a large number of rounds. But the expected value doesn’t ignore the probability of ruin. It’s already priced in: the expected value of 1.6 includes the 80% chance of success and the 20% chance of failure: . Similarly, the $256 expected value for two rounds already accounts for the chance of zero; you can see how by multiplying out (which shows the three possibilities which have value zero, and the one which doesn’t). Similarly for the th round: the expected value of already discounts the winnings by the (tiny) probability of success. (Otherwise, the sum would be$2^n instead.)

• One interpretation of this would be imitation learning: teaching a system to imitate human strategies, rather than optimize some objective of its own.

The problem with imitation learning is: since humans are pretty smart, a close imitation of a human strategy is probably going to involve planning in the deliberate service of some values. So if you set a big neural network on the problem of imitating humans, it will develop its own preferences and ability to plan. This is a recipe for an inner optimizer. Its values and planning will have to line up with humans in typical cases, but in extreme cases (eg adversarial examples), it could be very different. This can be a big problem, because the existence of such an AI could itself push us to extreme cases where the AI has trouble generalizing.

Another interpretation of your idea could be “approval-directed agents”. These are not trained to imitate humans, but rather, trained based on human approval of actions. However, unlike reinforcement learners, they don’t plan ahead to maximize expected approval. They only learn to take specific actions more when they are approved of, and less when they earn disapproval.

Unlike imitation learners, approval-directed agents can be more capable than human trainers. However, unlike reinforcement learning agents, approval-directed agents don’t have any incentive to take over control of their reward buttons. All the planning ahead comes from humans, looking at particular sorts of actions and deciding that they’re good.

Unfortunately, this still faces basically the same problem as imitation learning. Because humans are approving/​disapproving based on complicated models of the world and detailed thoughts about the consequences of actions, a big neural network has good reason to replicate those faculties within itself. You get an inner optimizer again, with the risks of misalignment that this brings.

• I have often wondered why use of IQ in hiring isn’t more common, so I just sorta believed it when you said it’s illegal, even though I probably looked into it and figured out it wasn’t on a previous occasion.

OTOH, businesses might be afraid of getting sued anyway, based on the supreme court case.

• “The map is not the territory” does seem like a step up from “the name that can be named is not the eternal name”. Though, that could be a translation issue.

• As GuySrinavasan says, do the math. It doesn’t work out. Maximizing geometric growth rate is not the same as maximizing mean value. It turns out Kelly favors the first at a severe cost to the second.

This is my big motivator for writing stuff like this: discussions of Kelly usually prove an optimality notion like expected growth rate, and then leave it to the reader to notice that this doesn’t at all imply more usual optimality notions. Most readers don’t notice; it’s very natural to assume that “Kelly maximizes growth rate” entails “Kelly maximizes expected wealth”.

But if Kelly maximized expected wealth, then that would probably have been proved instead of this geometric-growth-rate property. You have to approach mathematics the same way you approach political debates, sometimes. Keep an eye out for when theorems answer something only superficially similar to the question you would have asked.

• Yep, I actually note this in footnote 3. I didn’t change section 2 because I still think that if each of these is individually bad, it’s pretty questionable to use them as justification for Kelly.

Note that if a strategy is better or equal in every quantile, and strictly better in some, compared to some , then expected utility maximization will prefer to , no matter what the utility function is (so long as more money is considered better, ie utility is monotonic).

So all expected utility maximizers would endorse an all-quantile-optimizing strategy, if one existed. This isn’t a controversial property from the EU perspective!

But it’s easy to construct bets which prove that maximizing one quantile is not always consistent with maximizing another; there are trade-offs, so there’s not generally a strategy which maximizes all quantiles.

So it’s critically important that Kelly is only approximately doing this, in the limit. If Kelly had this property precisely, then all expected utility maximizers would use the Kelly strategy.

In particular, at a fixed finite time, there’s a quantile for the all-win sequence. However, since this quantile becomes smaller and smaller, it vanishes in the limit. At finite time, the expected-money-maximizer is optimizing this extreme quantile, but the Kelly strategy is making trade-offs which are suboptimal for that quantile.

(Note: maybe I’m misunderstanding what johnswentworth said here, but if solving for any x%-quantile maximizer always yields Kelly, then Kelly maximizes for all quantiles, correct?)

That’s my belief too, but I haven’t verified it. It’s clear from the usual derivation that it’s approximately mode-maximizing. And I think I can see why it’s approximately median-maximizing by staring at the wikipedia page for log-normal long enough and crossing my eyes just right [satire].

• Hopefully it’s clear to readers that I picked a contrarian title for fun, rather than because it’s the best description of our disagreement.

Somehow I’m suspicious that we still have pretty big implicit disagreements about what kinds of arguments are OK to make; like if someone asked you to explain Kelly to them at a party, you might still rant about how it’s not about utility, and you’d still say something along the lines of Ole Peters’ time-averaging stuff. Or maybe I’m just reacting to the way you’re not specifically saying you were wrong about any of the stuff you wrote. But I’m not saying “you’re wrong and you should recant”, I’m saying I’m hopeful that there’s still a productive disagreement between us that we can learn from. EG, if you have in you any defense of Ole Peters or similar arguments, I would like to hear it.

• I would be interested if you could say more about the CRRA-optimality point. Does this give us the class of utility functions which prefer fractional Kelly to full Kelly? What do those functions look like?

• Upvoted for levels 5, 6, and 7, which I didn’t expect to buy, but totally buy.

• I think there are a couple of responses the holy-madman type can give:

• The holy-madman aesthetic is actually pretty nice. Human values include truth, which requires coherent thought. And in fiction, we especially enjoy heroes who go after coherent goals. So in practice in our current world, the tails don’t come apart much. At worst, people who manage to be more agentic aren’t making too big of a sacrifice in the incarnation department. And perhaps they’re actually better-off in that respect.

• A coherent agent is basically what happens when you can split up the problem of deciding what to do and doing it, because most of the expected utility is in the rest of the world. An effective altruist who cares about cosmic waste probably thinks your argument is referring to something pretty negligible in comparison. Even if you argue functional decision theory means you’re controlling all similar agents, not just yourself, that could still be pretty negligible.

• Right, I think this is a pretty plausible hypothesis.

Here’s another perspective: Scott is writing the perspective of (something like) the memes, who exert some control but don’t have root access. The memes have a lot of control over when we feel good or bad about ourselves (this is a primary control mechanism they have). But the underlying biological organism has more control over what we actually do or don’t do.

The memes also don’t have a great deal of self-awareness of this split agency. They see themselves as the biological organism. So they’re actually a bit puzzled about why the organism doesn’t maximize the memetic values all the time.

One strategy which the memes use, in response to this situation, is to crank up the guilt-o-meter whenever actions don’t reflect explicitly endorsed values.

Scott and Nate are both arguing against this strategy. Scott’s SSC perspective is something like: “Don’t feel guilty all the time. You don’t have to go all the way with your principles. It’s OK to apply those principles selectively, so long as you make sure you’re not doing it in a biased way to get what you want.”

This is basically sympathetic to the “you should feel guilty if you do bad things” idea, but arguing about how to set the threshold.

Nate’s Minding Our Way perspective is instead: “Guilt isn’t an emotion that a unified agent would feel. So you must be a fractured agent. You’re at war with yourself; what you need is a peace treaty. Work to recognize your fractured architecture, and negotiate better and better treaties. After a while you’ll be acting like a unified agent.”

• Right, I agree with your distinction. I was thinking of this as something Scott was ignoring, when he wrote about selling all your possessions. I don’t want to read into it too much, since it was an offhand example of what it would look like to go all the way in the taking-altruism-seriously direction. But it does seem like Scott (at the time) implicitly believed that going too far would include things of this sort. (That’s the point of his example!) So when you say:

The effect Scott probably worries about is the following:

I’m like, no, I don’t think Scott was explicitly reasoning this way. Infinite Debt was not about how altruists need to think long-term about what does the most good. It was a post about how it’s OK not to do that all the time, and principles like altruism should be allowed to ask arbitrarily much from us. Yes, you can make an argument “thinking about the long-term good all the time isn’t the best way to produce the most long-term good” and “asking people to be as good as possible isn’t the best way to get them to be as good as possible” and things along those lines. But for better or worse, that’s not the argument in the post.

• It may come from confidence in his calibration about a claim instead of in the claim itself.

Minding Our Way addresses this very phenomenon in Confidence All The Way Up. To my eye, Scott Alexander articulates his uncertainty with an air of meta-uncertainty; even when he sounds certain, he sounds tentatively uncertain. For example, his posts sometimes proceed in sections where each tells a strong story, but the next section contradicts the story, telling a new story from an opposite perspective. This gives a sense that no matter how strong an argument is, it could be knocked down by an even stronger argument which blindsides you. This kind of thing is actually another obsession of Scott’s (by my estimation).

In contrast, Nate Soares articulates his uncertainty with an air of meta-confidence; he’s uncertain, but he knows a lot about where that uncertainty comes from and what would change his mind. He can put numbers to it. If he’s not sure about what would change his mind, he can tell you about how he would figure it out. And so on.

• This (to be clear) is not fractional Kelly, where I think we’re talking about a situation where the fraction is constant.

In the same way that “the Kelly strategy” in practice refers to betting a variable fraction of your wealth (even if the simple scenarios used to illustrate/​derive the formula involve the same bet repeatedly, so the Kelly strategy is one which implies betting a fixed fraction of wealth), I think it’s perfectly sensible to use “fractional Kelly” to describe a strategy which takes a variable fraction of the Kelly bet, using some formula to determine the fraction (even if the argument we use to establish the formula is one where a constant Kelly fraction is optimal).

What I would take issue with would be an argument for fractional Kelly which assumed we should use a constant Kelly fraction (as I said, “tying the agent’s hands” by only looking at strategies where some constant Kelly fraction is chosen). Because then it’s not clear whether some fractional-Kelly is the best strategy for the described scenario; it’s only clear that you’ve found some formula for which fractional-Kelly is best in a scenario, given that you’re using some fractional Kelly.

Which was one of my concerns about what might be going on with the first argument.

The result that “uncertainty ⇒ go sub-Kelly” is robust to different models of uncertainty.

I find myself really wishing that you’d use slightly more Bayesian terminology. Kelly betting is already a rule for betting under uncertainty. You’re specifically saying that meta-uncertainty implies sub-kelly. (Or parameter uncertainty, or whatever you want to call it.)

I’m trying to find the right Bayesian way to express this, without saying the word “True probability”.

I appreciate the effort :)

So the graph shows what happens if we take our uncertainty and keep it as-is, not updating on data, as we continue to update?

Yes. Think of it as having a series of bets on different events with the same uncertainty each time.

Right… so in this case, it pretty strongly seems to me like the usual argument for Kelly applies. If you have a series of different bets in which you have the same meta-uncertainty, either your meta-uncertainty is calibrated, in which case your probability estimates will be calibrated, so the Kelly argument works as usual, or your meta-uncertainty is uncalibrated, in which case I just go meta on my earlier objections: why aren’t we updating our meta-uncertainty? I’m fine with assuming repeated different bets (from different reference classes) with the same parameter uncertainty being applied to all of them so long as it’s apparently sensible to apply the same meta-uncertainty to all of them. But systematic errors in your parameter uncertainty (such that you can look at a calibration graph and see the problem) should trigger an update in the general priors you’re using.

Here I am considering ∫ (notice the Kelly fraction depending on inside the utility but not outside). “What is my expected utility, if I bet according to Kelly given my estimate”. (Ans: Not Full Kelly)

I think you are talking about the scenario ∫? (Ans: Full Kelly)

(Sorry, had trouble copying the formulae on greaterwrong)

I think what you’re pointing to here is very much like the difference between unbiased estimators and bayes-optimal estimators, right? Frequentists argue that unbiased estimators are better, because given any value of the true parameter, an unbiased estimator is in some sense doing a better job of approximating the right answer. Bayesians argue that Bayesian estimators are better, because of the bias-variance trade-off, and because you expect the Bayesian estimator to be more accurate in expectation (the whole point of accounting for the prior is to be more accurate in more typical situations).

I think the Bayesians pretty decisively win that particular argument; as an agent with a subjective perspective, you’re better off doing what’s best from within that subjective perspective. The Frequentist concept is optimizing based on a God’s-eye view, where we already know . In this case, it leads us astray. The God’s-eye view just isn’t the perspective from which a situated agent should optimize.

Similarly, I think it’s just not right to optimize the formula you give, rather than the one you attribute to me. If I have parameter uncertainty, then my notion of the expected value of using fractional Kelly is going to come from sampling from my parameter uncertainty, and checking what the expected payoffs are for each sample.

But then, as you know, that would just select a Kelly fraction of 1.

So if that formula describes your reasoning, I think you really are making the “true probability” mistake, and that’s why you’re struggling to put it in terms that are less objectionable from the Bayesian perspective. (Which, again, I don’t think is always right, but which I think is right in this case.)

(FYI, I’m not really arguing against fractional Kelly; full Kelly really does seem too high in some sense. I just don’t think this particular argument for fractional Kelly makes sense.)

Consider a scenario where we’re predicting a lot of (different) sports events. We could both be perfectly calibrated (what you say happens 20% of the time happens 20% of the time) etc, but I could be more “uncertain” with my predictions. If my prediction is always 50-50 I am calibrated, but I really shouldn’t be betting. This is about adjusting your strategy for this uncertainty.

I think what’s going on in this example is that you’re setting it up so that I know strictly more about sports than you. You aren’t willing to bet, because anything you know about the situation, I know better. In terms of your post, this is your second argument in favor of Kelly. And I think it’s the explanation here. I don’t think your meta-uncertainty has much to do with it.

Particularly if, as you posit, you’re quite confident that 50-50 is calibrated. You have no parameter uncertainty: your model is that of a fair coin, and you’re confident it’s the best model in the coin-flip model class.

BAYESIAN: Right… look, when I accepted the original Kelly argument, I wasn’t really imagining this circumstance where we face the exact same bet over and over. Rather, I was imagining I face lots of different situations. So long as my probabilities are calibrated, the long-run frequency argument works out the same way. Kelly looks optimal. So what’s your beef with me going “full Kelly” on those estimates?

No, my view were always closer to BAYESIAN here. I think we’re looking at a variety of different bets but where my probabilities are calibrated but uncertain. Being calibrated isn’t the same as being right. I have always assumed here that you are calibrated.

Then you concede the major assumption of BAYESIAN’s argument here! Under the calibration assumption, we can show that the long-run performance of Kelly is optimal (in the peculiar sense of optimality usually applied to Kelly, that is).

I’m curious how you would try and apply something like your formula to the mixed-bet case (ie, a case where you don’t have the same meta-uncertainty each time).

The strawman of your argument (which I’m struggling to understand where you differ) is. “A Bayesian with log-utility is repeatedly offered bets (mechanism for choosing bets unclear) against an unfair coin. His prior is that the coin comes up heads is uniform [0,1]. He should bet Full Kelly with p = 1⁄2 (or slightly less than Full Kelly once he’s updated for the odds he’s offered)”. I don’t think he should take any bets. (I’m guessing you would say that he would update his strategy each time to the point where he no longer takes any bets—but what would he do the first time? Would he take the bet?)

Here’s how I would fix this strawman. Note that the fixed strawman is still straw in the sense that I’m not actually arguing for full Kelly, I’m just trying to figure out your argument against it.

“A Bayesian with log-utility is repeatedly offered bets (coming from a rich, complex environment which I’m making no assumptions about, not even computability). His probabilities are, however, calibrated. Then full Kelly will be optimal.”

Probably there are a few different ways to mathify what I mean by “optimal” in this argument. Here are some observations/​conjectures:

• Full Kelly optimizes the expected utility of this agent, obviously. So if the agent really has log utility, and really is a Bayesian, clearly it’ll go full Kelly.

• After enough bets, since we’re calibrated, we can assume that the frequency of success for bets will closely match . So we can make the usual argument that full Kelly will be very close to optimal: ** Fractional Kelly, or other modified Kelly formulas, will make less money. ** In general, any other strategy will make less money in the long run, under the assumption that long-run frequencies match probabilities—so long as that strategy does not contain further information about the world.

(For example, in your example where you have an ignorant but calibrated 50-50 model, maybe the true world is “yes on even-numbered dates, no on odd”. A strategy based on this even-odd info could outperform full Kelly, obviously. The claim is that so long as you’re not doing something like that, full Kelly will be approximately best.)

I think there’s something which I’ve not made clear but I’m not 100% I know we’ve found what it is yet.

My current estimate is that this is 100% about the frequentist Gods-eye-view way of arguing, where you evaluate the optimality of something by supposing a “true probability” and thinking about how well different strategies do as a function of that.

If so, I’ll be curious to hear your defense of the gods-eye perspective in this case.

One thing I want to make clear is that I think there’s something wrong with your argument on consequentialist grounds.

Or maybe the graph is of a single step of Kelly investment, showing expected log returns? But then wouldn’t Kelly be optimal, given that Kelly maximizes log-wealth in expectation, and in this scenario the estimate is going to be right on average, when we sample from the prior?

Yeah—the latter—I will edit this to make it clearer. This is “expected utility” for one-period. (Which is equivalent to growth rate). I just took the chart from their paper and didn’t want to edit it. (Although that would have made things clearer. I think I’ll just generate the graph myself).

Looking at the bit I’ve emphasised. No! This is the point.

I want to emphasize that I also think there’s something consequentialistly weird about your position. As non-Bayesian as some arguments for Kelly are, we can fit the Kelly criterion with Bayes, by supposing logarithmic utility. So a consequentialist can see those arguments as just indirect ways of arguing for logarithmic utility.

Not so with your argument here. If we asses a gamble as having probability , then what could our model uncertainty have to do with anything? Model uncertainty can decrease our confidence that expected events will happen, but already prices that in. Model uncertainty also changes how we’ll reason later, since we’ll update on the results here (and wouldn’t otherwise do so). But, that doesn’t matter until later.

We’re saying: “Event might happen, with probability ; event might happen, with probability .” Our model uncertainty grants more nuance to this model by allowing us to update it on receiving more information; but in the absence of such an update, it cannot possibly be relevant to the consequences of our strategies in events and . Unless there’s some funny updateless stuff going on, which you’re clearly not supposing.

From a consequentialist perspective, then, it seems we’re forced to evaluate the expected utility in the same way whether we have meta-uncertainty or not.

• Thanks for writing this!

Just to be pedantic, I wanted to mention: if we take Fractional Kelly as the average-with-market-beliefs thing, it’s actually full Kelly in terms of our final probability estimate, having updated on the market :)

Concerning your first argument, that uncertainty leads to fractional Kelly—is the idea:

1. We have a probability estimate , which comes from estimating the true frequency ,

2. Our uncertainty follows a Beta distribution,

3. We have to commit to a fractional Kelly strategy based on our and never update that strategy ever again

?

So the graph shows what happens if we take our uncertainty and keep it as-is, not updating on data, as we continue to update?

Or is it that we keep updating (and hence reduce our uncertainty), but nonetheless, keep our Kelly fraction fixed (so we don’t converge to full Kelly even as we become increasingly certain)?

Also, I don’t understand the graph. (The third graph in your post.) You say that it shows growth rate vs Kelly fraction. Yet it’s labeled “expected utility”. I don’t know what “expected utility” means, since the expected utility should grow unboundedly as we increase the number of iterations.

Or maybe the graph is of a single step of Kelly investment, showing expected log returns? But then wouldn’t Kelly be optimal, given that Kelly maximizes log-wealth in expectation, and in this scenario the estimate is going to be right on average, when we sample from the prior?

Anyway, I’m puzzled about this one. What exactly is the take-away? Let’s say a more-or-less Bayesian person (with uncertainty about their utilities and probabilities) buys the various arguments for Kelly, so they say, “In practice, my utility is more or less logarithmic in cash, at least in so far as it pertains to situations where I have repeated opportunities to invest/​bet”.

Now lets assume there’s some uncertainty in that. (Bayesians might get a little uncomfortable here—posterior distributions for discrete events are point estimates. Instead imagine you view the event as a Bernoulli random variable with parameter p, and you have a posterior distribution for p.*).

BAYESIAN: Wait, what? I agree I’ll have parameter uncertainty. But we’ve already established that my utility is roughly logarithmic in money. My point estimate (my posterior) for this gamble paying off is . The optimal bet under these assumptions is Kelly. So what are you saying? Perhaps you’re arguing that my best-estimate probability isn’t really .

OTHER: No, is really your best-estimate probability. I’m pointing to your model uncertainty.

BAYESIAN: Perhaps you’re saying that my utility isn’t really logarithmic? That I should be more risk-averse in this situation?

OTHER: No, my argument doesn’t involve anything like that.

BAYESIAN: So what am I missing? Log utility, probability , therefore Kelly.

OTHER: Look, one of the ways we can argue for Kelly is by studying the iterated investment game, right? We look at the behavior of different strategies in the long term in that game. And we intuitively find that strategies which don’t maximize growth (EG the expected-money-maximizer) look pretty dumb. So we conclude that our values must me closer to the growth-maximizer, ie Kelly, strategy.

BAYESIAN: Right; that’s part of what convinced me that my values must be roughly logarithmic in money.

OTHER: So all I’m trying to do is examine the same game. But this time, rather than assuming we know the frequency of success from the beginning, I’m assuming we’re uncertain about that frequency.

BAYESIAN: Right… look, when I accepted the original Kelly argument, I wasn’t really imagining this circumstance where we face the exact same bet over and over. Rather, I was imagining I face lots of different situations. So long as my probabilities are calibrated, the long-run frequency argument works out the same way. Kelly looks optimal. So what’s your beef with me going “full Kelly” on those estimates?

OTHER: In those terms, I’m examining the case where probabilities aren’t calibrated.

BAYESIAN: That’s not so hard to fix, though. I can make a calibration graph of my long-term performance. I can try to adjust my probability estimates based on that. If my 70% probability events tend to come back true 60% of the time, I adjust for that in the future. I’ve done this. You’ve done this.

OTHER: Do you really think your estimates are calibrated, now?

BAYESIAN: Not precisely, but I could put more work into it if I wanted to. Is this your crux? Would you be happy for me to go Full Kelly if I could show you a perfect x=y line on my calibration graph? Are you saying you can calculate the value for my fractional Kelly strategy from my calibration graph?

OTHER: … maybe? I’d have to think about how to do the calculation. But look, even if you’re perfectly calibrated in terms of past data, you might be caught off guard by a sudden change in the state of affairs.

BAYESIAN: Hm. So let’s grant that there’s uncertainty in my calibration graph. Are you saying it’s not my current point-estimate of my calibration that matters, but rather, my uncertainty about my calibration?

OTHER: I fear we’re getting overly meta. I do think should be lower the more uncertain you are about your calibration you are, in addition to lower the lower your point-estimate calibration is. But let’s get a bit more concrete. Look at the graph. I’m showing that you can expect better returns with lower in this scenario. Is that not compelling?

BAYESIAN (who at this point regresses to just being Abram again): See, that’s my problem. I don’t understand the graph. I’m kind of stuck thinking that it represents someone with their hands tied behind their back, like they can’t perform a Bayes update to improve their estimate , or they can’t change their after the start, or something.

• Ah, yeah, I agree with your story.

Before the data comes in, the conspiracy theorist may not have a lot of predictions, or may have a lot of wrong predictions.

After the data comes in, though, the conspiracy theorist will have all sorts of stories about why the data fits perfectly with their theory.

My intention in what you quote was to consider the conspiracy theory in its fulness, after it’s been all fleshed out. This is usually the version of conspiracy theories I see.

That second version of the theory will be very likely, but have a very low prior probability. And when someone finds a conspiracy theory like that convincing, part of what’s going on may be that they confuse likelihood and probability. “It all makes sense! All the details fit!”

Whereas the original conspiracy theorist is making a very different kind of mistake.

• Agreed. The problem is with AI designs which don’t do that. It seems to me like this perspective is quite rare. For example, my post Policy Alignment was about something similar to this, but I got a ton of pushback in the comments—it seems to me like a lot of people really think the AI should use better AI concepts, not human concepts. At least they did back in 2018.

As you mention, this is partly due to overly reductionist world-views. If tables/​happiness aren’t reductively real, the fact that the AI is using those concepts is evidence that it’s dumb/​insane, right?

Illustrative excerpt from a comment there:

From an “engineering perspective”, if I was forced to choose something right now, it would be an AI “optimizing human utility according to AI beliefs” but asking for clarification when such choice diverges too much from the “policy-approval”.

Probably most of the problem was that my post didn’t frame things that well—I was mainly talking in terms of “beliefs”, rather than emphasizing ontology, which makes it easy to imagine AI beliefs are about the same concepts but just more accurate. John’s description of the pointers problem might be enough to re-frame things to the point where “you need to start from human concepts, and improve them in ways humans endorse” is bordering on obvious.

(Plus I arguably was too focused on giving a specific mathematical proposal rather than the general idea.)

• Welcome to LessWrong! I think the other commenters are being a bit harsh to a first-time poster ;p though perhaps you wouldn’t have it any other way, given your subject matter. So I just want to say that I agree with your overall message; lots of university courses have poor epistemic standards. But then, lots of society has poor epistemic standards, so is this a real surprise?

Additional comment: this view would be a bit more controversial among lesswrong users, but I believe that this also applies to some math courses. Everyone should be taught formal logic, budgeting, probability, and data analysis (and other things), but for most people calculus and matrix math really isn’t needed.

I agree wrt prioritization of math topics, but this is a lot different from the topic of most of your post, which is accuracy. I don’t find this kind of not-commonly-useful argument to be much of a strike against a class. Because if you do take those classes, then you can do those technical jobs (at least, if you get far enough).

Too often it’s the kid in math class complaining “this will never be useful to me”. It’s too easy of an out. If you were training an RL system, you would just put it straight to work and have it do the actual thing it needs to learn thousands of times. But humans get a lot of benefit from broad knowledge, including preparedness for unexpected situations, and having analogies to work from. If you want to do really innovative work, it pays to learn a lot of individually “useless” stuff. (If you don’t want to do really innovative work, OK, sure.)

If someone were approaching school with an attitude of getting as much out of it as possible, it seems like they’d want to identify skills to level up (EG they might identify art history as not being a skill to level up), and then grind those skills as much as they can. Any of those proficiencies, once you grind enough, becomes a marketable skill. (Even if you make an economically poor choice, like law or art, there will be ways to succeed, and potential applications beyond getting a job. Not that I’m recommending making economically poor choices.)

It seems better to have an attitude of collecting as many of those as possible, rather than as little as possible. So long as you’re at school anyway.

• I wondered if someone would bring this up! Yes, some people take this as a strong argument that utilities simply have to be bounded in order to be well-defined at all. AFAIK this is just called a “bounded utility function”. Many of the standard representation theorems also imply that utility is bounded; this simply isn’t mentioned as often as other properties of utility.

However, I am not one of the people who takes this view. It’s perfectly consistent to define preferences which must be treated as unbounded utility. In doing so, we also have to specify our preferences about infinite lotteries. The divergent sum doesn’t make this impossible; instead, what it does is allow us to take many different possible values (within some consistency constraints). So for example, a lottery with 50% probability of +1 util, 25% of −3, 1/​8th chance of +9, 116 −27, etc can be assigned any expected value whatsoever. Its evaluation is subjective! So in this framework, preferences encode more information than just a utility for each possible world; we can’t calculate all the expected values just from that. We also have to know how the agent subjectively values infinite lotteries. But this is fine!

How that works is a bit technical and I don’t want to get into it atm. From a mathematical perspective, it’s pretty “standard/​obvious” stuff (for a graduate-level mathematician, anyway). But I don’t think many professional philosophers have picked up on this? The literature on Infinite Ethics seems mostly ignorant of it?