These “meta-values” you mention are just values applied to appraisal of values. So in these terms it’s possible to have values about meta-values and to value change in meta-values. Value drift becomes instrumentally undesirable with greater power over the things you value, and this argument against value drift is not particularly sensitive to your values (or “meta-values”). Even if you prefer for your values to change according to their “natural evolution”, it’s still useful for them not to change. For change in values to be a good decision, you need to see value drift as more terminally valuable than its opportunity cost (decrease in value of the future according to your present values, in the event that your present values undergo value drift).
Vladimir_Nesov
The strongest argument against value drift (meaning the kind of change in current values that involves change in idealized values) is instrumental usefulness of future values that pursue idealized present values. This says nothing about terminal value of value drift, and a priori we should expect that people hold presence of value drift as a terminal value, because there is no reason for the haphazard human values to single out the possibility of zero value drift as most valuable. Value drift is just another thing that happens in the world, like kittens. Of course valuable value drift must observe proper form even as it breaks idealized values, since most changes are not improvements.
The instrumental argument is not that strong when your own personal future values don’t happen to control the world. So the argument still applies to AIs that have significant influence over what happens in the future, but not to ordinary people, especially not to people whose values are not particularly unusual.
Thanks!
Sure, any resource would do (like copper coins needed to pay for maintenance of equipment), or just generic “units of resources”. My concern is that the term “utility” is used incorrectly in an otherwise excellent post that is tangentially related to the topic where its technical meaning matters, potentially propagating this popular misreading.
Utility is not a resource. In the usual expected utility setting, utility functions don’t care about affine transformations (all decisions remain the same), for example decreasing all utilities by 1000 doesn’t change anything, and there is no significance to utility being positive vs. negative. So requirements to have at least five units of utility in order to play a round of a game shouldn’t make sense.
In this post, there is a resource whose utility could maybe possibly be given by the identity function. But in that case it could also be given by the “times two minus 87” utility function and lead to the same decisions. It’s not really clear that utility is identity, since a 50⁄50 lottery between ending up with zero or ten units of the resource seems much worse than the certainty of obtaining five units. (“If you’re Elliott, [zero] is a super scary result to imagine.”)
To see why it is wrong in general, consider an extreme case: a universal law, which you mostly already believe to be true.
I feel that this example might create another misconception, that certainty usually begets greater certainty. So here’s an opposing example, where high confidence in a belief coexists with high probability that it’s going to get less and less certain. You are at a bus stop, and you believe that you’ll get to your destination on time, as busses here are usually reliable, though they only arrive once an hour and can deviate from schedule by several minutes. If you see a bus, you’ll probably arrive in time, but every minute that you see no bus, it’s evidence of something having gone wrong, so that the bus won’t arrive at all in the next hour. You expect a highly certain belief (that you’ll arrive on time) to decrease (a little bit), which is balanced by an unlikely alternative (for each given minute of waiting) of it going in the direction of greater certainty (if the bus does arrive within that minute).
(See the edit in the grandparent, my initial interpretation of the post was wrong.)
With bounded attention, noticing less relevant things more puts you into a worse position to be aware of the worlds you actually live in.
Edit: Steven’s comment gives a more central interpretation of the post, namely as a warning against the bailey/motte pair of effectively disbelieving something and defending that state of mind by claiming it’s irrelevant. (I think the motte is also invalid, it’s fine to be agnostic about things, even relevant ones. If hypothetically irrelevance argues against belief, it argues against disbelief as well.)
If it assigns equal probability to everything then nothing can be evidence in favor of it
Nope. If it assigns more probability to an observation than another hypothesis does (“It’s going to be raining tomorrow! Because AGI!), then the observation is evidence for it and against the other hypothesis. (Of course given how the actual world looks, anything that could be called “assigns equal probability to everything”, whatever that means, is going to quickly lose to any sensible model of the world.)
That said, I think being reasoned about instead of simulated really does have “infinite explanatory power”, in the sense that you can’t locate yourself in the world that does that based on an observation, since all observations are relevant to most situations where you are being reasoned about. So assigning probability to individual (categories of) observations is only (somewhat) possible for the instances of yourself that are simulated or exist natively in physics, not for instances that are reasoned about.
I agree that it’s possible for feelings to be relevant (or for factual beliefs to be relevant). But discouragement of discussion shouldn’t be enacted through feelings, feelings should just be info that prompts further activity, which might have nothing to do with discouragement of discussion. So there is no issue with Vanessa’s first comment and parts of the rest of the discussion that clarified the situation. A lot of the rest of it though wasn’t constructive in building any sort of valid argument that rests on the foundation of info about feelings.
“Shut up and calculate” no longer suffices when you want to figure out something about reality that is not about prediction of observations (or if you are interested in unusual kinds of reality, where even prediction of observations looks unlike it does in our world). So this concerns many philosophical questions, in particular decision theory (where you want to figure out what to do and how to think about what to do). The relationship with decision theory is the same as with physics: you want to replace reality with something more specific. But if you haven’t found a sufficiently good replacement, forcing a bad replacement is worse than fumbling with the preformal idea of reality.
“Shut up and calculate” serves as an excellent replacement for truth in most situations. It’s about hypotheses/theories and observations, not reality, so within its area of applicability it makes the idea of reality irrelevant.
Ideas, beliefs, feelings, norms, and preferences can be elucidated/reformulated, ascertained/refuted, endorsed/eschewed, reinforced/depreciated, or specialized/coordinated, not just respectively but in many combinations. Typically all these distinctions are mixed together. Naming them with words whose meaning is already common knowledge might be necessary to cheaply draw attention to relevant distinctions. Otherwise it takes too long to frame the thought, at which point most people have already left the conversation.
I don’t get the impression Said was describing or channeling a particular culture. Rather, he was making specific observations that are primarily concerned with what they actually refer to. There is a difference between the hypothesis that such claims implicitly suggest a general point about some guiding principles that the author endorses, and the hypothesis that their style pattern-matches such a general point and thus perpetuates norms that you consider undesirable.
In the context where the resulting norms are undesirable, the impact of these options is the same, but arguments that are valid for them are different, which is important for nudging the style of discussion in the direction you hope for. To prevent a norm from taking hold, it’s crucial to understand what the norm is, so that you won’t unintentionally fuel it.
no, this isn’t the default culture people should be expecting to be consensus on LW
I don’t just agree with many of the things Said said in the comments here, but see them as undoubtedly correct, while other points are more debatable, so I second his question about specifics of this nefarious “this”.
The thesis seems to be to caution against automatically blaming those who might’ve benefited from a disaster, as often enough things don’t happen in a goal-directed fashion. (“To fathom a strange plot, one technique is to look at what ended up happening, assume it was the intended result, and ask who benefited.”) Not sure this is a widespread enough heuristic to be worth reining in in the form of an unconditional advice.
Cultivate [...] that on the margin you are always going to underestimate the value of long term investment in habits and virtue cultivation
Why though? Shouldn’t you recalibrate immediately to make this no longer predictable? Or is such recalibration the meaning of the quoted sentence? In that case, why phrase it so, it seems to risk overcorrection, not noticing when the opposite advice becomes relevant, or else requires undue caution in following your own advice, at which point it becomes a self-fulfilling flaw/advice combo? (Following the advice cautiously ensures that the flaw is not fully removed, and so the advice remains relevant.)
I agree, it’s just not a primary thing that’s happening, the coercion of the discipline (conflict between values and behavior) is more prominent than reinforcement of values where the discipline becomes necessary (partially by definition, since if it works well, the discipline is not necessary after all). For this reason, it’s misleading to characterise the effect of this policy as reinforcement of current values, though that probably happens as well. Not sure how that’s balanced by rebellious urges.
(I disagree with my statements above in the thread in the context where preventing value drift is much more important than preventing suffering from coercion of behavior to unaligned values.)
I would think that applying schelling fences to reinforce current values reduces the amount of expected drift in the future
It reinforces the position endorsed by current values, not the current values themselves. (I’m not saying this about Schelling fences in general, which have their uses, rather about leveraging of status quo and commitment norms via reliable application of simple rules, chosen to signal current (past, idealized) values.) This hurts people with future changed values without preventing the change in values.
what specifically you think is making the error of making it difficult to re-align with current values
The effect on prevention of change in values is negative only in the sense of opportunity cost and because of the possibility of confusing this activity for something useful, which inhibits seeking something actually useful. It’s analogous to the issues caused by homeopathy. (Though I’m skeptical about value drift being harmful for humans.)
Here’s the link, found from my account by searching for “collaborative truthseeking”. There is a “Posts from Anyone/You/Your Friends” radio control on the left of the search page, so should probably work on your own posts as well.