G Gordon Worley III
Karma: 2,572
NewTop
[Question] Why should EA care about rationality (and vice-versa)?
[Question] What precisely do we mean by AI alignment?
I want something to indicate post length and sometimes use this to make decisions about what to read. Read time is fine, word count would also be fine, something else might also be fine.
Likely no coherent sense could be made of that ideal.
Indeed, this is exactly the problem of the criterion.
In terms of decision cycles, I’m a fan of Boyd’s OODA (Observe-Orient-Decide-Act), or my preferred respelling of it as NODA to replace Observe with Notice so you get a different letter for each step: I think it does a good job of explaining what’s going on in my head when I’m dealing with things, and it seems to generalize well over different timescales.
Right, the whole things seems like a rather strange confusion to me, since the is-ought gap is a problem certain kinds of anti-realists face but is not a problem for most realists since for them morals facts are still facts. So it seems to me, not being familiar with Harris, an alternative interpretation is that Harris is a moral realist and so believes there is no is-ought gap and thus this business with dialectical v. logical explanations is superfluous.
So I don’t exactly disagree with your conclusion or the points you make, but I feel like there is something in this framing that sets up the possibility of creating confusion down the line. It’s a little hard to pin down where this is coming from, but I’ll try anyway by looking at this paragraph that I think may illustrate the crux:
A definition of chair is wrong if it doesn’t approximate the boundaries of the mental set of examples of chairs that we have in our minds. If your definition answers “yes” to the question “is your mobile phone a chair if you sit on it?“, then this is a wrong/bad/incorrect definition.
So there’s a certain perspective from which your framing of a-definition-of-chair-is-wrong-if-it-includes-mobile-phones makes sense because it’s not useful to the purpose of asking “can I sit on it such that I’m happier about sitting?” But this is a deeply teleological approach and as such depends thoroughly on the worldview from which chairness is being assessed. I think that’s fair as far as chairness goes and is ultimately fair about any question of ontology we might ask, since from where else can be construct ontology but from where we are perceiving the world? But I also sense a privileging of a particular worldview, namely a human one, that may artificially limit the sorts of useful categories we are willing to consider. After all, to a biped the size of a hamster a mobile phone might make an excellent chair.
I’m interesting to see where you go with this, but having not read the subsequent posts I’ll register now my suspicion that there is some deep philosophical disagreement I’m going to have with where you end up although I can’t quite put my finger on what it is right now, but I suspect it’s something like making assumptions that are pragmatically reasonable under normal circumstances but don’t hold when considering minds in general.
I like this advice as far as it goes, but I’ll just point out you seem to be talking about a folk version of stoicism that is not necessarily the same thing as stoicism proper, since that philosophy is more precise than “don’t worry too much” or something similar. You can have replaced “stoicism” with “zen” and this would have read the same I think and I would have had the same comment. There is probably some other, better word to describe the category of things you’re pointing at that doesn’t have a more specific meaning.
Sorry you got down voted so much; I think this is a pretty solid introduction to the topic (although I find the presentation style weird for my own tastes, but I recognize it as following a pattern that others like a lot). LW is unfortunately pretty hostile to continental philosophy (and a little hostile to philosophy in general, which is strange for a place that cares so much about epistemology) for reasons I perceive to be mostly tribal, although the most vociferous commenters against those topics would disagree and give reasons why continental philosophy is contrary to LW-style rationality. I think that’s a shame because you hit on some points that I think folks around here would be well served to explore more, like irreducibility and the problems for science caused by its embeddedness in itself and inability to adequately consider its own methods.
I liked a lot of this an appreciate the format.
One of the most pernicious issues I’ve encountered in these discussions is the conversation that goes “hey, look at this cool thing that rationality isn’t very good at; I’m pretty sure we can do better” and the response is usually “so what, that’s already part of rationality, and any failure to achieve that is due to people being still aspiring rationalists”. I’m not sure its possible to convey in words why from the post-rationalist side this looks like a mistake, because from the other side it looks like trying to cut off a part of the art that you as a rationalist already have access to. All I can offer as way of evidence that this is a reasonable view from the post-rationalist side is that people that end up in the post-rationalist camp generally have a story of like “oh, yeah, I used to think that, but then this one thing flipped my thinking, everything clicked, and I was surprised at how dramatically little I understood or was capable of before” and this is happening to people who were already trained as rationalists.
Edit: Plausibly what I’m describing here is what you call a “degenerate case of nurture that is just about nice and polite” but I think there’s a lot more to it than common notions of niceness and politeness. 1) In the ideal case, it’s motivated by real caring, not social convention. 2) It’s more demanding than mere pleases and thank yous.
I’d agree with that: I think what you are pointing at with nurture culture has to be much deeper and richer than you are saying before it’s useful for more than just people getting along nicely while doing other things. Nurture culture is capable of the same kind of cutting, deep engagement that you are seeing with combat culture, but to do that it requires much more than shallow modeling of other people.
Nurture Culture relies on participants having the social skill to model each other’s minds to a further extent and execute more complicated social routines.[2]
This is an excellent point and one worth emphasizing. Nurture culture only really works with people capable of modeling each other in sufficient detail that they can really see the world from each others perspective. To use some jargon popular around here, they have to be able to pass each other’s intellectual Turing test.
To go a step further, I think this is tied in with psychological development. If we were to think of this in terms of Kegan, I’d predict nurture only really works in cases where either everyone is at least at the third stage AND sufficient mentally similar that they can model each other as being essentially like themselves only with a different flavor, or where everyone is at least in the fifth stage (removing the need for a qualification about level of similarity). To speculate, this probably makes nurture unappealing to many people because it places demands on them about their ability to model others that they can’t fulfill, assuming we ignore here the degenerate case of nurture that is just about being nice and polite rather than actually understanding each other.
This reminds me of the distinction between debate and dialectic. Both can be means to truth seeking, both have their own failure modes (debate can become about winning instead of the truth; dialectic can become confused without adequate experience with synthesis), and different people can have a preference for one over the other. Thinking in terms of a culture though is perhaps better suited to what’s going on than talking about preference for a particular technique because it gets at something deeper fueling that preference for particular methods.
Also, for what it’s worth, I found this post useful enough at describing what I want to moderate towards that I’ve linked it now from my LW moderation guidelines to indicate I want people to prefer nurture to combat culture in the comments on my posts. I think a failure to have this well explained helps explain why things went in a direction I didn’t like in a recent contentious post I made.
So on the one hand this seems to echo Stuart Armstrong’s take on building oracle AI (paper here coauthored with Sandberg and Bostrom) where we might summarize the starting intuition as “build AI that is not an act-based agent so we avoid Goodharting in ways that pose an x-risk”. On the other, though, I remain suspicious of the idea that we can avoid dangerous Goodharting because optimizing for the measure of a variable rather than the variable itself is baked in at such a deep level that I’m inclined to think it more likely that we’ve fooled ourselves or failed to see far enough rather than overcoming Goodharting if we think that’s what we’ve found a way to do. Since you’ve just proposed an idea rather than something very specific I can’t say much more, but I think things of this class of approaches are unlikely to work, and in this case specifically my thinking caches out as predicting we’d never see this Scientist AI reach a point where we could trust it to do what we mean.
An alternative to needing to seek novelty would be to cultivate equanimity. Past me would have shared your desire to find ways to have novelty, but present me is more equanimous and so more indifferent between different conditions (not totally indifferent, of course, just not different on so many dimensions). In a scenario where people are living forever, they’d have plenty of time to stumble in to something like a spiritual practice with meditation that would help them better cope with the long forever.
Thanks for also sharing this as text (on the MIRI blog); makes it much easier to read.
I break this down in a slightly different way. I think “value” is a coherent concept, but it’s a much broader concept than we normally give it created for. I’ve sometimes used the word “axia” in place of “value” (note “axia” is singular; it’s a Greek word and just looks like a plural would in Latin; the Greek plural is “axies” but “axias” is probably acceptable in English) to avoid association with the confused specificity of “value”. So instead I go in the direction of considering anything that is the value of a thought to be sensible as an axia/value. This is a different direction than most people pick because I want to cover the entire space of value-like things rather than exclude some value-like things to get certain properties that make it easier to do math. I wrote some words about this in the final post in a multi-part series of posts I did about a year ago if you want to read more.
At our meetup people are free to meditate how they like so long as it’s silent and otherwise not distracting to others: it’s simply a space for people who want to practice meditation to do so together. Many people find it hard to keep up their practice on their own, which is why sangha is valuable. We don’t totally provide all that sangha can, but we do provide at least some like-minded folks you can be with while meditating and talk to about meditating. So yes, it’s a regularly scheduled time when people can commit to meditating and the social component helps it stick.
We don’t normally give instruction, but if people ask I just give the most rudimentary of guidance: sit comfortably so you are at ease but not totally relaxed and attentive but not on edge, then draw your attention to your breath without trying to control either your attention or your breath so you just observe what’s happening. If you get distracted that’s fine, just notice and return to the breath; you can’t really meditate wrong because you are discovering a skill for yourself, and it’s better to err on the side of saying “oh great, I noticed I was distracted; I noticed something!” than “shit, I got distracted; guess I suck at meditating”.
I practice zen so I’m familiar with the kind of instruction you received. There’s more to it than posture but the form is the only thing we really have anything to talk about, because the goal is to just sit (this is the literal translation of shikantaza, the primary form of meditation in soto zen; another would be “simply getting on with sitting”), and learning how to do that is something we don’t believe can be effectively taught but is something you can discover for yourself through experiencing stillness, and that starts with stilling your body by sitting in a particular form that encourages that.
Incidentally I think this is why zen, despite its seemingly large position in the marketplace of ideas relative to other forms of Buddhism to the point that “zen” just means something like “calm” to many people, is not a very popular practice; it’s just not the sort of thing that resonates with lots of people. I think this is why vipassana has become so popular: it’s a style that better meets most (Western) people where they’re at. But personally I don’t mesh with vipassana or many other styles well, so I stick to shikantaza because it’s what works with me. Humans have enough variation that it makes sense to me that different styles, even if they seem only subtly different, will work with different people differently, so if you don’t like one way of meditating you may like another.
“But I have this nagging sense that even in that case the approximations may not do everything we want because they are trying to optimize on features that are unachievable for real agents”—This seems to be the central plank of your case. I don’t suppose you could spec it out some more? I have some idea of what you might be thinking, but I don’t want to put words in your mouth.
Well since I was confused my reasons for concern have dissolved and timelessness now feels like aiming in the direction of what you want, although it would probably make sense to consider stepping back further such that you are not trapped by, say, assumptions about the metaphysics of the world (think, allow yourself to go up a Tegmark level). So that aside I now wonder what you had in mind when I said this, since whatever i was thinking was not relevant.
I want to push back a bit against this perspective, since it’s one of several that I think take an overly narrow view of Buddhism and is fairly widespread. Buddhism is very wide and deep with a lot of very different practices that may share a common history but are very spread out in idea space. Some forms of Buddhism are arguably about wireheading, to be fair, but many are not.
For example, my own practice with Zen falls within the Mahayana branch of Buddhism which explicitly rejects the notion that it’s appropriate to practice only to achieve nirvana (liberation from suffering) and instead that the purpose of practice is to help all suffer less. Each of us is, of course, part of “all” and so we contribute to all suffering less by liberating ourselves from suffering, and we will often act unskillfully to help others so long as we are caught by our own suffering-inducing delusions, so we must focus on ourselves enough to be able to actually help others. But this focus on self is limited by an equal valuing of all beings, thus there is no attempt to attain nirvana except in the limit when perhaps all beings could attain it together.
This doesn’t even begin to get into some of the weirder stuff that is even farther away from looking like wireheading, but I think should give enough sense of the fact that thinking that “Buddhism is a guide to wireheading” is missing out on a lot of what goes on under the label of “Buddhism”.