No, but I can see how it may be necessary. I guess I’ve been lucky that so far my interests have aligned with jobs that pay well enough for it to not be an issue—I’m sure some fields are more constrained than others. I didn’t think that this would apply to programming, though. (That’s my field, too.)
Jim Pivarski
Karma: 169
Okay, I just did a deep-dive on the AI alignment problem and the Singularity on Wikipedia, and it will take me a while to digest all of that. My first impression is that it seems like an outlandish thing to worry about, but I am going to think about it more because I can easily imagine the situation reversed.
Among the things I came across was that Eliezer was writing about this in 1996, and predicted
Plug in the numbers for current computing speeds, the current doubling time, and an estimate for the raw processing power of the human brain, and the numbers match in: 2021.
GPT-3 has some tens to hundreds of billion parameters and the human brain has 86 billion neurons, and I know it’s hand-waving because model parameters aren’t equivalent to human neurons, but—not bad! On the other hand, we’re seeing now what this numerical correspondence translates to in real life, and it’s interestingly different than what
weI had imagined. AI is passing the Turing test, but the Turing test no longer feels like a hard line in the sand; it doesn’t seem to be testing what it was intended to test.No one knows how to solve it, and it’s likely only rationalists could.
Understanding what, exactly, human values are would be a first step toward expressing it in AI. I hadn’t expected meta-ethics to get so applied.
...
You know what’s really odd? The word “singularity” appears only 33 times in the Sequences, mostly as “when I attended the Singularity Summit, someone said...” and such, without explanation of what it was. Most of the references were in the autobiographical section, which I didn’t read as deeply as the rest.
This is a good example of needing to watch my words: the same sentence, interpreted from the point of view of no-free-will, could mean the complex function of biochemical determinism playing out, resulting in what the human organism actually does.
What I meant was the utility function of consequentialism: for each possible goal , you have some preference of how good that is , and so what you’re trying to do is to maximize over . It’s presupposing that you have some ability to choose instead of , although there are some compatibilist views of free will and determinism that blur the line.
My point in that paragraph, though, is that you might have a perfectly rational machinery for optimizing , but one has to also choose . The way you choose can’t be by optimizing over . The reasons one has for choosing also can’t be directly derived from scientific observations about the physical world, because (paraphrasing David Hume), an “is” does not imply an “ought.” So the way we choose , whatever that is, requires some kind of argumentation or feeling that is not derivable from the scientific method or Bayes’ theorem.
Okay, I take the word “should” to refer to a spectrum with ethics on one end (strong “should”) and aesthetics on the other (weak “should”). It’s possible that this is a wider use of the word “ethics” or “aesthetics” as others would have. Maybe those other things people are thinking about don’t lie on a linear spectrum?
So, for example, when you’re doing an algebra problem, “you should subtract the same amount from both sides of the equation, not just one side,” is a choice to stay within the rules of algebra. Not doing so leads to less interesting results (everything being “equal” to everything else). I’m not sure whether that’s closer to the ethics end or the aesthetics end; maybe it depends on whether the math is pure or applied.
But getting back to my meaning in that paragraph, the Sequences had a large section on ethics, and choosing to be rational comes through the text as a strong imperative. And then, having opinions on a “should” statement (whether you call it “ethics” or not) comes from beyond-experimental reasoning, because (by David Hume), an “is” does not imply a “should.”
I wasn’t intending to take a side in utilitarianism/consequentialism; I just meant that, ultimately, a decision is made from intuition. It can’t be deductive all the way down.
If you replace “true” with “accurate,” what does “accurate” mean?
I would have thought that “accurate” means that the distance between the model result and the true result is small, so it contains a notion of truth and a notion of distance.
I had seen Tegmark’s four multiverses before, but relating it to this is something I hadn’t considered. The first level of multiverses, with different initial conditions but the same laws, is a lot like how I understand Pearl’s ensemble of possible worlds.
As for the universe being a mathematical structure, that seems to be pretty much what physicists assume by default. The formulations of string theory that I’ve seen (casually—I’ve never worked in that area) replace the space-time manifold with a non-geometric matrix, making it math all the way down.
Even if it is math all the way down, I would consider physical reality to be a different thing from mathematical reality because the physical fact is which mathematical structure is actual. The same way that Chalmers calls consciousness a “further fact” above and beyond material brains, the fact of which mathematical structure is the one that is our world, as opposed to just being logically possible, is a further fact above and beyond the mathematical possibilities.
You might not be able to wiggle it enough to get what you want (depending on what you want), but you need to be able to wiggle it, yes.
Given the way “A causes B” is used in everyday speech and among philosophers, it seems that it needs to have some notion of “if A, then B” (with a possible addition of “is more likely”). To use “casual” to mean “deterministic” would be confusing—different enough, anyway, that this usage needs to be called out as unrelated, to avoid confusion. “Smoking causes cancer” is not deterministic (some smokers are lucky) and includes the idea that not smoking is a possibility.
I know there’s the wildly different usage of “to cause” in relativity to mean “is before, with time-like separation,” and that is also terribly confusing. It should be called out more.
And even more deeply than door-to-door conversations, political and religious beliefs spread through long-term friend and romantic relationships, even unintentionally.
I can attest to this first-hand because I converted from atheism to Catholicism (25 years ago) by the unintended example of my girlfriend-then-wife, and then I saw the pattern repeat as a volunteer in RCIA, an education program for people who have decided to become Catholic (during the months before confirmation), and pre-Cana, another program for couples who plan to be married in the church (also months-long). The pattern in which a romantic relationship among different-religion (including no-religion) couples eventually ends up with one or the other converting is extremely common. I’d say that maybe 90% of the people in RCIA had a Catholic significant other, and maybe 40% of the couples in pre-Cana were mixed couples that became both-Catholic. What this vantage point didn’t show me was the fraction in which the Catholic member of the couple converted away or maybe just got less involved and decided against being married Catholic (and therefore no pre-Cana). I assume that happens approximately as often. But it still shows that being friends or more than friends is an extremely strong motivator for changing one’s views, whichever direction it goes.
Since it happened to me personally, the key thing in my case was that I didn’t start with a clear idea of what Catholics (or some Catholics, anyway) actually believe. In reading this article and the ones linked from it, I came to Talking Snakes: A Cautionary Tale, which illustrates the point very well: scottalexander quoted Bill Maher as saying that Christians believe that sin was caused by a talking snake, and scottalexander himself got into a conversation with a Muslim in Cairo who thought he believed that monkeys turned into humans. Both are wild caricatures of what someone else believes, or at least a way of phrasing it that leads to the wrong mental image. In other words, miscommunication. What I found when I spent a lot of time with a Catholic—who wasn’t trying to convert me—was that what some Catholics (can’t attest for all of them) meant by the statements in their creed isn’t at all the ridiculous things that written creed could be made to sound like.
In general, that point of view is the one Yudkowsky dismissed in Outside the Laboratory, which is to say that physical and religious statements are in different reality-boxes, but he dismissed it out of hand. Maybe there are large groups of people who interpret religious statements the same way they interpret the front page of the newspaper, but it would take a long-term relationship, with continuous communication, to even find out if that is true, for a specific individual. They might say that they’re biblical literalists on the web or fill out surveys that way, but what someone means by their words can be very surprising. (Which is to say, philosophy is hard.) Incidentally, another group I was involved in, a Faith and Reason study group in which all of the members were grad students in the physical sciences, couldn’t even find anyone who believed in religious claims that countered physical facts. Our social networks didn’t include any.
Long-term, empathic communication trades the birds-eye view of surveys for narrow depth. Surely, the people I’ve come in contact with are not representative of the whole, but they’re not crazy, either.
I think it would be more correct to say that a focus on believing particular assertions is a fairly recent trend in religion, encompassing the past millennium or two, but really picking up in the last few centuries.
It happened in or between Christianity and Islam (as isusr points out), and they probably both influenced each other. For example, Protestant Christianity focuses a lot more on a holy book than Catholicism and Orthodox Christianity, but in a way that resembles Islam’s veneration of the Quran: citing verses to prove points. Since then, Catholics and Orthodox have also stepped up their focus on the Bible. There’s a lot of cross-pollination.
In the last century or so, religious statements have been presented as a kind of alternate-science (e.g. Young Earth science), presumably to respond to an apparent threat, but this is a very new way of taking about religion. There were biblical literalists (and non-literalists) throughout Christian history, but ancient theologians would probably accuse these people of missing the point.
Meanwhile, religions with only recent sustained contract with Christianity and Islam (past half-millennium) and religions that preceded them focus a lot more on practice, i.e. ritual and social behaviors. Some belief is implicit (e.g. why leave offerings for gods or ancestors if you don’t think they exist in a form that would benefit from the offerings?), but they are much less the focus.
I’m still Catholic. I was answering your question and it got long, so I moved it to a post: Answer to a question: what do I think about God’s communication patterns?
I was answering a question, and I took it out of the comments stream to keep from derailing that stream. But perhaps doing that made it more visible, which is not what I had intended.
It seems that religion is just an off-limits topic here, regardless of whether it seems like it might be relevant in some conversation. At the very least, I’ll never bring it up again.
As a statement about “What is the most fundamental type of thing?”, it can’t be justified by deduction or an appeal to observation the way that a statement about a particular physical thing in the universe can be. Like, if you want to find out why phosphorescent rocks glow, you can look at them under microscopes, experiment with them chemically, apply theories of molecular structure taken from other observations, etc. But if you ask things like, “Do phosphorescent rocks glow for any reason at all?” “Is nature comprehensible?” “Does it all resolve to mathematics at root, or something else?”, those sorts of questions are not answerable by deduction and/or observation because they’re about the applicability of deduction and/or observation. We—our culture, over the past few hundred years—chose to make the unexplained leap to think of matter in mathematical/scientific ways, affording it no agency of its own, no “vis viva,” no “it just wants to,” “there’s a nature spirit,” etc. I believe that was a good choice (I’m a physicist, after all); I think that the insight the mathematical and physical sciences appears to provide (which is a feeling) is connected to a real, objective world on the other side of our perceptions. But we can’t call that leap rational.
In the above, I was talking about another assumption one can make about Everything (is “cosmos” a good word?), more generally than the physical universe itself. The idea that Everything exists at all for a thought-like reason—intention—is not more or less justified than for no reason, or out of logical necessity (what would that logical necessity argument be, and why is the argument itself necessary?), or various other things philosophers have come up with. It’s the subject matter that makes this undecidable, the fact that we’re trying to point to Everything. Any position you could have on that is a leap.
Sometimes, when person A leaps one way and person B leaps another way, A thinks B just hasn’t understood or conceptualized A’s position. That isn’t necessarily the case. I was fully willing to see the cosmos as a pointless absurdity (I had read a lot of Camus), and when I saw that my wife’s idea that the cosmos was intended by a mind makes sense, too, I thought, “I can go with this.” In fact, it seemed to solve a problem, providing objectivity to ethics, why we should do any specific thing instead of some other specific thing.
As I said later, I’m now less enthusiastic about the “opinions of the cosmos → objective ethics” argument because it doesn’t connect with how we can know that a particular action is good or bad: we’re still going with our feelings. Knowing that there is a correct choice but not knowing if our feelings align with that correct choice is not helpful. By analogy, I could posit the existence of a particle in the physical universe that has exactly zero influence on any of the particles that give us sense-perceptions. Maybe it’s true that that particle exists, but it’s not useful for the theory.
So yes: it’s an unexplained leap, this is subject matter for which nothing is possible except unexplained leaps, I brought it in because I thought it could give an account of actions being good or bad, but in the end, it effectively doesn’t.
Right, as far as I know. Years ago, someone confided in me (I’m not saying who) that they experienced a perceptual miracle: something they could see that no one else could see. That kind of miracle is consistent with physical facts—it’s a subjective decision to conclude whether the vision is meaningful or not.
I once asked at that Faith and Reason study group “if angels have backs.” That is, if Mary is seeing Gabriel talking to her, whether only as much as she needs to see is manifested—namely, the front. Everyone else was of the opinion that this was beside the point, as in it doesn’t really matter. I conceded that we’d never know, anyway.
Maybe I’m overstating this lack of belief in miracles: nearly all Christians, liberal Catholics included, believe that Jesus rose from the dead, which you’d think is a physical event—blood flowing again, muscles moving, etc. On the other hand, it’s an odd sort of body to be appearing and disappearing, passing through locked doors, being unrecognized or recognized at will, etc. I’ve never understood how eating fish is supposed to prove that he’s not a ghost when he passed through a locked door in the same scene. This doesn’t go against physical fact, it just confuses the issue for me. I don’t know quite what is meant by “alive again.”
And to really clinch the oddness of the language, nearly all Catholics, liberal Catholics included, believe that the Eucharist “physically” becomes the body and blood of Jesus, while also being indistinguishable from bread at all physical levels—they will say that microscopes and chemical analysis would reveal gluten, not muscle and red blood cells. These can’t be statements in the same reality-box, the first use of “physically” (quoted above) must be in an interpretive sense like the sense in which I interpret the material tubes hanging off my hands as “my” fingers. That’s a choice of interpretation that can neither be confirmed nor denied by any experimental measurements—experimental measurements only conclude that these fleshy tubes are made of cells and bones and such, I’m the one who calls it part of my body. That’s what I’m assuming my co-religionists believe about the Eucharist because the direct statement is not contrary with fact, it’s contrary to logic.
Actually, in all cases when I ask about this, the response I get is along the lines of “that’s beside the point,” i.e. the metaphysics of miracles isn’t a subject of interest. It is the case that my community is equally unconcerned with claims for physical miracles, like real blood coming out of a statue-Mary’s eyes or something. There are other groups who are very focused on things like that, and this isn’t one of them.
Have you ever been stuck debugging code and made a breakthrough by explaining it to someone else, even if they weren’t following what you’re saying? I think that’s happened here, so thanks for asking me the question!
The religious community I’m in is not keen on proving the physical reality of miracles, the way that some will put a lot of effort into explaining, for instance, how and why the sun stayed still in the sky when Joshua prayed for it. (My community would quickly call something like that mythological.) The miracles that my community does assert—I was wrong when I said that they don’t assert any miracles—are not an affront to physical evidence, they’re an affront to logic.
Saying “affront to logic” makes it sound bad, but these are statements that are not supposed to be logical—that’s not their social purpose. (Wrong “language game,” as Wittgenstein put it.) The positions taken on Jesus’s resurrection and the Eucharist, as described above, are not illogical but antilogical: they’re constructed in such a way as to make analysis impossible, on purpose. We didn’t just not notice that we’re saying “the Eucharist is physically body and blood” and also “materially, it’s bread,” which is an obvious contradiction, in the same way that “Tell me you’re X without telling me you’re X,” is an impossible imperative—it can’t be the same “tell” in both cases.
I’ve previously noticed this about the Trinity. Most of the early heresies were trinitarian, and it was the most reasonable-sounding theories that were rejected as heresies. One mainstream statement is, “The Father is not the Son, the Son is not the Spirit, and the Spirit is not the Father, but the Father is God, the Son is God, and the Spirit is God.” That deliberately breaks the transitivity of the word “is,” so it’s not an equivalence relation. If you ask someone about the logic of that, they’ll remind you it’s a mystery, which puts it in a category of things that are not allowed to be figured out; they’re intended for contemplation.
This is sounding to me like a koan. Zen koans are also supposed to be contemplated but not solved. In mainstream Buddhism (broader than Zen), I came across this astonishing statement, that none of the following are true about the Buddha after his death:
The Buddha exists.
The Buddha does not exist.
The Buddha exists and does not exist.
The Buddha neither exists nor does not exist.
Just as the trinitarian formulation breaks transitivity, the above breaks the law of excluded middle. Or maybe not “breaks,” since it’s possible to have logical systems in which a relation is not transitive and the law of excluded middle is not used in proofs, but the people who think about these things are not in a hurry to replace them with formulations that are logically sound. That’s clearly not the point.
So it’s antilogical, which renders moot the question of whether it’s consistent with physical reality. That’s why my initial impression was, “No, they/we don’t believe in physical miracles.”
A quibble: arguments against God in the gaps are arguments against God as an explanation of some physical phenomena. “Does the universe have a face?” (poetically speaking) is not a gap that could ever be discovered by experiment.
As you (and Yudkowsky, and eventually Hofstadter) rightly point out, there isn’t a universally compelling foundation to logic or reasons for things in general. In Reality and reality-boxes, I called the unifying feature among the uses of the word “reality” as a “degree of undeniableness,” since anything can be flat-out denied, it’s just harder to do so with some things than with others.
That all is fine when we’re talking about metaphysics that doesn’t connect with any physical measurements, but what I’d really like to know is how—without grounding—we can determine what to do. That is, how to conclude, even for one’s self, that one action is wrong and another is right (which happens every time we do even the most trivial of actions).
That’s why I was interested in the universe having a face, for the cosmos to have opinions about human actions. I’ve said elsewhere in this comment thread that I’m not very keen on that argument anymore because the mere existence—posited existence—of true good and true bad doesn’t help when our only access to ideas of good and bad are our feelings. It seems that when we’re deciding to act, we’re only pitting one set of feelings (e.g. social duty) against another (e.g. personal desires). I seem to be back to Emotivism when it comes to meta-ethics and I’m wondering if there’s a way to be convinced otherwise.
Thank you for this response! (I have a few more books to add to my reading list.) Your post from 13 years ago is a very good explanation, too.
Ironically, though:
Here’s an experiment for everyone to try: think it good to eat babies. Don’t merely imagine thinking that: actually think it.
I have heard of an indigenous Australian tradition in which children were carefully, reverently turned into a blood-soup and consumed by the community (read in a book years ago, but there’s this online). And I do try to imagine what it’s like to live in this way. (I don’t think they considered it a normal, everyday thing to eat babies, but that the emotional shock had a power that could perhaps be used as a kind of magic.)
But I get your point; it’s like what I’ve been calling “degree of undeniableness.” (Budziszewki compares it to 2 + 2 = 4 and you compare it to observing that a red thing is red: logical deduction and physical observation can be denied, but it’s difficult to do so.) It’s very hard for me to agree that it’s good to eat babies. Even in the above-mentioned culture, I think it might have been a struggle, an aspect of society that was tossed as soon as they saw other ways of living. Maybe it’s not so much about what human attitudes exist—which covers a lot of extremes—as what’s easy to maintain and what gets tossed as soon as it’s recognized as not necessary.
(It’s not lost on me that the previous paragraph applies to all attitudes, not just ethics, but also smiling universes.)
I understand the softness of categories, and I don’t mind that you would use the available data to not put me in the Christian box. Some things that you don’t see are that I engage in Catholic practices, like going to mass (which is precisely why I canned an earlier draft and I’m writing again now).
If I gave the impression that Jesus is an iteration in general improvement of morality, then I mischaracterized my belief and my community’s: we believe that Jesus is God—whatever that means. I have to add the “whatever that means” because it seems like a doctrine that deliberately confounds logic, like the bit about Buddha here, when paired with Christians’ transcendent notion of God. If we thought of gods as giants who lived on Mt. Olympus, then one of them becoming physical like Zeus-the-swan wouldn’t be a problem, but we go out of our way to describe God as being more like Plato’s Zeus, which is everything that a limited, embodied, human being isn’t. Catholics emphasize saints as evidence of continuing improvement, and the apostles are often portrayed as not understanding what was happening, but Jesus (and Mary) are untouchable.
On the other hand, I look at stories like Matthew 15:27, in which a Canaanite woman appears to teach Jesus about tolerance—at the beginning of the story, it seems like he didn’t know. Most people I talk to say that it was like Socratic questioning—he really did know—but maybe the divine part of him is that he caught on and accepted the correction? While God-as-hypostatic perfection can’t learn and improve (being outside of time), God-as-a-human being can and this is what it looks like? That sort of consideration is in the “whatever that means” phrase I used above.
Okay, now on the point about not mentioning heaven: not many people that I know do. Whereas I had to clarify that we follow the Jesus-is-God doctrine—quite heavily, it’s a frequent topic—I usually only hear about heaven at funerals. While I’m sure that the people around me believe in it as “consciousness does not extinguish at death,” the subject of heaven and hell come with a heavy dose of “this terminology/imagery is metaphorical.” They’d be quick to point out that heaven (and hell) is not a “place” and I think some popes have made comments about hell being a state and not a place. (In particular, I remember one from the 90′s, but that would be a few popes ago.)
You’re right that the 2 Thessalonians letter sounds like demons eating your guts, and anything in the canonical set of books is considered as writing inspired by God (with or without their authors’ understanding)—it doesn’t matter that the writer claims to be Paul and might have not been Paul. (Attributing works to your group’s founder seems to have been more common in the ancient world. I think it’s not controversial that there were three “Isaiahs.”)
Two things about that, though: Catholics don’t put equal weight on everything in the Bible—they’re all above a certain threshold of importance, but not equally important—and there’s no actual fire and brimstone imagery in it, mostly just about being “shut out,” the kind of imagery that Jesus used, for instance, in the wise and foolish virgins parable (Matthew 25). Meanwhile, a lot of material that didn’t make it into the Bible but was influential in the early church did have more viscerally imagined rewards and punishments after death. It was an idea that was in the air at the time in Judaism (except for the Sadducees), and only got a little bit into the canonical Bible.
So, clearly, Catholics would hold that there’s some kind of good and bad afterlife, and most of what I’ve heard has that you’ll either be “with God” or “not,” and not being with God is the bad thing in itself, irrespective of any gut-eating demons. Depictions of heaven and hell like C.S. Lewis’s The Great Divorce are popular. (Napoleon Bonaparte is all alone in a huge mansion, repeating to himself that it was everyone else’s fault...)
As for “No one comes to the Father except through me,” I had never connected that to the afterlife before; it has always seemed more like a general coming-to-Jesus saying (in life, for the sake of living, not specifically the afterlife).
It is the case (my impression, which would be interesting to verify with a survey because it’s an easy-to-ask question) that Catholics believe that there are non-Christians “with God” after death, i.e. in heaven. Even if they have to weasel out of some suggestively worded biblical passages (e.g. “good people who don’t acknowledge Christ are mystically going to the Father through Jesus”), or not even try to explain it (e.g. “God will figure it out/above my pay grade”), there’s a strong cultural current against making God look mean. Or a trickster, as you said in your original question.
Personally—maybe you’ll consider me even less of a Christian because of this—I don’t see an afterlife as something that happens to us as individuals. When there is talk about what heaven is like—hypostatic union, non-glass darkly—it doesn’t seem like much psychological continuity with one’s living ego. I’m not quite the person who inhabited this body 20 years ago, since my mind has changed a lot and what defines a person apart from their mind? So if we become outside-of-time, in-union-with-God, experiencing reality in a totally different way, how is that even me? Even following standard doctrine, the vision of heaven seems to have been exalted to such a degree that it’s no longer relevant: what I recognize of what I am will die when my body dies, and maybe something mystical that I don’t recognize goes off and does something else. But these are my own private musings (which I might change or maybe feel I have a better understanding of later) and not representative of Catholics or Christians in general.
Interesting: I’ve had the same thought and did the same experiment, though it wasn’t a tooth removal, but some tooth-drilling that I was assured would not be touching a nerve. The normal anesthesia would have been local Novocaine, and I hate how Novocaine feels for the rest of the day. (So it was a choice between two sensations, over two different time periods.) Without the Novocaine, it was like a distant, dull pounding, like falling on a bone, which can be managed. I did this more than once, but my current dentist argued more strongly against it and I acquiesced rather easily.
The main thing I was worried about was controlling my body—I didn’t want to flail and disrupt the dentist.
Just like you (and Celia Green) said about the preparation involved, I’d make a distinction between unexpected pain and expected-and-prepared-for pain. You can affect how you feel about a dentist visit, but not a sudden, stabbing pain in the back. (That may be a System-1, System-2 thing.) I’ve also found that I can relax into something cold—sitting on a stone in winter—but not something hot—being near a fire. We can choose to modify our will about some things, but others are too low-level and force themselves upon us from below.
(Which is part of the topic of “mind breaks down into smaller pieces” that I’m thinking about.)
I don’t know why I get these Less Wrong articles in my email, but I read this one because of a startling premise: choosing a job based on its monetary value as an investment. I don’t suppose there’s anything wrong with that, it’s just a bit mind-blowing for me. Maybe culture shock? But if so, what culture is this?
Making judgments with limited information is a thing, and what you say about asymmetric loss functions makes total sense. (In other words, I’m on board with the point you were trying to make with this article.) It’s just the idea of applying it to choosing a job, with total dollars earned as an optimization function that surprises me. Maybe that’s what you meant by
The “other reasons” are probably the reasons I would generally think of, such as whether the things you get to work on are interesting, how much self-direction you get or want, what you think should change in the world and whether the job makes an impact in that area, how it fits into the rest of your life, like the length of the commute, etc.