Also, look at his bet with Bryan Caplan. He’s not joking.
And, also, Jesus, Everyone! Gradient Descent, is just, like, a deadly architecture. When I think about current architectures, they make Azathoth look smart and cuddly. There’s nothing friendly in there, even if we can get cool stuff out right now.
I don’t even know anymore what it is like to not see it this way. Does anyone have a good defense that current ML techniques can be stopped from having a deadly range of action?
I think the OP’s overarching concern is something like a narrow utilitarianism whose decision algorithm takes EV over only a limited number of horizons and decision sizes. There is unknown EV in exploring the world more personally and in reproducing knowledge and skills. My hunch is that such optimization of human life takes these different aspects at least multiplicatively.
Expected value calculations have limits for decisions which will affect your worldview, i.e. exploration. Or decisions along the axis of goods which you don’t have a good model for, i.e. education.
While I would say your timeline is generally too long, I think the steps are pretty good. This was a visceral read for me.
Some sociological points:
I think you don’t give anti-AI development voices much credence and that’s a mistake. Yes, there will be economic incentives, but social incentives can overcome those if the right people start to object to further specialized LLM development.
Although you have a fairly worked out thought on AI development, where the path is clear, for AIS the fact that you ended with a coin flip almost seems like slight of hand! The scenario you describe was clearly heading towards self-destruction, not just for narrative tenor and story based reasons, but for the lack of finding a foundation upon which to ground true AIS.
With improved robotics, there might be solutions to escape certain AI social traps by changing the balance of power between information and physical world development. I have in mind the idea that AI seems incredibly valuable today because physical progress has been so weak. But if physical progress becomes strong and fast, then allowing/forcing AI to slow down will be socially easier. It’s an elasticity of demand model for competing goods. Does that make sense? If not, I’d happily elaborate.
I think if you live in a context where having kids is a norm, that is, where the local knowledge and family -friendship support of having and raising kids prevails, then truly arguments are a waste of time. You have freedom of choice, knowing well what that option entails.
But I think most people are not in a situation like mruwnik where they have seen large families in action; they don’t really have the freedom to have a large family, since the metis is missing.
In any case, I think any ethical philosophy worth a penny includes an ethics of family, economics, and societal growth. Philosophical argument on its own might not serve as reason to have kids. But our philosophies, examined or unexamined, often serve as justifications of momentous life choices of this sort. So I think I will reject that ‘logicking’ about having kids is a waste of time. Especially when people cite reasons philosophical and ethical for having/not having kids all the time.
Well said! Good thoughts. Since, you bring this up at the same time as I have been thinking about it, I feel obligated to add my current thoughts now, even though they are as yet not fully developed.
I also just started reading Walter Isaacson’s biography of Ben Franklin and have been reading a few article about the so-called “Republic of Letters” which existed outside of the academy, made real scientific progress, and was contributed to by both academics and enthusiasts alike. Here are some of the dynamics that seemed to make the Republic of Letters and the British clubs work:
1) a drive to invite promising people to the group, even if they are not up-to-speed yet.
2) those personal, friendly invitations go to up and coming writers and thinkers who still have enough slack in their time to join a new community. People at the beginning of their career are the life of an organization.
3) an expectation that members not just observe but also write, share, and present essays, and host occasional focused discussions.
“The Tails Coming Apart as a Metaphor for Life” should be retitled “The Tails Coming Apart as a Metaphor for Earth since 1800.” Scott does three things, 1) he notices that happiness research is framing dependent, 2) he notices that happiness is a human level term, but not specific at the extremes, 3) he considers how this relates to deep seated divergences in moral intuitions becoming ever more apparent in our world.
He hints at why moral divergence occurs with his examples. His extreme case of hedonic utilitarianism, converting the entire mass of the universe into nervous tissue experiencing raw euphoria, represents a ludicrous extension of the realm of the possible: wireheading, methadone, subverting factory farming. Each of these is dependent upon technology and modern economies, and presents real ethical questions. None of these were live issues for people hundreds of years ago. The tails of their rival moralities didn’t come apart – or at least not very often or in fundamental ways. Back then Jesuits and Confucians could meet in China and agree on something like the “nature of the prudent man.” But in the words of Lonergan that version of the prudent man, Prudent Man 1.0, is obsolete: “We do not trust the prudent man’s memory but keep files and records and develop systems of information retrieval. We do not trust the prudent man’s ingenuity but call in efficiency experts or set problems for operations research. We do not trust the prudent man’s judgment but employ computers to forecast demand,” and he goes on. For from the moment VisiCalc primed the world for a future of data aggregation, Prudent Man 1.0 has been hiding in the bathroom bewildered by modern business efficiency and moon landings.
Let’s take Scott’s analogy of the Bay Area Transit system entirely literally, and ask the mathematical question: when do parallel lines come apart or converge? Recall Euclid’s Fifth Postulate, the one saying that parallel lines will never intersect. For almost a couple thousand years no one could figure out why it was true. But it wasn’t true, and it wasn’t false. Parallel lines come apart or converge in most spaces. Only, alas, only on a flat plane in a regular Euclidean space ℝ3 do they obey Euclid’s Fifth and stay equidistant.
So what is happening when the tails come apart in morality? Even simple technologies extend our capacities, and each technological step extending the reach of rational consciousness into the world transforms the shape of the moral landscape which we get to engage with. Technological progress requires new norms to develop around it. And so the normative rules of a 16th century Scottish barony don’t become false; they become incomparable.
Now the Fifth postulate was true in a local sense, being useful for building roads, cathedrals, and monuments. And reflective, conventional morality continues to be useful and of inestimable importance for dealing with work, friends, and community. However, it becomes the wrong tool to use when considering technology laws, factory farming, or existential risk. We have to develop new tools.
Scott’s concludes that we have mercifully stayed within the bounds where we are able to correlate the contents of rival moral ideas. But I think it likely that this is getting harder and harder to do each decade. Maybe we need another, albeit very different, Bernard Riemann to develop a new math demonstrating how to navigate all the wild moral landscapes we will come into contact with.
We Human-Sized Creatures access a surprisingly detailed reality with our superhuman tools. May the next decade be a good adventure, filled with new insights, syntheses, and surprising convergences in the tails.
You seem to value loyalty and generally want to be a tit-for-tat player + forgiveness. That makes sense, as does calibrating for the external environment in the case of PDs. You also view the intellectual commons as being increasingly corrupted by prejudiced discrimination.
This makes sense to me. This makes sense, but I think the thinking here is getting overwhelmed by the emotional strength of politicization. In general, it is easy to exaggerate how much defection is actually going on, and so give oneself an excuse for nursing wounds and grudges longer than absolutely necessary. I’m guilty of this and so speak from some experience. So I think there’s an emotional mistake here. Perhaps fixed by seeking out “philosophical repose”, a bit more detachment from reacting to the opinions and thoughts of others.
Intellectually, I think you are making an important mistake. Cooperation in PDs and moral philosophy are two different, though related, things. Welfarist utilitarianism does not require impartiality, but choosing those actions which maximize good consequences. Since your reason for rejecting welfsrism is because of the consequences of not punishing free riders, you have not yet rejected utilitarianism. You are rejecting impartiality as a necessary tenant of utilitarianism, but that is not the same thing as rejecting utilitarianism. You have to go one step further and give non-consequentialist reasons for rejecting them.
Hope that helps.
I am working through questions about paradigms and historiography right now. These questions drove me to write this creative speech. I went from, “Is there such a thing in history as ‘just the facts?’” and from there I went to is there anything in cartography as “just the facts.” This reductio ab absurdum I hope shows that maps are used for different purposes, and there are better and worse maps for different purposes. We are looking for maps which fit our purposes. The right maps for the right purposes.
According to the line of reasoning in the reductio, there is no map which is “just the facts” without also being “all the facts” and thus becoming the territory itself.
What does this say about the craft of history? I don’t know.
An excellent quote: “a market-based society that uses social pressure, rather than government, as the regulator, is not [the result of] some automatic market process: it’s the result of human intention and coordinated action.”
I think too often people work on things under the illusion that everything else just “takes care of itself.”
Everything requires effort.
Efforts in the long-run are elastic.
Thus, everything is elastic.
Don’t take for granted that industry will take care of itself, or art, or music, or AI safety, or basic reading, writing, and arithmetic skills. It’s all effort—all the way down..
Love this! Having my first child really reinforced by views that IQ needs constant outlets to explore and express, and thus that the frame of the environment determines what types of skills the IQ puts effort into solidifying and reinforcing. I look at my little gradient descent machine scooting down the stairs, and I think to myself, “What a wonderful world.”
There are so many arguments trying to be had at once here that it’s making my head spin.
Here’s one. What do we mean by self-help?
I think by self-help Scott is thinking about becoming psychologically a well-adjusted person. But what I think Jacobian means by “rationalist self-help” is coming to a gears level understanding of how the world works to aid in becoming well-adapted. So while Scott is right that we shouldn’t expect rationalist self-help to be significantly better than other self-help techniques for becoming a well-adjusted person, Jacobian is right that rationalist self-help is an attempt to become both a well-adjusted person AND a person who participates in developing an understanding of how the world works.
So perhaps you want to learn how to navigate the space of relationships, but you also have this added constraint that you want the theory of how to navigate relationships to be part of a larger understanding of the universe, and not just some hanging chad of random methods without satisfactory explanations of how or why they work. That is to say, you are not willing to settle for unexamined common sense. If that is the case, then rationalist self-help is useful in a way that standard self-help is not.
A little addendum. This is not a new idea. Socrates thought of philosophy as way of life, and tried to found a philosophy which would not only help people discover more truths, but also make them better, braver, and more just people. Stoics and Epicureans continued the tradition of philosophy as a way of life. Since then, there have always been people who have made a way of life out of applying the methods of rationality to normal human endeavors, and human society since then has pretty much always been complicated enough to marginally reward them for the effort.
Helping make more accurate predictions about the future by reducing the “X isn’t allowed to happen” effect (or, as Anna Salamon once put it, “putting X into the realm of the thinkable”).
This point, along with your larger module here, is something, I promote often and has helped me and my friends immensely! Just last night, a distressed friend called me and was having a panic attack about a guest-speaker he had invited to his college campus. He was worried this guest-speaker was going to do something morally questionable at another person’s expense publicly and that would reflect badly on him and his organization. The speaker did just such a thing a little over a week ago at an Ivy League, so his fear has a rational trigger. My task was to 1) reinforce to him that he is easily smart enough to deal with these challenges and 2) that in the unlikely event of the worst happening he should be prepared. Eventually we got to imagining the worse case scenario, and he came up with a few precautions and fail-safes to protect himself and others.
I am very proud of him. Moral of the story: don’t shut down, don’t pigeon-hole yourself, think about what is within your power and what is not, prepare your mind and your world for the coming turbulence.
I stand by this comment.
What could cause me to change my mind? Here are my cruxes.
If character assessment posts about particular people can be shown to cause a useful actions or ways of thinking for readers more often than they distract readers by unverifiable gossip.
If character assessment posts about particular people is used as a case study for reasoning about particular people to teach a broader lesson.
If character assessment posts about particular people allows community members to protect themselves from a real danger.
However, my beliefs are that these types of posts are juicy gossip that fuel idle speculation and status hierarchy games and serve no purpose except to make those who engage with content worse people who think more simplistically about human behavior and motivation. Even though this particular post is done fairly well for what it is, I think it is “bad form” and, perhaps, on the wrong site.
I read this post when it initially came out. It resonated with me to such an extent that even three weeks ago, I found myself referencing it when counseling a colleague on how to deal with a student whose heterodoxy caused the colleague to make isolated demands for rigor from this student.
The author’s argument that Nurture Culture should be the default still resonates with me, but I think there are important amendments and caveats that should be made. The author said:
“To a fair extent, it doesn’t even matter if you believe that someone is truly, deeply mistaken. It is important foremost that you validate them and their contribution, show that whatever they think, you still respect and welcome them.”
There is an immense amount of truth in this. Concede what you can when you can. Find a way to validate the aspects of a person’s point which you can agree with, especially with the person you tend to disagree with most or are more likely to consider an airhead, adversary, or smart-aleck. This has led me to a great amount of success in my organization. As Robin Hanson once asked pointedly, “Do you want to appear revolutionary, or be revolutionary?” Esse quam videri.
Combat Culture is a purer form of Socratic Method. When we have a proposer and a skeptic, we can call this the adversarial division of labor: You propose the idea, I tell you why you are wrong. You rephrase your idea. Rinse and repeat until the conversation reaches one of three stopping points: aporia, agreement, or an agreement to disagree. In the Talmud example, both are proposers of a position and both are skeptics of the other persons’ interpretation.
Nurture Culture does not bypass the adversarial division of labor, but it does put constraints on it—and for good reason. A healthy combat culture can only exist when a set of rare conditions are met. Ruby’s follow-up post outlined those conditions. But Nurture Culture is how we still make progress despite real world conditions like needing consensus, or not everyone being equal in status or knowledge, or some people having more skin in the game than others.
So here are some important things I would add more emphasis to from the original article after about a hundred iterations of important disagreements at work and in intellectual pursuits since 2018.
1. Nurture Culture assumes knowledge and status asymmetries.
2. Nurture Culture demands a lot of personal patience.
3. Nurture Culture invites you to consider what even the most wrong have right.
4. Sometimes you can overcome a disagreement at the round table by talking to your chief adversary privately and reaching a consensus, then returning for the next meeting on the same page.
While these might be two cultures, it’s helpful to remember that there are cultures to either side of these two: Inflexible Orthodoxy and Milquetoast Relativism. A Combat Culture can resolve into a dominant force with weaponized arguments owning the culture, while a Nurture Culture can become so limply committed to their nominal goals that no one speaks out against anything that runs counter to the mission.
There is some confusion here. Asking Less Wrong flavored questions using theological terms generally requires misusing the terms. This is unfortunate, because these questions are really interesting, but most us don’t have the requisite understanding of theology to do it well (including myself obviously(although, I venture that I might know more than most(#nesting))). So, my answer will be really disappointing.
In the monotheistic theology of Islam (represented by Al-Farabi), Judaism (represented by Maimonides), and, Christianity (represented by Thomas Aquinas), when it is said that God is omnipotent, they are saying God is not lacking in power, not that God can actually will to do any particular action whatsoever. In this way God is restrained. For example, God cannot create a rock so heavy that he cannot lift it because that is not a logical possibility. Or as a mathematician once said, “Nonsense is nonsense, even if you say it about God.”
To your question about a loving God’s possibility to deceive. This is a tough question because it is several in one. Can God deceive, can God’s nature be learned about through observation of the created universe, and can God deceive about his nature? The first two questions are contested within each faith tradition, the third question (which I think is most relevant here) third is not disputed by the three philosophers. They all would say, “No.”
I’m going to summarize a really long arguments the best I can: since God is a self-caused simple being (having no parts and lacking in no quality), his intellect (it’s an operation) can only be directed toward Truth and his will (it’s his other operation) can only be directed toward the Good (which is love).
This argument requires that we agree that Truth and Goodness have a primary level of existence, whereas falsity and evil exist contingently on the existence of truth and goodness. Since God has no parts, he cannot be oriented towards the composite essences of falsity and evil.
This is definitely an unsatisfying solution for most of us. The major problem for us approaching Theological Epistemology, as I see it, is that we have to start by explaining what metaphysics we are willing to accept and what we aren’t.
I for one am definitely worse off.
I now have to read Scott on Substack instead of SSC.
Scott doesn’t write sweet things that could attract nasty flies anymore.
Here is a quick list of things that spring to mind when I evaluate intellectuals. Any score does not necessarily need to cash out in a ranking. There are different types of intellectuals that play different purposes in the tapestry of the life of the mind.
How specialized is this person’s knowledge?
What are the areas outside of specialization that this person has above average knowledge about?
How good is this person at writing/arguing/debating in favor of their own case?
How good is this person at characterizing the case of other people?
What are this person’s biggest weaknesses both personally and intellectually?
You think that the Will of Melkor could be potentially un-corrupted? Though all evil in Arda is his doing, bitterness, greed, excessive heat, piercing cold, avaricious orcs, other creatures that lust for blood or power, and even darker things than this, that is not enough to despair of him? Aye, you don’t yet realize what Morgoth is. He is pride and cruelty and rage. Morgoth’s spirit cannot change, because it presumes the only worthwhile fact is its own continued self-expression. His cruelty wishes to wrench all that’s beautiful and true in the world into darker purposes, to turn good intentions to bad ends. And his rage, oh, his rage is a starless cry of cosmic dismay that all the evils of the world are undone one by one.
Tell me not of the turning of Melkor, even now he is chained up on the other side of night, yet still whispering his will throughout Middle-Earth, and plotting to break the Gates and return again to destroy the Sun and the Moon and all living things. You are telling me that same guy who wants to tear down all the universe in supreme envy can be turned to work for Good?
This is Morgoth we are talking about, not Anakin Skywalker.
P.S. I take your point; but the Maiar and Valar do act indirectly in Middle-Earth, though creation is over as are the wars against Sauron and Ar-Pharazon. They do not just accept the the gradual decline of the world.
I will never relinquish my Sword of Fandom +10. :)
I learned those from Wittgenstein, Aristotle, and Umberto Eco.
Numerical obsession can simplify important semantic subtleties. And semantic pyrotechnics can make my reasoning more muddled. Both are needed. As is, I suppose, a follow up post on the non-LW ideas that pay a lot of rent.
Bryan Caplan and Eliezer are resolving their Doomsday bet by having Bryan Caplan pay Eliezer upfront and if the doomsday scenario does not happen by Jan 1 2030, Eliezer will give Bryan his payout. It’s a pretty method for betting on doomsday.