Mitchell_Porter

Karma: 10,251

Mitchell_Porter 26 Apr 2026 3:48 UTC
2 points
0
in reply to: Kajus’s comment on: A research agenda for the final year
I think ethics is an emergent rational order imposed upon impulses that arise from expectations of pain and pleasure, emotions like empathy and moral disgust. and perhaps other sources. It will be hard to know whether there is one true ethics as the natural convergent ideal, without first understanding morality as a phenomenon of consciousness, including how it relates to non-moral factors in human decision making.
I think Schopenhauer or Nietzsche at one point lists saint, artist, and scholar as three distinct human ideals. Note that only the saint is governed by morality; the artist is ruled by aesthetics, and the scholar, we could say, by epistemology. I do suspect that moral, aesthetic, and epistemic normativity are all distinct factors in human psychology (and this is not meant to be an exhaustive list), and that understanding the “human decision procedure” requires identifying their psychological and ontological roots (e.g. how they emerge in an ontology like @algekalipso’s “process-topological monads”), and that this kind of understanding is necessary to obtain the correct metaethics, and to carry out a process like Coherent Extrapolated Volition.
Also, there is probably a different decision-theoretic ontology for entities like non-conscious AIs, in which the basic cause and effect of “decision making” doesn’t even involve qualia, emotion, or conscious thought.

Mitchell_Porter 25 Apr 2026 17:34 UTC
2 points
0
on: What holds AI safety together? Co-authorship networks from 200 papers
I appreciate these impressionistic samples from the totality of the research community. It makes sense that Anthropic is adjacent to Oxford (e.g. since Anthropic is EA-adjacent) while Deep Mind is adjacent to Berkeley (more rooted, like Google, in traditional comp-sci academia?). Interesting that OpenAI stands apart from them—does it in any way derive from OpenAI having been the independent R&D powerhouse that first challenged Deep Mind’s hegemony?
It would be interesting to see similar representations for “capabilities research”, since that is what’s driving us over the precipice.

Mitchell_Porter 25 Apr 2026 7:07 UTC
2 points
0
on: Reasons not to trust AI
The risk that an ASI would optimize the world according to alien values inadvertently acquired while learning human values seems a very plausible “failure mode” of superintelligence. (Discussion with GPT-5.5 which ends with preliminary thoughts on how to avoid such failures in the context of CEV-like alignment.)

Mitchell_Porter 25 Apr 2026 2:10 UTC
6 points
3
in reply to: Random Name’s comment on: Random Name’s Shortform
Hi. It used to be about how to think, now it’s about the AI takeover of the world. “Sequence Highlights” in the left-hand menu is a selection from the “how to think” era. Good luck!

Mitchell_Porter 24 Apr 2026 17:21 UTC
3 points
0
in reply to: SpectrumDT’s comment on: Why do many rationalists like cryptocurrency?
I did not mean market value. I meant the kind of value that makes the world better.
One person’s market value is another person’s purchasing power. If I had just a tenth of a bitcoin, I should be able to keep at bay, for a few more crucial months, certain evil consequences which I do not presently know how I will otherwise avoid.
In other words, cryptocurrency is a currency and it enables all the mundane things, like purchase of goods and services, and transfer of purchasing power, which are among the benefits of money in general.
This sounds as though you have some kind of action in mind.
I don’t (and unfortunately I don’t have the luxury to think about how to reform it, right now). World finance is complicated and it is part of the social machinery that feeds and clothes billions. I think it’s not easily judged, or changed for the better.

Mitchell_Porter 24 Apr 2026 11:33 UTC
2 points
0
in reply to: Richard_Ngo’s comment on: A research agenda for the final year
starting point … reasons
Given how advanced AI is, right now, I consider superintelligence this year entirely realistic.

Mitchell_Porter 23 Apr 2026 15:53 UTC
2 points
0
in reply to: Richard_Ngo’s comment on: A research agenda for the final year
Let me belatedly express thanks for the support.
About this:
I agree it’s a terrible plan conditional on nothing you doing after the next year mattering
I take this to mean that, if frontier AI (e.g. perhaps scaffolded Mythos) really was going to take over the world later this year, it would be better to work on a global pause on frontier AI, than to try to solve CEV-level alignment.
This is like choosing which miracle to aim at—the political miracle of halting the worldwide capabilities race in AI, or the scientific miracle of resolving all the safety issues in time? I don’t know which is objectively more likely to be achieved within eight months starting from today. But the second feels more thinkable to me, partly because I’m simply more familiar with the technical world and its intellectual crown jewels, the big sweeping theories and frameworks which are our best evidence that the human mind can achieve breakthrough solutions to fundamental problems. Such breakthroughs typically have a big distinct idea at their heart, which then has massive ramifications. To have our scientific miracle, above all, we need the right ideas.
As for the political miracle… the volatility of the political and geopolitical world is such that I do not absolutely rule out a surprise last-minute coalition that stops AI progress. But it would take a remarkably potent political idea to overcome the wealth-seeking and power-seeking forces that are driving the race.

Mitchell_Porter 23 Apr 2026 14:14 UTC
9 points
0
on: Why do many rationalists like cryptocurrency?
Note: I have never owned any cryptocurrency.
cryptocurrency has negligible value
1 Bitcoin is currently worth about AUD$108,000. The current total value of all Bitcoin is around $1.5 trillion USD; of all Ethereum, around $280 billion USD. If only I had some of this negligible value for myself!
Perhaps you deplore that rogue states and organized crime are among the holders of cryptocurrency. May I nonetheless point out that the majority of paid evil that has been done in the world, was and is paid for using old-fashioned national currencies?
At this point, crypto is just another medium of exchange. Like national fiat-based economies, the crypto economy extends from the realm of ordinary practical transactions, all the way up to a realm of “whales” whose cryptic movements decide whether the currency is going up or down in value. Yes, the commanding heights of crypto must contain some sinister elements. Meanwhile, traditional currency and commodity markets are shaped by decisions of war and dictatorship, and I would guess that 99+% of the wealth of the world’s oligarchs is held in assets other than crypto. If you want to clean up the Augean stables of world finance, crypto probably isn’t very high on the list.

Mitchell_Porter 23 Apr 2026 10:15 UTC
38 points
12
on: An Angry Review of Greg Egan’s “Didicosm”
Allow me a few observations and then see if you still feel the same.
First, have you heard that the daughter of Hugh Everett killed herself (some years after his death), and in her suicide note she said she was leaving to join him in another world where he was still alive? Or something like that.
I don’t know enough to say whether the idea of the multiverse actually had a bad effect on her life, or whether it provided her with some comfort. But it’s most likely that Everett’s daughter was among the inspirations of this story. And the larger point is that weird ideas have consequences (hello Roko’s basilisk, hello Zizians...), whether or not they are true, and whether or not they truly have the implications that people associate with them.
I also suspect that Greg Egan has become very uncomfortable with people taking scenarios of transcendence from science fiction, and trying to live them literally. It’s been quite a while since he wrote a story in which the protagonists started human and made their way to some fundamentally better plane of existence. Undoubtedly the character Derek Linderman is an attack on pop-science and pseudoscience sensationalism; but may I suggest that also lurking here, is Egan’s own fear or even guilt that as science fiction writers, he and his colleagues will do in reality, what Linderman did in this story—that is, propagate exciting falsehoods that will wreck the lives of some of their readers.
This is why Egan is avoiding stories of transcendence—out of fear that these will become comforting or enervating falsehoods for the superfans who take it literally. (I believe Egan was raised religious but became an atheist.) It’s why Charlotte’s own consolation is that, amid the grifting and the delusions of the world, she played a part in fostering the genuine search for truth. It’s even why he is apriori against the idea of an LLM-powered singularity happening in the real world.
So this isn’t just an attack on Max Tegmark. It’s a story of someone whose life was gravely affected by a scientific myth of transcendence, who made a career of falsifying the myth and succeeded, only for there to be no existential change because new myths are always available, but who at least managed to pass on the flame of true rationalism in an otherwise fallen world. Whatever Egan may think of Tegmark, in this story he’s just one ingredient, along with lore about closed cosmologies, the story of Everett’s daughter, and whatever else went into the mix.
Fiction writers are constantly getting in trouble for drawing on aspects of real people as an ingredient for their fictions, whether it’s for depicting them libelously, stealing their life story unacknowledged, or whatever. And probably Egan quite dislikes Tegmark, for multiple reasons. But possibly you can appreciate that there is also a story here. It’s not a transhumanist story, it’s not the human condition transformed by technology, but it is fiction about science and the impact of science on human life.
One last thing that I almost missed: One of the minor characters in Vernor Vinge’s Marooned in Realtime is called Derek Lindemann (slightly different spelling). And Vinge’s Lindemann, in a very different way, does what Egan’s Linderman did. So I think the similarity of names is no coincidence.

Mitchell_Porter 22 Apr 2026 5:24 UTC
−18 points
−3
on: Preventing extinction from ASI on a $50M yearly budget
I think I could make a substantial contribution to solving alignment on the same timescale at a budget of $50k or less

Mitchell_Porter 22 Apr 2026 4:15 UTC
2 points
0
in reply to: Canaletto’s comment on: Automating philosophy if Timothy Williamson is correct
Those are all forms of progress in which you have someone else do the thinking. I’m talking about whether there can be philosophical progress without any thinking anywhere at all.
(I have sidestepped the scenario in which AI achieves philosophical progress by thinking, because we don’t understand what thinking is, enough to say whether an AI is thinking or not, which brings us back to the first answer, that progress regarding the nature of consciousness, intentionality, thinking, etc, is needed before you have any chance of formalizing or automating philosophy.)

Mitchell_Porter 22 Apr 2026 2:39 UTC
1 point
−5
on: Opus 4.7 Part 2: Capabilities and Reactions
Regarding deanonymization, I’m mildly curious about one thing: if it’s your own prose, and the AI knows who you are, it will have some extra reason to think that the unknown author might be you. Do people do anything to counteract this, e.g. by using incognito?

Mitchell_Porter 21 Apr 2026 6:21 UTC
4 points
0
in reply to: Wei Dai’s comment on: Automating philosophy if Timothy Williamson is correct
Bear with me while I just naively tackle the question, what is philosophy?
Where to start? Well, we could start with “I think therefore I am”, which is the “E equals M C squared” of philosophy, its most famous proposition. Where did that proposition come from, what is the process that produced it?
I was familiar with an anecdote about Descartes, that while serving as a soldier as a young man, he was shut up one night in a house, and had some thoughts that were formative for his philosophy. I had hoped that we might be able to say, that was the night that he first tried out universal doubt, but that doesn’t seem to be the case. It’s more that he had some big ideas (like algebraic geometry), and realized that he could spend his life developing them.
So all we can say is that Descartes thought a lot, and that at some point he had thoughts which he distilled into the cogito argument, which he later put into a book. Can we at least say what sequence of thoughts is characteristic of the cogito? You have to end up asking yourself, is there anything I can’t doubt? And after doubting your memories, your sensations, all your beliefs, you end up saying: I am definitely doubting, which is a kind of thinking, and I cannot be thinking unless I exist, so I definitely exist.
What kind of thoughts were these? Descartes was reasoning about reality, and about what we can know about reality. That itself might pass as a first approximate definition of philosophy. But perhaps it’s not enough; perhaps there are other mental faculties which are an essential part of the process. How do I know what I am thinking, or that I am thinking—is this self-knowledge obtained through reasoning? Maybe it’s due to a different faculty, more akin to perception but directed inwardly—this is sometimes called intuition (this is intended as a technical name for inner perception, it’s something different from intuition in the sense of creative leaps to complex truth through pattern matching). In characterizing the cognition that produces philosophy, we might want to include intuition along with reasoning, as faculties that are involved…
I’ll pause here because I’m not aiming to produce an ultimately precise characterization of philosophy’s nature, but just to illustrate what’s involved. We can say: philosophy is a kind of thinking, it is directed at certain very general or fundamental or mysterious topics (we stipulate this to distinguish it from thought which occurs within the bounds of some other discipline), there will be some arbitrariness or edge cases, and so on.
Now suppose we want to “formalize” (or even “automate”) philosophy. One of the very first questions would be: must philosophy and philosophical progress, occur in the form of thinking? If it does, then you also have a big part of the reason why there’s no well-known formalization of philosophy: because we don’t have a “formalization” of thinking. In turn I see at least two perspectives on this. One would be to focus on “general intelligence”, and say, we won’t know what thinking is, until we know how general intelligence works. The other would be to focus on consciousness, and say, we don’t know how to think about thinking because we don’t know how to think about consciousness.
But that is not entirely true; we actually do know some ways to think about consciousness. It’s just that they do not fit into the methods of formal and natural science, because those by construction have excluded consciousness as an object. To think about consciousness, first you must allow yourself to use your natural faculty of thinking informally about it, and then you can try to be rigorous about it. This kind of thinking about consciousness has been pursued in philosophy, in literature, in psychology… And to be fair, pretty much anyone who comes up with a theory of consciousness, even if it is a wrong theory, must do a certain amount of informal-to-formal thinking-about-thinking.
So that’s one approach to why we lack formalization of philosophy: it won’t come without progress on consciousness.
On the other hand, suppose we go back to the question: must philosophy and philosophical progress occur in the form of thinking? - and suppose we try out the answer no. Suppose we entertain the idea of making progress on the subject matter of philosophy just via computation, without the use of consciousness. Maybe it’s possible, but it seems like you would need to have formalized philosophy first, in order to then pursue it mechanically. And this leads me back to the previous barrier: you can’t formalize philosophical thinking until you understand thinking, and you can’t understand thinking without understanding consciousness, and you can’t understand consciousness without allowing yourself to first freely use the natural capacity for informal thinking about consciousness, and seeing it where it leads.
Incidentally, there is a school of thought due to Edward Zalta which aims to formalize ontology. Zalta’s work might be of interest to would-be formalizers of philosophy in general.

Mitchell_Porter 20 Apr 2026 10:58 UTC
2 points
0
in reply to: Mikhail Samin’s comment on: Vladimir Putin’s CEV is probably not that bad
CEV is a process that combines extrapolated volitions of individual humans, which is meant to depend fully on the state of every particular person and their wishes about how they wish are to be extrapolated
I think it is unclear what the exact initial data are supposed to be, or needs to be.
The value system that CEV outputs is going to be abstract at some level. It won’t say directly “if someone has a toothache, fix the toothache”; that should follow from a more general principle, combined with the nature of toothaches. The same goes for the extrapolations of individuals and the aggregations of their preferences: the CEV value system in action has to care about particulars, but what it does with those particulars, will be governed by an abstract definition.
The question is, what do we need to know about humanity, in order for CEV’s Value Extrapolation Procedure to arrive at the correct abstract definition? This is hard to answer if we don’t know what the VEP is in any detail. But finding a correct VEP is also part of the process.
Apparently a popular proposal for the VEP is something like “upload 10,000 philosophers and let them deliberate for as many subjective years as they need to solve all CEV’s problems and arrive at a consensus”, or similar proposals according to which there is a digital parliament of human proxies (e.g. Jan Leike’s “simulated deliberative democracy”).
I guess this defines a possible VEP, but I have long thought that a better VEP would involve theoretical identification of the existing “human decision procedure” (which I assume is a topic for cognitive neuroscience, and which in the individual is determined through a mix of genes, culture, and life incidents), and then extrapolating that. And again, the human decision procedure would in some way be a template, a schema whose details are “filled out” differently in different individuals (similar to how we learn the grammar and vocabulary of our native languages); and some of CEV’s extrapolation would depend on those details, some of it only on the structure of the schema.
You might even expect that Leike’s democracy would arrive at something like this, rather than just deciding everything via a vote among our extrapolated higher selves, forever. But then do you need the whole digression into upload societies devoted to the task of alignment? You just do AI-assisted neuroscience, figure out how human nature actually works, and “extrapolate” that.
Years ago, I thought that might be what would happen. Instead, the VEP that our frontier AI companies are employing, is to engage in value learning from the training corpora, as part of general world-modeling, and then refining and activating it with RHLF, constitutions, and so forth.

Mitchell_Porter 20 Apr 2026 9:48 UTC
−4 points
−4
on: Reevaluating “AGI Ruin: A List of Lethalities” in 2026
Can you give us a tldr: Is AI going to kill us all, or not? Is AI going to take over, or not? If AI does take over, what are the odds that it is human-friendly?

Mitchell_Porter 19 Apr 2026 5:47 UTC
7 points
0
in reply to: the gears to ascension’s comment on: Vladimir Putin’s CEV is probably not that bad
I agree very much that discussions of CEV could do with a lot more precision. In recent comments I have been led to talk about Extrapolation Machinery and Value Extrapolation Procedures, so I could make the point that different VEPs will produce different outputs from the same input, and when people talk about CEV, they often have quite different VEPs in mind…
Also, excuse me for using this opportunity to speak in capitals, but DOES ANYONE KNOW WHAT BECAME OF JUNE KU, because metaethical.ai was a shockingly good attempt to formalize CEV.

Mitchell_Porter 18 Apr 2026 7:47 UTC
0 points
0
on: You can only build safe ASI if ASI is globally banned
Good luck at your new job (American head of PauseAI?), but I don’t think you have much time left in which to rein in Anthropic and OpenAI, if you don’t want them to cross the threshold to ASI. (The other day, Trump talked about the need for a “kill switch” for advanced AI, so I guess that’s a start...)

Mitchell_Porter 17 Apr 2026 16:31 UTC
5 points
0
in reply to: cousin_it’s comment on: Let goodness conquer all that it can defend
This is a provocative comment and I don’t have time to develop half the themes I find in it.
For example: This kind of historical critique plays a role in the struggle within and between civilizations, about which peoples and values rise and fall. In part, the current war in the Middle East is between two regimes which exist in order to reverse ancient wrongs (the destruction of the Second Temple and the defeat of Ali at Karbala).
Obviously some of the West’s current cultural and political struggles—over topics like immigration, liberal democracy, even Jewish power apparently—have a heavy historical dimension, which are of this nature. @habryka is not volunteering to regard everything that the West has done as better than the alternative, but he wants to affirm that the existence of America is better than its nonexistence, even if choosing its existence includes everything that led to its existence in this timeline.
I believe you’re a Russian living in America, and Russia has been through such struggles many times—in passing from imperial heartland to Soviet republic to democratic republic—and they have included an element of psychological warfare in which external powers with different values allied with internal critics of Russia, to aim at either the destruction or transformation of Russia. Russia in turn has been a persistent critic of western imperialism, first within the framing of socialism versus capitalism, now within the framing of multipolarity versus unipolarity.
Your accounting of the sins of western colonialism has a framing which to me is political and not just about body counts. At least that’s what I thought when you included China’s century of humiliation, which obviously matters for China, but in terms of death toll is not on the same level as Indian famines or depopulation of the Americas (unless you want to attribute late Qing calamities like the Taiping rebellion directly to the colonial powers, rather than to China’s internal struggle). So China might not belong on the list of the West’s greatest sins—but it helps politically to include China in the list, if you’re building a coalition against present-day western power, as Russia has tried to do.
You disapprove of @habryka wanting to affirm these aspects of the American past. My questions are: Do you see analogies between America’s situation now, and Russia after imperialism or after communism (or even, potentially, after Putin)? And, as someone who I believe is a participant in America, what do you think is the correct attitude? Would you want to go all the way to the Bobby Fischer solution—whites back to Europe, blacks back to Africa (and implicitly everyone else back to their homelands), hand the American continent back to the indigenous peoples? Or is the rise of progressivism an appropriate response, perhaps analogous to the rise of communism in 20th-century Russia? Also, if American civilization and/or European colonialism had never existed, are there good, important, even crucial features of our present world, that might never have existed as well?

Mitchell_Porter 17 Apr 2026 3:11 UTC
5 points
0
on: Against Doom & Pause AI
Sorry for being a bad reader. I browsed your post but… We’re talking about the creation of artificial non-biological entities far smarter than any human being. To do that is to replace the human species as the most capable “lifeform” on Earth. Could you summarize for me, why you think that’s a good idea, and why you think we have any idea of the consequences?

Mitchell_Porter 16 Apr 2026 9:25 UTC
5 points
0
in reply to: Steven Byrnes’s comment on: Contra Byrnes on UV & cancer
I am from Queensland, Australia, the “skin cancer capital of the world”. Most of us get it, eventually. Is there some datum from here, that would help decide between your theory and @HedonicEscalator’s orthodoxy?