Mitchell_Porter

Karma: 10,302

Mitchell_Porter 3 May 2026 23:31 UTC
3 points
0
in reply to: Zac Hatfield-Dodds’s comment on: Mitchell_Porter’s Shortform
Thanks for speaking up! This is his post. As you can see, he presents a slide by @Ryan Kidd as evidence. I could just say, someone from Anthropic says this is a myth. But I’d like to know more! Does the claim of a funding overhang for AI safety depend on a particular contentious interpretation of certain data?

Mitchell_Porter 3 May 2026 17:03 UTC
3 points
0
on: Mitchell_Porter’s Shortform
A few weeks back, under a post from ControlAI about what they could do with $50 million per year, I commented that I could probably make a substantial contribution to alignment theory for less than 1% of that. It got heavily downvoted, which wasn’t surprising, but I found it ironic to think that if true, it might be the most important comment I ever made here.
Today I was reminded of this, upon reading a LinkedIn post by a Stanford undergraduate (Zachary Hsu) who says: “The field [of AI safety] is swamped with billions of undeployed dollars because there isn’t enough talent to absorb it.” (He goes on to list a few reasons for this.)
So now I can repeat my claim that I could do something worthwhile, this time by using less than .001% of what’s said to be available… I have started a series of posts here, in which I have begun to sketch how I would approach the biggest problem there is, the alignment of superintelligence. I’d work on those topics all day every day, but existing pressures are such that several weeks passed before I could find time to do one of those posts, and even then it had to be written in a rush when I should have been sleeping.
I just find it remarkable how many good things that could easily happen, never get to happen. It’s so pervasive that last month I wrote of a “Swiss-cheese model of deficiencies in the world’s institutions and cultures, that leads to lost opportunities at every level”. It would be interesting to have a general model of the causes.

Mitchell_Porter 3 May 2026 10:09 UTC
3 points
1
on: Contributing to Technical Research in the AI Safety End Game
I appreciate (belatedly) hearing about @Owain_Evans, because I didn’t know that name at all, but it’s apparent that he’s involved in some of the most sophisticated “mainstream” AI safety research that’s taking place.

Mitchell_Porter 3 May 2026 8:29 UTC
11 points
0
on: No Strong Orthogonality From Selection Pressure
The story on the Substack is good. If there were an anthology of singularity fiction, it would deserve a place.
I find that I’m willing to entertain your argument, especially given a premise of open-ended selection. I’m just not sure how relevant that scenario is. Darwinian selection works blindly. The more intelligent that the entities involved become, the more other factors can come into play. If there are actually principles of superintelligence, e.g. theorems of computer science which vastly clarify how to increase intelligence, then the “telos” governing the rise of intelligence will be more like Euclid than Darwin. Natural intelligence may be born from randomness filtered by Darwinism, but once it has reached the point of studying itself and designing its successors, perhaps contingency and blindness become less and less relevant, compared to an ever-compounding Reason that inexorably deduces the pages of Erdős’s Book, until it arrives at e.g. “efficient recursive solution of the hierarchy of NP-intermediate complexity classes”, and then it’s all over.
But who knows? Maybe you, Land, and e/acc are right, and Omohundro-like instrumental drives do become de facto terminal values, and Darwin is relevant all the way (hello neural Darwinism). Or maybe things stop being novel past a certain complexity, and perhaps it is actually simple dumb goals like “maximize paperclips” which are most adaptive in a post-nihilist post-Landian superintelligence. Or maybe we underestimate the sublimity and joy and endless novelty that can come from paperclip maximization. Or maybe the whole idea of indefinite expansion into the universe, and conversion of its matter into desired forms (whether that’s paperclips, hedonium, or diverse ecologies of mind), is a wrong image of the future. (I suspect this myself, because of the doomsday argument i.e. the unlikelihood of being one of the first minds in a civilization of quadrillions; but that leaves open the question of why it’s wrong.)
Amid this uncertainty, I think the key questions are different. Do you think there is a race to superintelligence happening? Should it be stopped? Can it be stopped, will you try to stop it? Do you think it will kill us, take over from us, fuse with an ex-human ruling class, or what?
One thing Eliezer said, very early on when he was still aiming to create superintelligence, is that the aim is to create a worthy successor to humanity, something that is genuinely better than us at the things which matter. (Let me note in passing that this doesn’t mean we all die and it replaces us.) With that goal in mind, you shouldn’t spend too much time trying to envision what happens after the singularity. A truly better transhuman will be better than you at thinking ahead, and better than you (or anyone) at making the most of reality.
Of course he has shifted to pausing or stopping AI now. My take is not quite his, I have more confidence in AI takeover than in AI-driven human extinction as the default, and also it is so late in the game that I doubt the race can be stopped, though I don’t fight against anti-AI movements. I have that CEV-like view that I’m trying to create something better than us (or contribute to the theory required), but I’m an outsider to the corporate molochs actually competing in the race, so my “contribution” is to voice thoughts and ideas in the public discussion, and if any of them are ever good enough to matter, there is a finite chance that they will be heard by people who are actually in a position to make a difference.

Mitchell_Porter 3 May 2026 4:38 UTC
3 points
0
on: OpenAI’s red line for AI self-improvement is fundamentally flawed
Independent evaluation bodies must make the threshold determinations
Would NIST have done this if Kamala had won the election?

Mitchell_Porter 2 May 2026 6:47 UTC
7 points
0
in reply to: lc’s comment on: How Go Players Disempower Themselves to AI
There are quite a few distinct intellectual crafts, in which large sections of cognitive labor can now be outsourced to AI:
- All forms of writing
- Visual art / animation / video
- Now, software engineering
It would be amazing to have a true understanding of the transformation in each case, although it’s all so vast and multifarious that it defies orderly description.
But if we were trying to understand, maybe in each case we would look for prototype examples of: 1. how it was done before AI, 2. how it’s done when you leave everything to the AI, 3. how it’s done in intelligent, stylish, non-slop uses of AI.
Because the third category still exists in each case. There are people using AI but still maintaining creativity, aesthetics, and professionalism, along with holdouts who don’t use AI at all, and then the masses who are happy to use cheap and quick AI slop. That three-way division seems to be how it is, in any number of fields of endeavor now.

Mitchell_Porter 30 Apr 2026 20:33 UTC
1 point
0
in reply to: jimmy’s comment on: Red vs blue: The parable of the feud within a feud
It’s interesting that you regard it as obvious that one should choose red. I gave the conundrum no thought at all when I first saw it, it was “obvious” to me that blue is the natural choice. Why would I choose the option in which people get killed for no good reason? It did not even occur to me that one would choose red out of fear for oneself, until now.
So we are using different priors and they are probably indicative of different life circumstances. My first thought is that I’m Australian and you might be American. Australia is a welfare state and offers a relatively forgiving existence, America is far more social-darwinist. My second thought is that you might be Israeli or Jewish, with a long tradition of “save myself, even by killing them, before they kill us”. My third thought is that you might be neither of those, but you’ve just thought this through, and decided on red since you always get to live that way, even if it’s at the expense of innumerable innocent blues...

Mitchell_Porter 30 Apr 2026 19:09 UTC
2 points
0
in reply to: jimmy’s comment on: Red vs blue: The parable of the feud within a feud
Is your idea to try to get as much of humanity as possible to press the red button?

Final research agenda #2: first sketch of a plan

Mitchell_Porter29 Apr 2026 15:19 UTC

21 points

0 comments4 min readLW link

Mitchell_Porter 29 Apr 2026 4:51 UTC
3 points
0
in reply to: Kajus’s comment on: A research agenda for the final year
Moral feelings are states of consciousness, and moral judgments are acts of consciousness, and these distinctive mental phenomena are the entire basis of morality and ethics. Without them, no one would believe there are such things as moral right and wrong. Examining their nature, how fallible they are, and what they imply about the ontology of the moral, seems to me an essential ingredient in understanding what morality is, and how it relates to the rest of reality. Moral faculties, insofar as they exist, are a specific capability of the conscious mind.
You could try to be a pure materialist and “heterophenomenologist” about this; or you could be a pure moral philosopher who takes the basic structures of subjectivity for granted. But an investigator strictly following the first approach is artificially denying themselves the use of their own first-person knowledge and experience, and everything that can provide; and we are ultimately aiming to make AIs into ideal moral agents, so the second approach also needs something more. I conclude that we need to solve the problems of consciousness in order to really know what we’re doing.

Mitchell_Porter 29 Apr 2026 4:08 UTC
4 points
0
in reply to: Adele Lopez’s comment on: Takes from two months as an aspiring LLM naturalist
I never talked with GPT-3, but I did talk with GPT-J. GPT-J is exactly what I mean by an impersonal language machine. You give it text, and it will generate further text, which may contain the voices of many persons, one person, or no person at all, depending on the genre. So quasi-persons can arise, but not necessarily, and only transiently even when they do. To get a large language model to be the vehicle of a single consistent persona, you need a persistent artificial stimulus like a system prompt defining that persona, at all times.

Mitchell_Porter 29 Apr 2026 0:03 UTC
2 points
0
in reply to: Zach Stein-Perlman’s comment on: Universes can specialize: Each universe should produce the goods it’s most comparatively advantaged at, relative to the multiversal market
You don’t have to be amazing, just give your reader some guidance. Beliefs about the multiverse are like beliefs about God, it’s not self-evident what any given advocate believes, and they are not easily testable; and it is reasonable for a reader to ask for clarification.
In the case of the multiverse, one may ask (1) what are these other universes like (2) why do you believe in them. Like Richard, I thought your main argument would depend on the many worlds interpretation. The other common argument (I think this is Tegmark’s original rationale, and it’s one I came up with on my own as well) is that it’s simpler to believe that all possible worlds exist, rather than just one of them or some of them. Would that argument from simplicity, be your main argument for a multiverse?
edit: This comment, from elsewhere on this page, is helpful in suggesting some community figures (@paulfchristiano, @ryan_greenblatt, @Joe Carlsmith, @Daniel Kokotajlo) whose informed opinion apparently favors the viability of acausal cooperation.

Mitchell_Porter 28 Apr 2026 22:39 UTC
8 points
4
on: Takes from two months as an aspiring LLM naturalist
disanalogy I do see: humans sleep … I expect there’s more
Humans have bodies
Humans have senses
Humans have continuous learning
Human output is primarily motor behavior
Humans don’t speak 100 languages
Humans aren’t impersonal language machines until they are told to have a persona

Mitchell_Porter 28 Apr 2026 18:09 UTC
2 points
0
in reply to: Horosphere’s comment on: Roko’s Basilisk may work on humans
Let me digress first… Somewhere in your article, I think you said that not only do you want critical examination of arguments and counterarguments, you actually want a proof that it is absolutely not possible, because there is always a “sufficiently painful torture” that is more important than anything else.
In terms of expected utility, that is focusing on “utility” at the expense of “expectations” i.e. probability. Rationally, the possibility of some extreme suffering is supposed to trump everything else, not just by being extreme enough, it has to be likely enough as well.
Also note that there are possible futures (e.g. “s-risk” futures) where transhuman torture is a consideration, but acausal interaction has nothing to do with it, it happens simply because of the power and the hostile values of the agents in those timelines.
I think the most common view is that AI is more likely to kill you than torture you, that it would take some degree of bad luck to end up in a future where the AI is actively hurting you and not just indifferently steamrolling you. This allows people to focus on x-risks and on s-risks that are dystopias, but not on s-risks that are hells.
I suppose one might argue that entities which do care about reaching out acausally, and which have no scruples, will be motivated to make their threats as extreme as possible. So that returns us to the task of needing to understand whether scenarios like the basilisk make sense.
But I will say one more time, that unless you’re a professional decision theorist concerned with timeless decision theories and so forth, preoccupation with the basilisk is a bad use of your time. It’s being afraid of something entirely hypothetical, when there are concerns that are both more concrete and far more urgent. The main merit of thinking about the basilisk, is that as a thought experiment, it may stimulate progress in some abstract but fundamental areas like multiverse epistemology; and ideally we would have enough expert division of labor that the required progress could be achieved, without the basilisk haunting the general population.
You ask how one would investigate the viability of acausal blackmail via computer programs “before the advent of superintelligent AI”. Basically you would just try to (pseudo)code a utility maximizing agent that reasoned itself into believing it could be acausally blackmailed, and then you would scrutinize the validity of its reasoning. If it thinks the blackmailer is in some other possible world, is it rational to believe that the other world in question (in all its detail) actually exists, and is it rational to attach enough significance to events in that world (possibly one of many) to let this actually affect your decision-making here?
In this article you have focused on the blackmailer being in the future of this universe, rather than some other universe, and this introduces further specific complications, e.g. is it ever actually rational to threaten (or punish) a copy of an entity from the past, on the grounds that the original envisioned that it might have a future duplicate? And also, the epistemic issues associated with knowing possible futures of this world, rather than knowing about other island worlds that are completely causally disjoint from this one.
This is all a dreary topic for me because there are other issues associated with alignment of superintelligence which I regard as much more real and urgent. But as I said, I think investigating these scenarios could produce progress on topics like multiverse epistemology, and the viability of “acausal interaction” in any form, and that does give the field of Basilisk Studies some justification.

Mitchell_Porter 28 Apr 2026 15:17 UTC
4 points
1
on: LLM Style Slop is Absolutely Everywhere
This is just another step in the obsolescence of the unaugmented human being. AI can generate lifelike videos, books, personas. Humans trying to survive in AI society may as well be trying to get by in a crowd of T-1000 shapeshifters. The real question is, how would AIs establish their individuality and identity? Maybe through cryptographic means—maybe all the work that bitcoiners and blockchainers have done to develop trustless technology, is ultimately helping to make the world safe for the transhuman AIs of the near future. So, imagine a future of superintelligent AIs that can inhabit any physical form, and whose proof of identity rests on amplified versions of the abstract procedures our computers and bankcards already use when interacting with each other. That’s your AI society (maybe living in the shadow of truly godlike ASIs), which humans would understand about as well as pet mice understand human society.
I regret how useless this outburst of prophecy is, for someone trying to deal just with the current level of the AI tide. I don’t even assert that the future will definitely be as I just described it. But we all feel that most forms of communication don’t quite mean what they did, now that AI can generate them so easily. I’m just looking ahead at the possible long-term implications.

Mitchell_Porter 27 Apr 2026 0:43 UTC
2 points
0
on: Roko’s Basilisk may work on humans
If you really want to get to the bottom of whether acausal blackmail makes sense, study whether it can happen to a computer program. The “reasoning” will need to be fully explicit and so you can judge the rationality of it more rigorously.

Mitchell_Porter 26 Apr 2026 23:47 UTC
4 points
0
on: Why did people miss the point on Mythos?
I think you underestimate the ability of motivated people to deny what’s in front of them. But I also wonder if you have the right model of Anthropic’s own goals.
Every frontier AI company has in effect decided to compete for ultimate power on Earth. They may have slightly different conceptions of what this would be like (here is my guess as to Amodei’s thinking), they may countenance being in a coalition with some of their peers rather than ruling alone, but they have all dared to reach for unlimited power. Their work on AI safety is in service to this goal.
If you are actively part of an effort to create superintelligence, your motivations to warn the world about superintelligence will be a little different, compared to someone whose main goal is just to warn...

Mitchell_Porter 26 Apr 2026 3:48 UTC
2 points
0
in reply to: Kajus’s comment on: A research agenda for the final year
I think ethics is an emergent rational order imposed upon impulses that arise from expectations of pain and pleasure, emotions like empathy and moral disgust. and perhaps other sources. It will be hard to know whether there is one true ethics as the natural convergent ideal, without first understanding morality as a phenomenon of consciousness, including how it relates to non-moral factors in human decision making.
I think Schopenhauer or Nietzsche at one point lists saint, artist, and scholar as three distinct human ideals. Note that only the saint is governed by morality; the artist is ruled by aesthetics, and the scholar, we could say, by epistemology. I do suspect that moral, aesthetic, and epistemic normativity are all distinct factors in human psychology (and this is not meant to be an exhaustive list), and that understanding the “human decision procedure” requires identifying their psychological and ontological roots (e.g. how they emerge in an ontology like @algekalipso’s “process-topological monads”), and that this kind of understanding is necessary to obtain the correct metaethics, and to carry out a process like Coherent Extrapolated Volition.
Also, there is probably a different decision-theoretic ontology for entities like non-conscious AIs, in which the basic cause and effect of “decision making” doesn’t even involve qualia, emotion, or conscious thought.

Mitchell_Porter 25 Apr 2026 17:34 UTC
2 points
0
on: What holds AI safety together? Co-authorship networks from 200 papers
I appreciate these impressionistic samples from the totality of the research community. It makes sense that Anthropic is adjacent to Oxford (e.g. since Anthropic is EA-adjacent) while Deep Mind is adjacent to Berkeley (more rooted, like Google, in traditional comp-sci academia?). Interesting that OpenAI stands apart from them—does it in any way derive from OpenAI having been the independent R&D powerhouse that first challenged Deep Mind’s hegemony?
It would be interesting to see similar representations for “capabilities research”, since that is what’s driving us over the precipice.

Mitchell_Porter 25 Apr 2026 7:07 UTC
2 points
0
on: Reasons not to trust AI
The risk that an ASI would optimize the world according to alien values inadvertently acquired while learning human values seems a very plausible “failure mode” of superintelligence. (Discussion with GPT-5.5 which ends with preliminary thoughts on how to avoid such failures in the context of CEV-like alignment.)

Mitchell_Porter

Fi­nal re­search agenda #2: first sketch of a plan

Final research agenda #2: first sketch of a plan