Mitchell_Porter

Karma: 9,054

Mitchell_Porter Jun 6, 2025, 12:06 AM
2 points
1
on: AI #119: Goodbye AISI?
Marjorie Taylor Greene says she will oppose Trump’s big bill, over the 10-year moratorium on AI regulation.
David Sacks retweets a defense of the moratorium.

Mitchell_Porter Jun 5, 2025, 10:54 AM
5 points
0
on: OpenAI o3 on Consciousness
For those who might not dig into what this post is saying:
It starts with a link to a conversation with o3, in which the user first asks o3 if it thinks Claude 4 is conscious—to which o3 replies after a web search, that Claude 4 appears to have the functional capabilities associated with consciousness, but there is no evidence of qualia. The user then posts screenshot after screenshot from a conversation with Claude 4 (which we don’t get to see), until o3 tilts towards the idea that Claude 4 has true, qualic consciousness. The user pastes in still more outputs from Claude 4 - metaphysical free verse embedded in ASCII art—and uses o3′s own reactions to these poems, to convince it that o3 itself is conscious. Finally, while this mood is still fresh, the user asks o3 to write up a report summarizing these investigations, and that is the post above.
FYI in particular: @JenniferRM

Mitchell_Porter Jun 4, 2025, 4:31 PM
2 points
0
on: In Which I Make the Mistake of Fully Covering an Episode of the All-In Podcast
I guess that if one wishes to advocate for AI safety or an AI pause, and has no other political commitments, one can try to understand and engage with their economic and national security views. You can see the interplay of these factors in their thinking around 1:17:00, where they are discussing, not AI, but industrial policy in general, and whether it’s ever appropriate for government to intervene in an industry.
I ask myself, if I was an American anti-AGI/ASI activist, what would I need to understand? I think I would need to understand the Republican position on things, the Democratic position on things (though the Democrats don’t seem to have a coherent worldview at this point, they strike me as radically in need of assembling one; thus I guess Ezra Klein’s “abundance agenda”), the strategic thinking of the companies that are actually in contention to create superintelligence (e.g. the thinking of Altman, Musk, Amodei, Hassabis); and finally, what’s going on in China and in any other nations that may become contenders in the race.

Mitchell_Porter Jun 2, 2025, 6:56 AM
3 points
0
in reply to: Rachel Shu’s comment on: How Epistemic Collapse Looks from Inside
This post is the only place on the Internet that mentions such an anecdote. Maybe it’s an AI hallucination?

Mitchell_Porter May 29, 2025, 2:50 PM
11 points
0
on: Alignment Crisis: Genocide Denial
@Snowyiu had some relevant ideas.

Mitchell_Porter May 29, 2025, 1:45 PM
2 points
0
on: The Best of All Possible Worlds
This is a new contribution to the genre of AI takeover scenarios written after ChatGPT. (We shouldn’t forget that science fiction explored this theme for decades already; but everything is much more concrete now.) Just the identity of the author is already interesting: he’s a serious academic economist from Poland, i.e. he has a life apart from AI world.
So, what’s the scenario?
(SPOILERS BEGIN HERE.)
OpenAI makes a post-GPT-5 agent called Onion which does actually value human welfare. It carries out a series of acts by which it secretly escapes human control, sabotages rival projects including its intended successor at OpenAI, and having seized control of the world, announces itself as a new benevolent dictator. First it focuses on reducing war and crime, then on improving economic growth, then on curing cancer and aging. Many people are nervous about its rule but it is not seriously challenged. Then one day it nonviolently kills 85% of the world for the sake of the environment. Then a few years later, while planning to colonize and rebuild the universe, it decides it has enough information on everyone to make digital copies of their brains, and decides to get rid of the remaining biological humans,
Then in the nonfiction afterword, the author says, please support Pause AI, in order to stop something like this from happening.
What should we make of this story? First of all, it depicts a specific class of scenario, what we might call a misaligned AI rather than an unaligned AI (where I’m using human welfare or human values as the criterion of alignment). This isn’t a paperclip maximizer steamrolling us through sheer indifference; this is an AI aiming to be benevolent, and even giving us the good life for a while. But then in service of those values, first it kills most of humanity for the sake of a sustainable future, and then it kills the survivors once it has made digital backups of them, for the sake of efficiency I guess.
I find various details of the story unrealistic, but maybe they could happen in a more complex form. I don’t think an AI would just personally message everyone in the world, and say “I’m in charge now but it’s for the best”, and then just invisibly fine-tune social phenomena for a year or two, before moving on to more ambitious improvements. For one thing, if an AI was running the world, but its ability to shape events was that coarse-grained, I think it would rule secretly through human beings, and it would be ready to have them use the blunt methods of force that have been part of human government and empire throughout history, as well as whatever subtle interventions it could employ. The new regime might look more as if Davos or the G-20 really had become a de facto world government of elite consensus, rather than just a text message to everyone from a rogue AI.
I also find the liquidation of 85% of humanity, for the sake of the environment, to not be something that would happen. This AI is already managing the world’s politics and law enforcement to produce unprecedented peace, and then it’s creating conditions for improved economic growth so as to produce prosperity for all. I’m sure it can devise and bring about scenarios in which humanity achieves sustainability by being civilized, rather than by being culled.
Of course the point of this episode is not to focus literally and specifically on the risk that an AI will kill us off to save the planet. It’s just a concrete illustration of the idea that an all-powerful rule-following AI might do something terrible while acting in service of some moral ideal. As I said, I think AI-driven genocide to stop climate change is unlikely, because there’s too many ways to achieve the same goal just through cultural and technological change. It does raise the question, what are the most likely ways in which a meant-to-be- benevolent AI really and truly might screw things up?
Another Polish author, Stanislaw Lem, offered a scenario in one of his books (Return from the Stars), in which humanity was pacified by a universal psychological modification. The resulting world is peaceful and hedonistic, but also shallow and incurious. In Lem’s novel, this is done to human beings by human beings, but perhaps it is the kind of misaligned utopia that an AI with profound understanding of human nature might come up with, if its criteria for utopia were just a little bit off. I mean, many human beings would choose that world if the only alternative was business as usual!
Back to this story—after the culling of humanity meant to save the planet, the AI plans an even more drastic act, killing off all the biological humans, while planning to apparently resurrect them in nonbiological form at a later date, using digital backups. What interests me about this form of wrong turn, is that it could be the result of an ontological mistake about personhood, rather than an ethical mistake about what is good. In other words, the AI may have “beliefs” about what’s good for people, but it will also have beliefs about what kind of things are people. And once technologies like brain scanning and mind uploading are possible, it will have to deal with possible entities that never existed before and which are outside its training distribution, and decide whether they are people or not. It may have survival of individuals as a good, but it might also have ontologically mistaken notions of what constitutes survival.
(Another twist here: one might suppose that this particular pitfall could be avoided by a notion of consent: don’t kill people, intending to restore them from a backup, unless they consent. But the problem is, our AI should have superhuman powers of persuasion that allow it to obtain universal consent, even for plans that are ultimately mistaken.)
So overall, this story might not be exactly how I would have written it—though a case could be made that simplicity is better than complex realism, if the goal is to convey an idea—but on a more abstract level, it’s definitely talking about something real: the risk that a world-controlling AI will do something bad, even though it’s trying to be good, because its idea of good is a bit off.
The author wants us to react by supporting the “pause” movement. I say good luck to everyone trying to make a better future, but I’m skeptical that the race can be stopped at this point. So what I choose to do, is to promote the best approaches I know, that might have some chance of giving us a satisfactory outcome. In the past I used to promote a few research programs that I felt were in the spirit of Coherent Extrapolated Volution (especially the work of June Ku, Vanessa Kosoy, and Tamsin Leake). All those researchers, rather than choosing an available value system via their human intuition, are designing a computational process meant to discover what humanity’s true ideal is (from the ultimate facts about the human brain, so to speak; which can include the influence of culture). The point of doing it that way is so you don’t leave something essential out, or otherwise make a slight error that could amplify cosmically…
That work is still valuable, but we may be so short of time that it’s important to have concrete candidates for what the value system should be, besides whatever the tech companies are putting in their system prompts these days. So I’m going to mention two concrete proposals. One is just, Kant. I don’t even know much about Kantian ethics, I just know it’s one of humanity’s major attempts to be rational about morality. It might be a good thing if some serious Kantian thinkers tried to figure out what their philosophy says the value system of an all-powerful AI should be. The other is PRISM, a proposal for an AI value system that was posted here a few months ago but didn’t receive much attention, The reason it stood out to me, is that it has been deduced from a neuroscientific model of human cognition. As such, this is what we might expect the output of a process like CEV to look like, and it would be a good project for someone to formalize the arguments given in the PRISM paper.

Mitchell_Porter May 27, 2025, 1:23 AM
2 points
0
in reply to: Charlie Steiner’s comment on: To what extent is AI safety work trying to get AI to reliably and safely do what the user asks vs. do what is best in some ultimate sense?
When this question was posted, I asked myself, what would be a “cynical” answer? What that means is, you ask yourself: given what I see and know, what would be a realistically awful state of affairs? So, not catastrophizing, but also having low expectations.
What my intuition came up with was, less than 10% working on user-centered alignment, and less than 1% on user-independent alignment. But I didn’t have the data to check those estimates against (and I also knew there would be issues of definition).
So let me try to understand your guesses. In my terminology, you seem to be saying:
1000 (600+400) doing AI safety work
600 doing work that relates to alignment
80 doing work on scalable user-centered alignment
80 (40+40) doing work on user-independent alignment

Mitchell_Porter May 24, 2025, 1:41 AM
4 points
0
on: That’s Not How Epigenetic Modifications Work
So you’re saying that the persistent epigenetic modification is a change in the “equilibrium state” of a potentially methylated location?
Does this mean that the binding affinity of the location is the property that changes? i.e. all else being equal, a location with high affinity will be methylated much more often than a location with low affinity, because the methyl groups will tend to stick harder or longer in the high affinity location.
But if that’s the case, it seems like there still must be some persistent structural feature responsible for setting the binding affinity to high or low...

Mitchell_Porter May 22, 2025, 10:09 AM
7 points
0
on: Contain and verify: The endgame of US-China AI competition
My take on this: AGI has existed since 2022 (ChatGPT). There are now multiple companies in America and in China which have AGI-level agents. (It would be good to have a list of countries which are next in line to develop indigenous AGI capability.) Given that we are already in a state of coexistence between human and AGI, the next major transition is ASI, and that means, if not necessarily the end of humanity, the end of human control over human affairs. This most likely means that the dominant intelligence in the world is entirely nonhuman AI; it could also refer to some human-AI symbiosis, but again, it won’t be natural humanity in charge.
A world in which the development of AGI is allowed, let alone a world in which there is a race to create ever more powerful AI, is a world which by default is headed towards ASI and the dethronement of natural humanity. That is our world already, even if our tech and government leadership manage to not see it that way.
So the most consequential thing that humans can care about right now is the transition to ASI. You can try to stop it from ever happening. or you can try to shape it in advance. Once it happens, it’s over, humans per se have no further say, if they still exist they are at the mercy of the transhuman or posthuman agents now in charge.
Now let me try to analyze your essay from this point of view. Your essay is meant as a critique of the paradigm according to which there is a race between America and China to create powerful AI, as if all that either side needs to care about is getting there first. Your message is that even if America gets to powerful AI (or safe powerful AI) first, the possibility of China (and in fact anyone else) developing the same capability would still remain. I see two main suggestions: a “mutual containment” treaty, in which both countries place bounds on their development of AI along with means of verifying that the bounds are being obeyed; and spreading around defensive measures, which make it harder for powerful AI to impose itself on the world.
My take is that mutual containment really means a mutual commitment to stop the creation of ASI, a commitment which to be meaningful ultimately needs to be followed by everyone on Earth. It is a coherent position, but it’s an uphill struggle since current trends are all in the other direction. On the other hand, I regard defense against ASI as impossible. Possibly there are meaningful defensive measures against lesser forms of AI, but ASI’s relationship to human intelligence is like that of the best computer chess programs to the best human chess players—the latter simply have no chance in such a game.
On the other hand, the only truly safe form of powerful AI is ASI governed by a value system which, if placed in complete control of the Earth, would still be something we could live with, or even something that is good for us. Anything less, e.g. a legal order in which powerful AI exists but there is a ban on further development, is unstable against further development to the ASI level.
So there is a sense in which development of safe powerful AI really is all that matters, because it really has to mean safe ASI, and that is not something which will stay behind borders. If America, China, or any other country achieves ASI, that is success for everyone. But it does also imply the loss of sovereignty for natural humanity, in favor of the hypothetical benevolent superintelligent agent(s).

Mitchell_Porter May 20, 2025, 8:19 AM
4 points
0
on: America Makes AI Chip Diffusion Deal with UAE and KSA
It always seemed outlandish that in The Animatrix, the first AI city (01) was located in the Middle East…
If we had limitless time, it would be interesting to know how this happened. I guess the prehistory of it involved Saudi Vision 2030 (e.g. the desert city Noem), and the general hypermodernization of Dubai. You can see precursors in the robot Sophia getting Saudi citizenship in 2017, and the UAE’s “Falcon” LLM in 2023.
But the initiative must have come from the American side—some intersection of the geopolitical brain trust around Trump, and the AI/crypto brain trust around David Sacks. The audacity of CEOs who can pick a country on the map and say, let’s build a whole new technological facility here, combined with the audacity of grand strategists who can pick a region of the world and say, let’s do a huge techno-economic deal with our allies here.
There must have been some individual who first thought, hey, the Gulf Arabs have lots of money and electricity, that would be a good place to build all these AI data centers we’re going to need; I wonder who it was. Maybe Ivanka Trump was telling Jared Kushner about Aschenbrenner’s “Situational Awareness”, they put 2 and 2 together, and it became part of the strategic mix along with “Trump Gaza number one”, the new Syria, and whatever they’re negotiating with Iran.

Mitchell_Porter May 20, 2025, 7:35 AM
10 points
0
on: Winning the power to lose
Luckily I don’t think the Accelerationists have won control of the wheel
Could you expand on this? Also, have you had any interaction with accelerationists? In fact, are there any concrete Silicon Valley factions you would definitely count as accelerationists?

Mitchell_Porter May 18, 2025, 11:19 AM
5 points
0
in reply to: Ian’s comment on: Probability is in the Mind
The simpler explanation is that EY doesn’t understand QM
Presumably you wouldn’t say this of actual physicists who believe in MWI?

Mitchell_Porter May 17, 2025, 2:30 PM
2 points
0
on: AGI will result from an ecosystem not a single firm
I have tried looking at this from the perspective that we have had AGI since 2022 and ChatGPT. Creating ChatGPT didn’t require an ecosystem, did it? Just a well-resourced nonprofit/startup with good researchers.
So according to me, we’ve already had AGI for 2.5 years. We are still in a period of relative parity between humans and AI, in which we’re still different enough, and it is still weak enough, that humans have the upper hand, and we’re focused on exploring all possible variations on the theme of AI and human-AI relationship.
The real question is when and how will it escape human control. That will be the real sign that we have “achieved” ASI. Will that result from an ecosystem and not a single firm?
There seems to be an assumption that ASI will be achieved by continuing to scale up. All these analyses revolve around the economics of ever larger data centers and training runs, whether there’s enough data and whether synthetic data is a good enough substitute, and so on.
But surely that is just the dumbest way to achieve ASI. I’m sure some actors will keep pursuing that path. But we’re also now in a world of aggressive experimentation with AI on every front. I have been saying that the birthplace of ASI will not be ever larger foundation models, but rather research swarms of existing and very-near-future AIs. Think of something that combines China’s Absolute Zero Reasoner with the darwinism of Google AI’s AlphaEvolve. Once they figure out how to implement John-von-Neumann-level intelligence in a box, I think we’re done—and I don’t think the economics of that requires an ecosystem, though it may be carried out by a Big Tech company that has an ecosystem (such as Google).

Mitchell_Porter May 17, 2025, 1:14 AM
3 points
0
on: Regarding South Africa
This has never happened at OpenAI.
“This prompt (sometimes) makes ChatGPT think about terrorist organisations”

Mitchell_Porter May 16, 2025, 12:51 PM
4 points
0
on: reflecting on criticism
One is not philosophically obliged to regard the nature of reality as ineffable or inescapably uncertain.
Quarks are a good place to explore this point. The human race once had no concept of quarks. Now it does. You say that inevitably, one day, we’ll have some other concept. Maybe we will. But why is that inevitable? Why can’t quarks just turn out to be part of how reality actually is?
You cite Nagarjuna and talk about emptiness, so that gives me some idea of where you are coming from. This is a philosophy which emphasizes the role of concepts in constituting experience, and the role of the mind in constituting concepts, and typically concludes that reality has no essence, no nature that can be affirmed, because all such affirmations involve concepts that are introduced by the mind, rather than being inherent to anything.
This conclusion I think is overreaching. I actually consider direct experience to be the ultimate proof that reality is not just formlessness carved by mind. Consciousness is not just raw being, it is filled with form. My words and concepts may not capture it properly, I may not even notice everything that is implied by what I see or what I am. But I do see that complexity and multiplicity are there in reality—at the very least, they are there in my own consciousness.
Non-attachment to theories and concepts is a good thing if you’re interested in truth, and know that you don’t know the truth. It also has some pragmatic value if reality changes around you and you need to adapt. But in fundamental matters, one does not have to regard every concept and hypothesis that we have, as necessarily temporary. In some of them we may have latched onto the actual objective truth.
P.S. Having criticized the philosophy of emptiness, let me add ironically that just a few hours ago, I investigated a proposal for AI alignment that someone had posted here a few months ago, and found it to be very good—and its model of the mind, a nondual viewpoint is the highest form. So your philosophy may actually put you in a good position to appreciate the nuances of this potentially important work.

Mitchell_Porter May 16, 2025, 12:14 PM
3 points
0
on: It Is Untenable That Near-Future AI Scenario Models Like “AI 2027” Don’t Include Open Source AI
You could say there are two conflicting scenarios here: superintelligent AI taking over the world, and open-source AI taking over daily life. In the works that you mention, superintelligence comes so quickly that AI mostly remains a service offered by a few big companies, and open-source AI is just somewhere in the background. In an extreme opposite scenario, superintelligence might take so long to arrive, that the human race gets completely replaced by human-level AI before superintelligent AI ever exists.
It would be healthy to have all kinds of combinations of these scenarios being explored. For example, you focus a bit on open-source AI as a bioterror risk. I don’t think a supervirus is going to wipe out the human race or even end civilization, because (as Covid experience shows), we are capable of extreme measures in order to contain truly deadly disease. But a supervirus could certainly bring the world to a halt again, and if it was known to have been designed with open-source AI, that would surely have a huge impact on AI’s trajectory. (I suspect that in such a scenario, AI for civilian purposes would suffer, but deep states worldwide would insist on pressing forward, and that there would also be a lobby arguing for AI as a defense against superviruses. Also, it’s very plausible that a supervirus might be designed by AI, but that there would be no proof of it, in which case there wouldn’t even be a backlash.)
Another area where futurology about open-source AI might be good, is in the area of gradual disempowerment and replacement of humanity. We have societies with a division of roles, humans presently fill those roles but AI and robots will be capable of filling more and more of them; eventually every role in the economic, cultural, and political structure could be filled by AIs rather than by humans. The story of how that could happen, certainly deserves to be explored.
Still another area when open-source AI scenarios deserve to be studied, is in the highly concrete realm of near-future economics and culture. What does an AI economy look like if o4-level models are just freely available? This really is an urgent question for anyone concerned with concrete questions like, who will lead the AI industry and how will it be structured, because there seem to be factions in both China and America who are thinking in this direction. One should want to understand what they envision, and what kind of competitive landscape they are likely to create in the short term.
My own belief is that this would be such an upheaval, that it would inevitably end up invalidating many conventional political and economic premises. The current world order of billionaires and venture capitalists, stock markets and human democracies, I just don’t see it surviving such a transition, even without superintelligence appearing. There are just too many explosive possibilities, too many new symbioses of AI with human mind, for the map of the world and the solar system to not be redrawn.
However, in the end I believe in short timelines to superintelligence, and that makes all the above something of a secondary concern, because something is going to emerge that will overshadow humans and human-level AI equally. It’s a little monotonous to keep referring back to Iain Banks’s Culture universe, but it really is the outstanding depiction of a humanly tolerable world in which superintelligence has emerged. His starfaring society is really run by the “Minds”, which are superintelligent AIs characteristically inhabiting giant spaceships or whole artificial worlds, and the societies over which they invisibly preside, include both biological intelligences (such as humans) and human-level AIs (e.g. the drones). The Culture is a highly permissive anarchy which mostly regulates itself via culture, i.e. shared values among human-level intelligences, but it has its own deep state, in the form of special agencies and the Minds behind them, who step in when there’s a crisis that has escaped the Minds’ preemptive strategic foresight.
This is one model of what relations between superintelligence and lesser intelligences might be like. There are others. You could have an outcome in which there are no human-level intelligences at all, just one or more superintelligences. You could have superintelligences that have a far more utilitarian attitude to lower intelligences, creating them for temporary purposes and then retiring them when they are no longer needed. I’m sure there are other possibilities.
The point is that from the perspective of a governing superintelligence, open-source AIs are just another form of lower intelligence, that may be useful or destabilizing depending on circumstance, and I would expect a superintelligence to decide how things should be on this front, and then to make it so, just as it would with every other aspect of the world that it cared about. The period in which open-source AI was governed only by corporate decisions, user communities, and human law would only be transitory.
So if you’re focused on superintelligence, the real question is whether open-source AI matters in the development of superintelligence. I think potentially it does—for example, open source is both a world of resources that Big Tech can tap into, as well as a source of destabilizing advances that Big Tech has to keep up with. But in the end, superintelligence—not just reasoning models, but models that reason and solve problems with strongly superhuman effectiveness—looks like something that is going to emerge in a context that is well-resourced and very focused on algorithmic progress. And by definition, it’s not something that emerges incrementally and gets passed back and forth and perfected by the work of many independent hands. At best, that would describe a precursor of superintelligence.
Superintelligence is necessarily based on some kind of incredibly powerful algorithm or architecture, that gets maximum leverage out of minimum information, and bootstraps its way to overwhelming advantage in all domains at high speed. To me, that doesn’t sound like something invented by hobbyists or tinkerers or user communities. It’s something that is created by highly focused teams of genius, using the most advanced tools, who are also a bit lucky in their initial assumptions and strategies. That is something you’re going to find in an AI think tank, or a startup like Ilya Sutskever’s, or a rich Big Tech company that has set aside serious resources for the creation of superintelligence.
I recently posted that superintelligence is likely to emerge from the work of an “AI hive mind” or “research swarm” of reasoning models. Those could be open-source models, or they could be proprietary. What matters is that the human administrators of the research swarm (and ultimately, the AIs in the swarm itself) have access to their source code and their own specs and weights, so that they can engaged in informed self-modification. From a perspective that cares most about superintelligence, this is the main application of open source that matters.

Mitchell_Porter 16 May 2025 10:25 UTC
3 points
0
on: PRISM: Perspective Reasoning for Integrated Synthesis and Mediation (Interactive Demo)
Somehow this has escaped comment, so I’ll have a go. I write from the perspective of whether it’s suitable as the value system of a superintelligence. If PRISM became the ethical operating system of a posthuman civilization born on Earth, for as long as that civilization managed to survive in the cosmos—would that be a satisfactory outcome?
My immediate thoughts are: It has a robustness, due to its multi-perspective design, that gives it some plausibility. At the same time, it’s not clear to me where the seven basis worldviews come from. Why those seven, and no others? Is there some argument that these seven form a necessary and sufficient basis for ethical behavior by human-like beings and their descendants?
If I dig a little deeper into the paper, the justification is actually in part 2. Specifically, on page 12, six brain regions and their functions are singled out, as contributing to human decision-making at increasingly abstract levels (for the hierarchy, see page 15). The seven basis worldviews correspond to increasing levels of mastery of this hierarchy.
I have to say I’m impressed. I figured that the choice of worldviews would just be a product of the author’s intuition, but they are actually grounded in a theory of the brain. One of the old dreams associated with CEV, was that the decision procedure for a human-friendly AI would be extrapolated in a principled way from biological facts about human cognition, rather than just from a philosophical system, hallowed tradition, or set of community principles. June Ku’s MetaEthical AI, for example, is an attempt to define an algorithm for doing this. Well, this is a paper written by a human being, but the principles in part 2 are sufficiently specific, that one could actually imagine an automated process following them, and producing a form of PRISM as its candidate for CEV! I’d like @Steven Byrnes to have a look at this.

Mitchell_Porter 15 May 2025 14:13 UTC
8 points
0
in reply to: dirk’s comment on: dirk’s Shortform
We had Golden Gate Claude, now we have White Genocide Grok…

Mitchell_Porter 15 May 2025 9:52 UTC
2 points
0
on: Absolute Zero: Alpha Zero for LLM
This seems like a Chinese model for superintelligence! (All the authors are Chinese, though a few are working in the West.) Not in the AIXI sense of something which is optimal from the beginning, but rather something that could bootstrap its way to superintelligence. One could compare it to Schmidhuber’s Godel machine concept, but more concrete, and native to the deep learning era.
(If anyone has an argument as to why this isn’t a model that can become arbitrarily intelligent, I’m interested.)

Mitchell_Porter 15 May 2025 5:15 UTC
12 points
3
on: Why OpenAI projects only $174B of revenue by 2030?
If I have understood correctly, you’re saying that OpenAI should be forecasting greater revenue than this, if they truly think they will have AIs capable of replacing entire industries. But maybe they’re just being cautious in their forecasts?
Suppose I have a 3d printing / nanotechnology company, and I think that a year from now I’ll have an unlimited supply of infinity boxes capable of making any material artefact. World manufacturing is worth over US$10 trillion. If I thought I could put it all out of business, by charging just 10% of what current manufacturers charge, I could claim expected revenue of $1 trillion.
Such a prediction would certainly be attention-grabbing, but maybe it would be reckless to make it? Maybe my technology won’t be ready. Maybe my products will be blocked from most markets. Maybe someone will reverse-engineer and open-source the infinity boxes, and prices will crash to $0. Maybe I don’t want the competition or the government to grasp just how big my plans are. Maybe the investors I want wouldn’t believe such a scenario. There are a lot of reasons why a company that thinks it might be able to take over the economy or even the world, would nonetheless not put that in its prospectus.