If you press them too closely, they will abruptly fall silent, loftily indicating by some phrase that the time for argument is past.
Jean-Paul Sartre
I sincerely believe that people will get hurt if these ideas return to society at large, Richard. Please don’t do this.
The extent to which modern governments hand out money causes a similar effect across western societies. They would not be as able to do this as much if their currencies were still on the gold standard, because it would be more obvious that they are insolvent.
modern governments
western societies
gold standard
insolvent
I’m afraid, if you’re actually trying to advance an argument, you’re going to need to be slightly more specific. Which government, in which society? Perhaps you will say, “all of the western governments”. Very well. Then which gold standard? I will remind you that “western societies” including Germany, the UK, and the US exited and entered the “gold standard”, or swapped out various forms of “gold standard”, all throughout the 20th century. Was Britain in 1931 or Germany in 1914 lacking in valor? I’m by no means an economic historian, and this is just wikipedia talking.
And even when countries obeyed the “gold standard”, policies varied strongly between regions, including how much currency could be redeemed for how much gold, whether citizens could redeem gold directly, the status of gold as a commodity, the trade of gold between countries etc. In what combination of policies can we find your notion of “honest work for honest pay”? Does virtue’s just reward include the ability to hold gold sovereigns as private property to be redeemed at any time at a national bank, or to exchange gold bullion across national borders? If so, at what exchange rates?
And what does “insolvent” mean? After world war II the world’s currencies were arranged such that the US dollar was the world’s reserve currency as part of the Bretton Woods agreement. The US obtained an exorbitant privilege, in that it could print paper and get money while other countries needed to produce goods or services. Was this the point at which the US became, as you say, “insolvent”? Yet it was the paramount superpower at the time, and the value of its currency was (supposedly) backed by gold—until it decided to break from the gold standard under Nixon and still more or less retained that privilege. But was America’s massive spending during the post world war II period and the Cold War (all powered directly or indirectly by this exorbitant privilege) a sign of degeneracy and decay? Surely not.
Okay, let us limit ourselves to the post-cold war, post-bretton woods period. To identify exactly where you think this massive amount of spending rewarding the non-virtuous is going, you’re going to need to leave the realm of abstract metaphor and actually state what you think your virtues are. Because the way I see it, in America social programs and government pensions are both now and in the past consistently some of the largest sources of federal spending. This is, as I understand it, also true in most European nations. Perhaps the “other people who are rewarded for slacking off” you are referring to are the poor, the sick, the young, and the elderly? In that case you should say so directly.
There are so many more issues missing from your view of labour, economics, virtue, and social organisation that I would be doing most theories a disservice to call it a theory. It is telling that, for example, you make no mention of oil in your comment. Oil a resource which has become so vital to the value of currency between nations that the US dollar is sometimes referred to as the petrodollar (since it is the international currency by which oil is priced). And if the societies of today are truly “insolvent”, I suspect that you will find it hard to find historical examples of “solvent” societies with which to compare them. National banks, international debt, and modern nation states as you would recognise them did not truly arise until maybe the 19th century. Before that, the fundamental notion of currency and obligation was entirely different in Europe, since most people lived in a feudal system. Serfs and people bonded to the land paid taxes not in gold but in goods, and did not receive payment for their work in currency either. We are living in possibly the first time in history that arguments about national solvency can be made, and economic data on this issue doesn’t really go back more than a century at best.
But there is a deeper issue at stake here. Why am I hounding you so much on these issues? From where do I get my sudden insistence on exactitude and rigour? After all, we should beware isolated demands for rigour. However, you are not raising these ideas as independent thoughts, but as a cluster. A cluster which points to a poorly defined yet pervasive moral and social degeneracy in modern society. A cluster which uses a vision of historic and traditional virtue as a high ideal, but cannot quite articulate what exact virtues it is chasing after. A cluster that makes vague and incoherent attempts to summarise and criticise modern institutions and economic systems. A cluster that invokes an explicitly racially and ethnically coded “people” as its core group, to be defended from “the others”, groups who are both culturally incompatible and an existential threat if they are allowed to integrate. A cluster that denigrates public servants, university intellectuals, and cultural elites as hopelessly perverse and degenerate in their cosmopolitanism. A cluster that frames the fear of difference as wise and justified. Yes, I will use that word.
I will repeat again, at the risk of sounding a bit like a broken record, that every time ethnonationalism is embraced the groups that are othered become framed as an existential threat to the majority. Calls for forcible relocation and elimination of the other in the name of the health and hygiene of the community become unavoidable. This is a pattern that worked its way through Europe during the two world wars, through the Partition of India and the later rise of the BJP, through the Rwandan and Rohingya Genocides, and through the Middle East today. The situation in Europe and America is already volatile. You are not being clever when you fan the flames. You are not so in control of the narrative that you can add just that touch of ethnic pride back into the meme pool to precisely steer society to an optimal maxima. People will get hurt, lose their homes and families, and most likely die.
To be honest, this post approaches a level of disjointedness from reality-as-I-understand-it that I fear I cannot accurately respond to it in a way that would satisfy the author.[1] However, if I don’t give some response, I suspect that it will join a growing cultural current within LessWrong which involves an intellectualised rationalisation of ethnonationalism, hygiene-oriented eugenics, oligarchy[2], and implicit or explicit support of political action and political violence for these ends. If this current becomes normalised (and it has, frankly, always been present in the rationalist sphere), it makes TESCREAL-style categorisations of the rationalist/EA/AI safety intellectual sphere bascially correct. Therefore, I want to commit to voicing my objection to this line of argument where I see it. There are parts of me that feel strongly against doing this. There is no glory to be gained in political arguments online. However, there is the possibility of avoiding shame, which as a virtue ethicist I’m sure the author will understand.
On the question of inherent racial or ethnic attributes, I suspect that I will not be able to settle the argument “on the facts”, given how long the online debate has been waged. Therefore I will cede as much as he wishes to stake “on the facts”. Despite ceding this ground, I still maintain my fundamental belief that all human beings (and, increasingly I would argue, all life) are worthy of a deep and sincerely-held respect, regardless of any intellectual or physical discrepancies between them. This is not a utilitarian argument, but a deontological or virtue-oriented one, where the basis is one of universal love and compassion. No one should be deprived of a socially and physically healthy standard of living because of the circumstances of their birth or genetics. Nor are people’s wants and needs more or less legitimate due to their intelligence, culture, belief or any other factor.
The arguments about inherent and unassailable differences between cultures presuppose what they aim to prove. They assert that, if culture is taken as an immutable, static prior that cannot be changed without being “diluted” or even “destroyed”, that cultures from different parts of the world are incompatible. Well, yes, if you presuppose that change is death, then change cannot happen, or else death follows. This attitude goes against any reasonable study of the past and of human society. So many of the words and ideas that are powering this conversation and this forum: “culture”, “virtue”, “algorithm”, even fundamental ideas like “zero”—are borrowed and integrated from other cultures. I will remind you again that it was not so long ago that, for example, French culture was considered utterly incompatible with or simply inferior to English culture, just as American culture was to British culture, as Polish culture was to American culture, or as Danish culture was to Swedish culture. So many differences that were once a matter of life and death quietly disappeared once the cultural battle lines shifted. We humans and all life forms in this universe are united by common world-models borne from within-lifetime observation of the world and bound by common laws of physics and information. Yes, shocks can happen from sudden changes in demographics. Yes, it is possible for tensions to arise and for relative needs between groups to be unmet, and therefore for conflict between groups to grow. But the missing mood is that of sorrow, because it is possible to do better. If it is not possible to do better...
The natural result of talk of immutable differences is life-or-death stakes. If they are different and irreconcilable to us, when we come into conflict with them we must destroy them, or otherwise secure some total strategic advantage. After all, this is game-theoretic conflict now, where we know that the outcome is explicitly zero-sum. Both parties cannot be convinced to cooperate and have incompatible terminal utility functions. The Nash equilibrium then is to defect.
And the longer they live and grow strong, the greater the threat is to us. For if in the future they secure some total strategic advantage, they would do the rational thing and wipe us out, seizing all future resources for themselves. I do not seek to exaggerate when I say that this is the argument that leads to final solutions and wars of hygiene. This is the Melian argument, brought to modern day geopolitics.
[...] and some treachery taking place inside, the Melians surrendered at discretion to the Athenians, who put to death all the grown men whom they took, and sold the women and children for slaves, and subsequently sent out five hundred colonists and inhabited the place themselves.
Thankfully, I believe that it is, in fact, possible to do better. That we have not found better coordination mechanisms is an engineering and social construction problem, not a problem of the failure to enact sufficiently powerful boundaries. Insofar as toxic egalitarianism and the managerial revolution have failed, their failures come from not understanding how to successfully coordinate under increasingly complex social and technical conditions, not from any inherent disdain of virtue or race to adopt increasingly ludicrous positions.
It is true, managing a global civilisation of 8.2 billion people is hard. But the author’s “solutions” of ethnic segregation and socially-enforced virtue ethics enforced by inter-group mutual distrust come from historical societies where the effective politically-active population is measured (at most) in the thousands to hundreds of thousands, and those societies were all more or less self-sufficient and autarkic. For better or for worse, we live in a society with globally connected dependencies in matters of trade, manufacturing, business, and culture. Our scientific, technological, and medical progress is dependent on research and collaboration that stretches across borders. We are already deeply reliant on massive AI systems (incl. Google and Facebook) to manage our civilisational information throughput, and we are becoming more and more tightly integrated as supply chains lengthen and cross-cultural relationships deepen. At the same time, the threats we face of advanced AI systems, global thermonuclear war, pandemics, climate change etc. also do not confine themselves to a single society or region. There is no easy way back to the old system of walls and borders. In light of this I can see no good argument for bringing back the demons of the last century today.
For the most obvious example, for the life of me I cannot understand how leaving the gold standard makes a culture less appreciative of any kind of moral virtue, unless you equate two very different senses of the word “value”.
E.g. under the name of e.g. “rule by the most qualified” and “rule by the most virtuous”. The line of argument descends through Moldbug, who has since become more happy to openly support fascism and ad-hoc oligarchy under Trump post-2024.
[Footnote removed, I judged it to be overly combative and not constructive]
I suppose another way to look at this is the overlapping components are the blanket states in some kind of time dependent markov blanket setup, right?
In the scenario you created you could treat as the some shielded state at time step , so . Then are states outside of the blanket, so (which group of states is and which is don’t really matter, so long as they are on either side of the blanket). [1]become , and become .
Then becomes the blanket such that
and
With all that implies. In fact you can just as easily have three shielded states, or four, using this formulation.
(the setup for this is shamelessly ripped off from @Gunnar_Zarncke ’s unsupervised agent detection work)
Did you miss an arrow going to ?
the parameters are technically real numbers and thus have infinity bits of storage capacity
Yeah, a lot of the stuff about ARNNs having superturing computing capacity rely on the Analog part of the (A)RNN and the weights being therefore real valued. RNNs with rational weights are strictly turing complete I think. (cf. this article)
If every bit of every weight were somehow used to store one bit of p, excepting those weights used to simulate the UTM, that should suffice to derive the conjecture, yes
It’s theoretically fine if our scheme is kind of inefficient in terms of how much code it can store in a given number of parameters
Assuming that the bits to parameters encoding can be relaxed, there’s some literature about redundant computations in neural networks. If the feature vectors in a weight matrix aren’t linearly independent, for example, the same computation can be “spread” over many linearly dependent features, with the result that there are no free parameters but the total amount of computational work is the same.
Otherwise, I don’t think it’ll be easy find a way for the remaining parameters to be fully free to vary, since it is a densely connected network. It might be interesting for you to look into the lottery ticket hypothesis literature, though even there I don’t think the remaining non-ticket weights can be set to any value.
My view is that the normal functioning of a structure like the LTBT or a board would be dramatically insufficient for governing transformative AI (boards normally have a much weaker function in practice than the ostensible purposes of the LTBT and the Anthropic board), so I’m not very satisfied by “the LTBT is behaving how a body of this sort would/should normally behave”.
I basically completely agree. For a related intuition pump, I have very little confidence that auditing AI capabilities company will meaningfully assist in governing transformative AI.
Just to clarify, if there was some way to “fill up” the weights of the RNN/Transformer such that no weights are “free”, that would satisfy the conditions that you’ve set up? As in, if the encoding of takes up every parameter, that would qualify as a valid solution for the purposes of ? (Forgive me, I’m still very new to AIT and the associated work, and my knowledge is still very poor)
This is so cool! I really like the idea and the exploration of it.
First of all, this is great stuff and very clearly written.
With regards to this:
Now, we’re going to modify the setup from the previous section, bounding our induction to make it computable. Instead of running programs, whereis drawn from the set of all bit strings of length, we restrict ourselves to programs of lengththat require a maximum ofspace and a maximum oftime steps.
Specifically, we construct a new bounded UTMfrom our UTM.only hascells of work tape available. If the head moves to the last cell of the work tape,immediately halts. Further,always halts itself afterexecution steps if it has not halted yet. This bounded induction will no longer be optimal in the sense of having an SI-style bound on its total error that depends only on the entropy and Kolmogorov complexity of the data-generating process, becausemight not be computable in timeand space.
And the open question
What kind of neural network architectures have a prior equivalent to that of a bounded universal Turing machine?
It seems intuitive to me that what you’re describing is an RNN. Recalling Andrei Karpathy’s writing on the unreasonable effectiveness of RNNs:
Moreover, as we’ll see in a bit, RNNs combine the input vector with their state vector with a fixed (but learned) function to produce a new state vector. This can in programming terms be interpreted as running a fixed program with certain inputs and some internal variables. Viewed this way, RNNs essentially describe programs. In fact, it is known thatRNNs are Turing-Completein the sense that they can to simulate arbitrary programs (with proper weights). But similar to universal approximation theorems for neural nets you shouldn’t read too much into this. In fact, forget I said anything.
If training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs.
More directly: Instead of bit-string programs with length T, we have the parameters of the RNN. The “work tape” with fixed length s+1 can be compared to the hidden state vector of the RNN, which also has a fixed length. The enforced halting condition is similar to running an RNN for t steps.
Edited to add: I see that you also noted the ability of RNNs to simulate UTMs with regards to conjecture 2. Sorry for the somewhat redundant comment. However, you noted that you didn’t succeed in proving conjecture 2 wrt transformers and chain of thought. Did you succeed for RNNs?
I’ve left a comment under Schulman’s comment that maybe explains slightly more.
This BOTEC attitude makes sense when you view the creation of AI technology and ai safety as a pure result of capital investment. The economic view of AI abstracts the development process to a black box which takes in investments as input and produces hardware/software out the other end. However, AI is still currently mostly driven by people. A large part of enabling AI development comes from a culture around AI, including hype, common knowledge, and social permissibility to pursue AI development as a startup/career pathway.
In that regard “AI safety people” starting AI companies, writing hype pieces that encourage natsec coded AI races, and investing in AI tech contribute far more than mere dollars. It creates a situation where AI safety as a movement becomes hopelessly confused about what someone “concerned about AI safety” should do with their life and career. The result is that someone “concerned about AI safety” can find groups and justifications for everything from protesting outside OpenAI to working as the CEO of OpenAI. I think this is intrinsically linked to the fundamental confusion behind the origins of the movement.
In short, whatever material and economic leverage investment plays produce may not be worth the dilution of the ideas and culture of AI safety as a whole. Is AI safety just going to become the next “ESG”, a thin flag of respectability draped over capabilities companies/racing companies?
Glad to hear that you found Witness relevant to your work. I would be very happy to discuss further since it appears we do have a strong area of overlap with regards to AI for governance and decentralised governance. My contact is on my website. I also understand if you are already quite overloaded with work and requests to connect, in which case I’m happy that you took the time to respond to my comment :)
The fund manages 2 billion dollars, and (from the linked webpage on their holdings) puts the money into chipmakers and other AI/tech companies. The money will be used to fund development of AI hardware and probably software, leading to growth in capabilities.
If you’re going to argue that 2 billion AUM is too small an amount to have an impact on the cash flows of these companies and make developing AI easier, I think you would be incorrect (even openai and anthropic are only raising single digit billions at the moment).
You might argue that other people would invest similar amounts anyways, so its better for the “people in the know” to do it and “earn to give”. I think that accelerating capabilities buildouts to use your cut of the profits to fund safety research is a bit like an arsonist donating to the fire station. I will also note that “if I don’t do it, they’ll just find somebody else” is a generic excuse built on the notion of a perfectly efficient market. In fact, this kind of reasoning allows you to do just about anything with an arbitrarily large negative impact for personal gain, so long as someone else exists who might do it if you don’t.
I keep checking back here to see if people have responded to this seemingly cut and dry breach of promise by the leadership, but the lack of commentary is somewhat worrying.
Both the ARC-AGI replication and GPT-5′s strong performance on agentic evals* have moved me away from there being a chance of a very rapid rise for a new AI paradigm. I expect my concrete predictions to be less likely to be true. However, I still stand by my original point behind this post: it is not a particular model but a line of research which is of concern, and I still think the research could bear fruit in unexpected and not-priced-in ways.
* - Including cyber!! The X-Bow report has really not been discussed here that much, people just seem to take openai’s word for GPT-5 not being a big step ahead in agentic threat models
Oh, damn. I feel so… Sad. For everyone. For the people who they once were, before moloch ate their brains. For us, now staring into the maw of the monster. For the world.
Hi, thanks for writing this up. As someone who did their dissertation on the motivations behind AI development, I came to the conclusion that our world’s information gathering and coordiantion mechanisms were already inextricably tied to AI usage. Therefore it is very heartening to see a healthy and cooperative vision for AI applications that is not focused on power-seeking and instrumental convergence. Contra to other commenters, I think that such designs do meaningfully reduce the risk of rogue AI systems, mostly because they would be built out of constrained AI systems that are inherently less agentic. The improved coordination capacity for society would also help us rein in unconstrained racing towards AGI in the name of capitalist or international competition. This was my most recent attempt at the problem of “AI for governance/coordination” that I produced for a hackathon, and I’m glad to hear others are also working on similar problems!
Heh, more evidence for the “modelling leads to empathy” thesis, I suppose. Even for non-immediate conspecifics, advanced modelling and theory of mind brain tech seems to lead to reuse of system 1 signals (empathy). This is (I would guess) good news for the alignment problem.
I think this is a replay of the contrast I mentioned here of “static” vs “dynamic” conceptions about AI. To the author of the original post, AI is an existing technology that has taken a particular shape, so its important to ask what harms that shape might cause in society. To AI safety folk, the shape is an intermediate stage and rapidly changing into a world ending superbeing, so asking about present harms (or, indeed, being overly worried about chatbot misalignment) is a distraction from the “core issue”.
Guessing the intention of others can also help predict future actions. “Alice who was court ordered to donate bed nets and doesn’t want to go to prison” vs. “Alice the EA who has read MacAskill cover to cover” have very different projected patterns of future behaviour, especially if the context changes (e.g. the court order is completed). Notably, that doesn’t stop them from both clicking the same buy button on Amazon today.