People are certainly looking at the alternatives.
For example, DSPy (“demonstrate-search-predict”) looks very promising: https://github.com/stanfordnlp/dspy
(Their README contains Section 6 “FAQ”, which compares a variety of existing alternatives, including LangChain.)
Emmett Shear continues his argument that trying to control AI is doomed
I think that a recent tweet thread by Michael Nielsen and the quoted one by Emmett Shear represent genuine progress towards making AI existential safety more tractable.
Michael Nielsen observes, in particular:
As far as I can see, alignment isn’t a property of an AI system. It’s a property of the entire world, and if you are trying to discuss it as a system property you will inevitably end up making bad mistakes
Since AI existential safety is a property of the whole ecosystem (and is, really, not too drastically different from World existential safety), this should be the starting point, rather than stand-alone properties of any particular AI system.
Emmett Shear writes:
Hopefully you’ve validated whatever your approach is, but only one of these is stable long term: care. Because care can be made stable under reflection, people are careful (not a coincidence, haha) when it comes to decisions that might impact those they care about.
And Zvi responds
Technically I would say: Powerful entities generally caring about X tends not to be a stable equilibrium, even if it is stable ‘on reflection’ within a given entity. It will only hold if caring more about X provides a competitive advantage against other similarly powerful entities, or if there can never be a variation in X-caring levels between such entities that arises other than through reflection, and also reflection never causes reductions in X-caring despite this being competitively advantageous. Also note that variation in what else you care about to what extent is effectively variation in X-caring.
Or more bluntly: The ones that don’t care, or care less, outcompete the ones that care.
Even the best case scenarios here, when they play out the ways we would hope, do not seem all that hopeful.
That all, of course, sets aside the question of whether we could get this ‘caring’ thing to operationally work in the first place. That seems very hard.
Let’s now consider this in light of what Michael Nielsen is saying.
I am going to only consider the case where we have plenty of powerful entities with long-term goals and long-term existence which care about their long-term goals and long-term existence. This seems to be the case which Zvi is considering here, and it is the case we understand the best, because we also live in the reality with plenty of powerful entities (ourselves, some organizations, etc) with long-term goals and long-term existence. So this is an incomplete consideration: it only includes the scenarios where powerful entities with long-term goals and long-terms existence retain a good fraction of overall available power.
So what do we really need? What are the properties we want the World to have? We need a good deal of conservation and non-destruction, and we need the interests of weaker, not the currently most smart or most powerful members of the overall ecosystem to be adequately taken into account.
Here is how we might be able to have a trajectory where these properties are stable, despite all drastic changes of the self-modifying and self-improving ecosystem.
An arbitrary ASI entity (just like an unaugmented human) cannot fully predict the future. In particular, it does not know where it might eventually end up in terms of relative smartness or relative power (relative to the most powerful ASI entities or to the ASI ecosystem as a whole). So if any given entity wants to be long-term safe, it is strongly interested in the ASI society having general principles and practices of protecting its members on various levels of smartness and power. If only the smartest and most powerful are protected, then no entity is long-term safe on the individual level.
This might be enough to produce effective counter-weight to unrestricted competition (just like human societies have mechanisms against unrestricted competition). Basically, smarter-than-human entities on all levels of power are likely to be interested in the overall society having general principles and practices of protecting its members on various levels of smartness and power, and that’s why they’ll care enough for the overall society to continue to self-regulate and to enforce these principles.
This is not yet the solution, but I think this is pointing in the right direction...
Thanks, this is very interesting.
I wonder if this approach is extendable to learning to predict the next word from a corpus of texts...
The first layer might perhaps still be embedding from words to vectors, but what should one do then? What would be a possible minimum viable dataset?
Perhaps, in the spirit of PoC of the paper, one might consider binary sequences of 0s and 1s, and have only two words, 0 and 1, and ask what would it take to have a good predictor of the next 0 or 1 given a long sequence of those as a context. This might be a good starting point, and then one might consider different examples of that problem (different examples of (sets of) sequences of 0 and 1 to learn from).
This looks interesting, thanks!
This post could benefit from an extended summary.
In lieu of such a summary, in addition to the abstract
This paper introduces semantic features as a candidate conceptual framework for building inherently interpretable neural networks. A proof of concept model for informative subproblem of MNIST consists of 4 such layers with the total of 5K learnable parameters. The model is well-motivated, inherently interpretable, requires little hyperparameter tuning and achieves human-level adversarial test accuracy—with no form of adversarial training! These results and the general nature of the approach warrant further research on semantic features. The code is available at this https URL
I’ll quote a paragraph from Section 1.2, “The core idea”
This paper introduces semantic features as a general idea for sharing weights inside a neural network layer. [...] The concept is similar to that of “inverse rendering” in Capsule Networks where features have many possible states and the best-matching state has to be found. Identifying different states by the subsequent layers gives rise to controlled dimensionality reduction. Thus semantic features aim to capture the core characteristic of any semantic entity—having many possible states but being at exactly one state at a time. This is in fact a pretty strong regularization. As shown in this paper, choosing appropriate layers of semantic features for the [Minimum Viable Dataset] results in what can be considered as a white box neural network.
Is there a simple way to run against a given Kaggle competition after that particular competition is over?
These are reasonable benchmarks, but is there a way to make them less of a moving target, so that the ability to run against a given competition extends into the future?
When I ponder all this I usually try to focus on the key difficulty of AI existential safety. ASIs and ecosystems of ASIs normally tend to self-modify and self-improve rapidly. Arbitrary values and goals are unlikely to be preserved through radical self-modifications.
So one question is: what are values and goals which might be natural, what are values and goals many members of the ASI ecosystem including entities which are much smarter and much more powerful than unaugmented humans might be naturally inclined to preserve through drastic self-modifications, and which might also have corollaries which are good from viewpoints of various human values and interests?
Basically, values and goals which are likely to be preserved through drastic self-modifications are probably non-anthropocentric, but they might have sufficiently anthropocentric corollaries, so that even the interests of unaugmented humans are protected (and also so that humans who would like to self-augment or to merge with AI systems can safely do so).
So what do we really need? We need a good deal of conservation and non-destruction, and we need the interests of weaker, not the currently most smart or most powerful members of the overall ecosystem to be adequately taken into account.
What might the road towards adoption of those values (of preservation and protection of weaker entities and their interests) and incorporation of those kinds of goals be, so that these values and goals are preserved as the ASI ecosystem and many of its members self-modify drastically? One really needs the situation where many entities on varying levels of smartness and power care a lot about those values and goals.
An arbitrary ASI entity (just like an unaugmented human) cannot fully predict the future. In particular, it does not know where it might eventually end up in terms of relative smartness or relative power (relative to the most powerful ASI entities or to the ASI ecosystem as a whole). So if any given entity wants to be long-term safe, it is strongly interested in the ASI society having general principles and practices of protecting its members on various levels of smartness and power. If only the smartest and most powerful are protected, then no entity is long-term safe on the individual level.
This might be a reasonable starting point towards having values and goals which are likely to be preserved through drastic self-modifications and self-improvements and which are likely to imply good future for unaugmented humans and for the augmented humans as well.
Perhaps, we can nudge things towards making this kind of future more likely. (The alternatives are bleak, unrestricted drastic competition and, perhaps, even warfare between supersmart entities which would probably end badly not just for us, but for the ASI ecosystem as well)...
The worlds were artificial superintelligence (ASI) is coming very soon with only roughly current levels of compute, and where ASI by default goes catastrophically badly, are not worlds I believe we can afford to save.
For those kinds of worlds, we probably can’t afford to save them by heavy-handed measures (meaning that not only the price is too much to pay, given that it would have to be paid in all other worlds too, but also that it would not be possible to avoid evasion of those measures, if it is already relatively easy to create ASI).
But this does not mean that more light-weight measures are hopeless. For example, when we look at Ilya’s thought process he has shared with us last year, what he has been trying to ponder have been various potentially feasible relatively light-weight measures to change this default of ASI going catastrophically badly to more acceptable outcomes: Ilya Sutskever’s thoughts on AI safety (July 2023): a transcript with my comments.
More people should do more thinking of this kind. AI existential safety is a pre-paradigmatic field of study. A lot of potentially good approaches and possible non-standard angles have not been thought of at all. A lot of potentially good approaches and possible non-standard angles have been touched upon very lightly and need to be explored further. The broad consensus in the “AI existential safety” community seems to be that we currently don’t have good approaches we can comfortably rely upon for future ASI systems. This should encourage more people to look for completely novel approaches (and in order to cover the worlds where timelines are short, some of those approaches have to be simple rather than ultra-complicated, otherwise they would not be ready by the time we need them).
In particular, it is certainly true that
A smarter thing that is more capable and competitive and efficient and persuasive being kept under indefinite control by a dumber thing that is less capable and competitive and efficient and persuasive is a rather unnatural and unlikely outcome. It should be treated as such until proven otherwise.
So it makes sense spending more time brainstorming the approaches which would not require “a smarter thing being kept under indefinite control by a dumber thing”. What might be a relatively light-weight intervention which would avoid ASI going catastrophically badly without trying to rely on this unrealistic “control route”? I think this is underexplored. We tend to impose too many assumptions, e.g. the assumption that ASI has to figure out a good approximation to our Coherent Extrapolated Volition and to care about that in order for things to go well. People might want to start from scratch instead. What if we just have a goal of having good chances for a good trajectory conditional on ASI, and we don’t assume that any particular feature of earlier approaches to AI existential safety is a must-have, but only this goal is a must-have. Then this gives one a wider space of possibilities to brainstorm: what other properties besides caring about a good approximation to our Coherent Extrapolated Volition might be sufficient for the trajectory of the world with ASI to likely be good from our point of view?
I don’t see a non-paywalled version of this paper, but here are two closely related preprints by the same authors:
“Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features”, https://arxiv.org/abs/2212.13881
“Mechanism of feature learning in convolutional neural networks”, https://arxiv.org/abs/2309.00570
Paul Christiano: “Catastrophic Misalignment of Large Language Models”
Talk recording: https://www.youtube.com/watch?v=FyTb81SS_cs (no transcript or chat replay yet)
Talk page: https://pli.princeton.edu/events/2024/catastrophic-misalignment-large-language-models
Sam Altman watch: He invests $180 million into Retro Bio to fight aging. I have no idea if they can execute, but this kind of investment is great and bodes many good things. Kudos.
Note that this looks like a relatively old investment, it’s not another $180 million, it looks like the $180 million you might have heard about a while ago… (Gwern also thinks so.)
Jack Clark: Salary for tech staff of EU AI Office (develop and assess evaluations for gen models like big LLMs) is… 3,877 – 4,387 Euros a month, aka $4242 – $4800 USD.
Most tech co internships $8k+ a month.
He seems to be comparing an EU AI Office job with American tech salaries.
I think people tend to underestimate how much lower tech salaries tend to be in EU compared to American standards.
(Of course, things do strongly depend on the location as well, both within the EU and with the US).
Like I said, if we try to apply forceful measures we might delay it for some time (at the price of people aging and dying from old age and illnesses to the tune of dozens of millions per year due to the continuing delay; but we might think that the risks are so high that the price is worth it, and we might think that the price of everyone who is alive today eventually dying of old age is worth it, although some of us might disagree with that and might say that taking the risk of foom is better than the guaranteed eventual death of old age or other illness; there is a room for disagreement on which of these risks it is preferable to choose).
But if we are talking about avoiding foom indefinitely, we should start with asking ourselves, how easy or how difficult is it to achieve. How long before a small group of people equipped with home computers can create foom?
And the results of this analysis are not pretty. Computers are ultimate self-modifying devices, they can produce code which programs them. Methods to produce computer code much better than we do it today are not all that difficult, they are just in the collective cognitive blindspot, like backpropagation was for a long time, like ReLU activations were for decades, like residual connectivity in neural machines was in the collective cognitive blindspot for unreasonably long time. But this state of those enhanced methods of producing new computer code being relatively unknown would not last forever.
And just like backpropagation, ReLU, or residual connections, these methods are not all that difficult, it’s not like if a single “genius” who might discover them would refrain from sharing them, they would remain unknown. People keep rediscovering and rediscovering those methods, they are not that tricky (backpropagation was independently discovered more than 10 times between 1970 and 1986, before people stopped ignoring it and started to use it).
It’s just the case that the memetic fitness of those methods is currently low, just like memetic fitness of backpropagation, ReLU, and residual connections used to be low in the strangest possible ways. But this would not last, the understanding of how to have competent self-improvement in small-scale software on ordinary laptops will gradually form and proliferate. At the end of the day, we’ll end up having to either phase out universal computers (at least those which are as powerful as our home computers today) or to find ways to control them very tightly, so that people are no longer free to program their own computers as they see fit.
Perhaps, humans will chose to do that, I don’t know. Nor do I know whether they would succeed in a Butlerian jihad of this kind, or whether some of the side effects of trying to get rid of computers would become X-risks themselves. In any case, in order to avoid foom indefinitely, people will have to take drastic, radical measures which would make everyone miserable, would kill a lot of people, and might create other existential risks.
I think it’s safer if the most competent leading group tries to go ahead, that our eventual chances of survival are higher along this path, compared to the likely alternatives.
I do think that the risks on the alternative paths are very high too; the great powers are continuing to inch towards major nuclear confrontation; we are enabling more and more people to create diverse super-covid-like artificial pandemics with 30% mortality or more; things are pretty bad in terms of major risks this civilization is facing; instead of asking “what’s your P(doom)” we should be asking people, “what’s your P(doom) conditional on foom and what’s your P(doom) conditional on no foom happening”. My P(doom) is not small, but is it higher conditional on foom, than conditional on no foom? I don’t know...
I do expect a foom (via AGI or via other route), and my timelines are much shorter than 5 years.
But algorithms for AI are improving faster than hardware (people seem to quote doubling in compute efficiency approximately each 8 months), so if one simply bans training runs above fixed compute thresholds, one trades off a bit of extra time before a particular achievement vs increase of number of active players achieving it a bit later (basically, this delays the most well-equipped companies a bit and democratizes the race, which is not necessarily better).
We can make bans progressively tighter, so we can buy some time, but as the algorithms progress further, it is not unlikely that we might at some point face the choice of banning computers altogether or facing a foom. So eventually we are likely going to face huge risks anyway.
I do think it’s time to focus not on “aligning” or “controlling” self-modifying ecosystems of self-modifying super-intelligences, but on figuring out how to increase the chances that a possible foom goes well for us instead of killing us. I believe that thinking only in terms of “aligning” or “controlling” limits the space of possible approaches to AI existential safety, and that approaches not based on notions of “alignment” and “control” might be more fruitful.
And, yes, I think our chances are better if the most thoughtful of practitioners achieve that first. For example, Ilya Sutskever’s thinking on the subject has been very promising (which is why I tend to favor OpenAI if he continues to lead the AI safety effort there, but I would be much more skeptical of them otherwise).
Militaries are certainly doing that, I agree.
However, I am not sure they are creative enough and not-control-freaks enough to try to build seriously self-modifying systems. They also don’t mind spending tons of money and allocating large teams, so they might not be aiming for artificial AI researchers all that much. And they are afraid to lose control (they know how to control people, but artificial self-modifying systems are something else).
Whereas a team in a garage is creative, is short on resources and quite interested in creating a team of artificial co-workers to help them (a success in that leads to a serious recursive self-improvement situation automatically), and might not hesitate to try other recursive self-improvements schemas (we are seeing more and more descriptions of novel recursive self-improvements schemas in recent publications), so they might end up with a foom even before they build more conventional artificial AI researchers (a sufficiently powerful self-referential metalearning schema might result in that; a typical experience is that all those recursive self-improvement schemas saturate disappointingly early, so the teams will be pushing harder at them trying to prevent premature saturation, and someone might succeed too well).
Basically, having “true AGI” means being able to create competent artificial AI researchers, which are sufficient for very serious recursive self-improvement capabilities, but one might also obtain drastic recursive self-improvement capabilities way before achieving anything like “true AGI”. “True AGI” is sufficient to start a far reaching recursive self-improvement, but there is no reason to think that “true AGI” is necessary for that (being more persistent at hacking the currently crippled self-improvement schemas and at studying ways to improve them might be enough).
Both of these points precisely reflect our current circumstances.
No, there is plenty of room between the current circumstances and the bottom. We might be back to Eliezer’s “an unknown random team creates a fooming AI in a garage” old threat model, if we curtail the current high-end too much.
Just like there is plenty of room between legal pharmacy and black market for pain relievers (even when the name of the drug is the same).
It’s very easy to make things worse.
It may not even be possible to accidentally make these two things worse with regulation.
It’s probably possible. But regulation is often good, and we do need more regulation for AI.
In this post we are not talking about regulation, we are talking about prohibition, which is a different story.
a technical remark:
(2) Covered Company.
(A) In General. The term ‘”covered company’” means an entity that operates, directly or indirectly (including through a parent company, subsidiary, or affiliate), a website, desktop application, mobile application, or augmented or immersive technology application that meets all of the following criteria:
Permits user account creation:
Allows users to create an account or profile to generate, share, and view text, images, videos, real-time communications, or similar content.
Has a large user base: Has more than 1,000,000 monthly active users for at least two out of the three months preceding a relevant determination by the President under subsection (3)(B).
Enables user-generated content: Enables one or more users to generate or distribute content that can be viewed by other users of the platform.
Allows viewing of user-generated content: Enables one or more users to view content generated by other users of the platform.
(B) EXCLUSION.—The term ‘‘covered company’’ does not include an entity that operates a website, desktop application, mobile application, or augmented or immersive technology application whose primary purpose is to allow users to post product reviews, business reviews, or travel information and reviews.
So it needs to be some form of social media, of some kind: You need to have a million users, account creation and viewing and creation of user-generated content.
Note that Russian or Chinese equivalents of Gmail also seem to match. A large user base (the law does not even state that this has to be US user base), and users certainly do generate and view content.
(And yes, a foreign adversary control over e-mail servers used by people living here might easily be a natsec problem, even if there are no viral influence aspects.)
So, deliberately or accidentally, the true technical reach of the bill seems to be somewhat larger than just social media...
There are some world-destroying things that we have to ban, for now; for everything else, there’s Mastercard libertarian techno-optimism.
This seems to suggest that gradually banning and phasing out almost all computers is not on the table (which is probably a good thing, as I am “not sure” we would want to live in a computer-less society). I can imagine something as radical as gradually banning and phasing out almost all computers actually working (or not, depending on the structure of global society, and also noting that the sacrifice would be pretty bad).
But if this kind of radical measures are not on the table, I think, the main effects of a Prohibition will be similar to the effects of our drug war (the availability is lower, the quality is often pretty bad, and the potency is occasionally very high, there might indeed be an additional lag before this kind of extra high potency emerges).
So, translating to advanced AI: “the quality is often pretty bad” translates to all kinds of safety measures often being non-existent, “the potency is occasionally very high” translates to completely unregulated and uncontrolled spikes of capability (possibly including “true foom”), “there might indeed be an additional lag before this kind of extra high potency emerges” translates to indeed buying some time in exchange for the consequences down the road.
I am not sure that the eventual P(doom) is lower, if we do try to go down this road, and I would not be surprised if this would make the eventual P(doom) higher.
Time to AGI, in weeks
I like it better than the usual habit of giving this estimate in years.
Especially, with the mode of https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/ being near the border between 2026 and 2027.
Thanks! I think your discussion of the new Meaning Alignment Institute publication (the substack post and the paper) in the Aligning a Smarter Than Human Intelligence is Difficult section is very useful.
I wonder if it makes sense to republish it as a separate post, so that more people see it...