There are just some weeks where everything happens
OhadA
Karma: 187
The recursiveness of recursive self improvement is hard to wrap my head around. It’s not just building an AI that builds better AI, it’s building an AI that builds better AI-building AI, and so on. The more layers you expand in the recursion the more powerful it becomes. Also, building an AI that builds AI-building AI is a distinct goal than (say) building an AI that builds medicine-AI or math-AI. Banning the former seems more important than banning the latter.
...Slowly, then all at once
1122 AI company employees have signed this statement asking the US government to help prepare for an AI slowdown:
https://www.pacingthefrontier.com/We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
Sentinel’s Global Risk Watch just released this week’s analysis, which is marked as yellow (🟡) due to the OpenAI / Hugging Face breach:
https://blog.sentinel-team.org/p/openai-model-breaches-containmentThey have posted weekly roundups for over two years, and today marks the first time that an AI-related event comes with a warning. The previous yellow-level alerts were only for geopolitical events like the Iran war.
Pretty significant!
Cats and dogs today live in a world of superintelligent beings who love and care for them. They have a great quality of life for the most part, but not a flawless utopia like many of us would want to experience ourselves. In particular we restrict them from doing things they like to do because we believe it’s for their own good.
Similarly, there could be a future where we’re being taken of and live a safe and abundant life, but we are no longer in charge of our destiny, and are not allowed to do things that can harm us (like eating junk food or going hiking).
Thanks, this is a good objection, and I don’t have a good distinction. Now I’m actually not sure that Solomonoff induction is legible according to my original definitions, as the shortest program for a given sequence will likely be just as obfuscated and hard to understand as neural networks.
I would like to replace “legible” with something on the lines of: “can be analyzed using formal methods that show what it can or can’t do”; but this is still a messy definition, and it might exclude e.g. SAT Solvers (which are a formal tool, but the question of which SAT instances can be solved efficiently by it still involves heuristics and is not fully understood).
There are three necessary (though not sufficient) properties a safe AI system should have:
Feasible—can be built in practice,
Legible—we can point out to its internals and understand what it does and why,
Powerful—has strong capabilities in various cognitive domains.
Symbolic computation techniques are feasible and legible but not powerful.
Deep learning systems are feasible and powerful but not legible (the hope of interpretability is to change that).
Solomonoff Induction (and its agentic form AIXI) are powerful and legible but not feasible.In order for a safe AI plan to have a chance of working, it will likely have to use ideas from these three different approaches and combine them into one coherent framework that has the “best of all worlds”.
A misaligned AI will be incentivized to weaken the AI pause efforts, and it might achieve that by sowing chaos and distrust between pause advocates. This can have unfortunate consequences for the safety movement over the next few years.
If AI labs were nationalized by the US government for national security reasons, the USG might be in a more convenient position to enforce an AI pause. With or without nationalization we’d have to do the hard work of convincing the government to pause, but if it’s nationalized then they have already swallowed one pill, which is strongly regulating AI and destroying the competition. From that state they’d only have to pause one big AGI project instead of many different competing projects at once.
(On the other hand, if we’re in that state then the US-China race will likely intensify which can make things harder for global coordination)
Misalignment risk doesn’t require dishonest and deceptive entities—they could start out “aligned”, and truly think they want what’s best, and only in an out-of-distribution world start to think about doing something else.
For example, if I was granted total control over all people in the world, I don’t imagine myself terribly abusing it. I can claim I won’t do it and probably pass a lie detector. I still think it’s a bad idea to grant me total control over the world because we can’t rely on me thinking the same way when I actually have the power to do otherwise.
I’ve noticed that making predictions on whether things go wrong is a good way to handle my compulsive thoughts.
For example, I got on a plane and was worried on whether I’m waiting at the correct place, whether my suitcase was handled well, whether someone else will take my suitcase at exit, etc.
I’ve set up a Fatebook prediction on whether I’ll successfully get on the flight and return with all my equipment, which forced me to quantify the probability of failure (fairly small!). While the worries are not completely imaginary, it helps to not waste too much time on unlikely events (if the worst case scenarios aren’t that bad).
The three big AI companies are now extremely close in capabilities, and seem to be basically within a month or so off each other.
A possible explanation is that one of the labs does have a leap in capabilities for private usage, but they have a bigger gap between private and public models, and only release improved products when competition demands.
Are there other reasons as to why this convergence happens? are all new improvements just natural enough so the three top labs develop them at the same time? if so, why are all other labs so behind?
This has some weird consequences—for example, if I were to develop a terminal disease my p-value for others will rise significantly, but not because they all became better friends, only because my life has become much shorter in expectancy (and thus has less expected value).
This honestly makes me a bit emotional, getting a personal message like that from a model you built must feel surreal
There’s the question in each education level of whether having a Magical Answer Tool makes your habits better or worse.
Clearly elementary schoolers need to learn how to solve problems on their own, and they probably can’t be trusted with the Magical Answer Tool for that purpose.
Professional researchers are skilled enough in solving problems, so they’ll find a good use of the Magical Answer Tool even when (especially when!) studying new fields they don’t have an intuition in.
So, there’s some point between elementary school and research work that makes this tool more helpful than harmful.
I tend to think it’s in the undergrad level; once you enter university level mathematics (/physics/philosophy/etc), you’re expected to understand things on your own. I’m sure that many will be tempted by the Magical Answer Tool to their own disadvantage, but honestly, this might be a Skill Issue.
A great consequence of smart AIs: every textbook with exercises is now a textbook with solutions. If you get stuck at a hard exercise, you no longer need to consult with friends or ask math stack exchange, you just let the model answer it (and even ask for partial hints instead of a full solution).
Textbook writers should also find it easy now to generate a document with hints and solutions to all exercises in the textbook. Someone should probably automate this?
The problem I’m trying to understand is more of a meta/proof-theoretic one: why do some arithmetical claims have a proof only when passing through non-arithmetical language?
Formally, we have the Peano Axioms which characterize all first-order properties of . An example statement for the ternary Goldbach conjecture:
has a proof in Peano Axioms, but the only proof found so far passes through complex analysis. This means that the only known way to prove in PA is to encode all properties of complex analysis within the language of natural numbers.
This sounds horribly inefficient; intuitively, it sounds like that any “natural” statement provable in PA should be provable using tools from this system, and not by encoding concepts from a different field.
This does relate to some interesting concepts in logic, like Godel’s speed-up theorem. I didn’t mention these considerations at all in my post, might be good material for a part II 🙂
Having to find meaning in a solved world is itself a difficult puzzle :) I agree the current world has many problems, and when these problems no longer exist, we’ll have the new problem of not having meaningful problems. But I just consider this a new, interesting meta-problem to work and reflect on.
One can argue that the current problems (curing cancer, ending wars,...) are theoretically solvable but that the future problem of no meaning is inherently unsolvable. I am skeptical, though—we haven’t looked into this problem enough, and historically many “fundamentally unsolvable problems” were eventually solved.
I’ve seen a common take that AI can’t be the great filter of advanced civilizations, because misaligned AI will expand throughout the universe and we would have seen its effects by now.
I’m not convinced, as an AI doesn’t have to be superintelligent to wipe out its creators.
It can be capable enough to synthesize a virus eliminating all life on the planet, but still not be capable enough to recurse itself to ASI and/or conquer the galaxy.
This misaligned AI can even be myopic and not consider the long-term consequences of its actions; it could see the existing biological life as an immediate threat, eliminate them, and then die out soon later when there’s no one to maintain its servers. Cancer cells do something similar.