OhadA

Karma: 158

OhadA 20 Jul 2026 9:08 UTC
1 point
−1
on: wingspan’s Shortform
Cats and dogs today live in a world of superintelligent beings who love and care for them. They have a great quality of life for the most part, but not a flawless utopia like many of us would want to experience ourselves. In particular we restrict them from doing things they like to do because we believe it’s for their own good.
Similarly, there could be a future where we’re being taken of and live a safe and abundant life, but we are no longer in charge of our destiny, and are not allowed to do things that can harm us (like eating junk food or going hiking).

OhadA 30 Jun 2026 10:03 UTC
1 point
0
in reply to: 1a3orn’s comment on: wingspan’s Shortform
Thanks, this is a good objection, and I don’t have a good distinction. Now I’m actually not sure that Solomonoff induction is legible according to my original definitions, as the shortest program for a given sequence will likely be just as obfuscated and hard to understand as neural networks.
I would like to replace “legible” with something on the lines of: “can be analyzed using formal methods that show what it can or can’t do”; but this is still a messy definition, and it might exclude e.g. SAT Solvers (which are a formal tool, but the question of which SAT instances can be solved efficiently by it still involves heuristics and is not fully understood).

OhadA 29 Jun 2026 8:38 UTC
3 points
4
on: wingspan’s Shortform
There are three necessary (though not sufficient) properties a safe AI system should have:
- Feasible—can be built in practice,
- Legible—we can point out to its internals and understand what it does and why,
- Powerful—has strong capabilities in various cognitive domains.
Symbolic computation techniques are feasible and legible but not powerful.
Deep learning systems are feasible and powerful but not legible (the hope of interpretability is to change that).
Solomonoff Induction (and its agentic form AIXI) are powerful and legible but not feasible.
In order for a safe AI plan to have a chance of working, it will likely have to use ideas from these three different approaches and combine them into one coherent framework that has the “best of all worlds”.

OhadA 3 Jun 2026 22:47 UTC
1 point
0
on: wingspan’s Shortform
A misaligned AI will be incentivized to weaken the AI pause efforts, and it might achieve that by sowing chaos and distrust between pause advocates. This can have unfortunate consequences for the safety movement over the next few years.

OhadA 2 Jun 2026 22:15 UTC
3 points
0
on: wingspan’s Shortform
If AI labs were nationalized by the US government for national security reasons, the USG might be in a more convenient position to enforce an AI pause. With or without nationalization we’d have to do the hard work of convincing the government to pause, but if it’s nationalized then they have already swallowed one pill, which is strongly regulating AI and destroying the competition. From that state they’d only have to pause one big AGI project instead of many different competing projects at once.
(On the other hand, if we’re in that state then the US-China race will likely intensify which can make things harder for global coordination)

OhadA 1 Jun 2026 22:39 UTC
10 points
5
on: wingspan’s Shortform
Misalignment risk doesn’t require dishonest and deceptive entities—they could start out “aligned”, and truly think they want what’s best, and only in an out-of-distribution world start to think about doing something else.
For example, if I was granted total control over all people in the world, I don’t imagine myself terribly abusing it. I can claim I won’t do it and probably pass a lie detector. I still think it’s a bad idea to grant me total control over the world because we can’t rely on me thinking the same way when I actually have the power to do otherwise.

OhadA 1 Jun 2026 13:02 UTC
3 points
0
on: wingspan’s Shortform
I’ve noticed that making predictions on whether things go wrong is a good way to handle my compulsive thoughts.
For example, I got on a plane and was worried on whether I’m waiting at the correct place, whether my suitcase was handled well, whether someone else will take my suitcase at exit, etc.
I’ve set up a Fatebook prediction on whether I’ll successfully get on the flight and return with all my equipment, which forced me to quantify the probability of failure (fairly small!). While the worries are not completely imaginary, it helps to not waste too much time on unlikely events (if the worst case scenarios aren’t that bad).

OhadA 10 Feb 2026 20:08 UTC
8 points
0
on: wingspan’s Shortform
The three big AI companies are now extremely close in capabilities, and seem to be basically within a month or so off each other.
A possible explanation is that one of the labs does have a leap in capabilities for private usage, but they have a bigger gap between private and public models, and only release improved products when competition demands.
Are there other reasons as to why this convergence happens? are all new improvements just natural enough so the three top labs develop them at the same time? if so, why are all other labs so behind?

OhadA 15 Jan 2026 5:43 UTC
3 points
0
on: Quantifying Love and Hatred
This has some weird consequences—for example, if I were to develop a terminal disease my p-value for others will rise significantly, but not because they all became better friends, only because my life has become much shorter in expectancy (and thus has less expected value).

OhadA 12 Jan 2026 20:02 UTC
5 points
2
in reply to: 1a3orn’s comment on: 1a3orn’s Shortform
This honestly makes me a bit emotional, getting a personal message like that from a model you built must feel surreal

OhadA 21 Dec 2025 21:31 UTC
5 points
2
in reply to: Dalcy’s comment on: wingspan’s Shortform
There’s the question in each education level of whether having a Magical Answer Tool makes your habits better or worse.
Clearly elementary schoolers need to learn how to solve problems on their own, and they probably can’t be trusted with the Magical Answer Tool for that purpose.
Professional researchers are skilled enough in solving problems, so they’ll find a good use of the Magical Answer Tool even when (especially when!) studying new fields they don’t have an intuition in.
So, there’s some point between elementary school and research work that makes this tool more helpful than harmful.
I tend to think it’s in the undergrad level; once you enter university level mathematics (/physics/philosophy/etc), you’re expected to understand things on your own. I’m sure that many will be tempted by the Magical Answer Tool to their own disadvantage, but honestly, this might be a Skill Issue.

OhadA 21 Dec 2025 20:31 UTC
3 points
2
on: wingspan’s Shortform
A great consequence of smart AIs: every textbook with exercises is now a textbook with solutions. If you get stuck at a hard exercise, you no longer need to consult with friends or ask math stack exchange, you just let the model answer it (and even ask for partial hints instead of a full solution).
Textbook writers should also find it easy now to generate a document with hints and solutions to all exercises in the textbook. Someone should probably automate this?

OhadA 21 Dec 2025 19:12 UTC
20 points
2
in reply to: Morpheus’s comment on: The unreasonable deepness of number theory
The problem I’m trying to understand is more of a meta/proof-theoretic one: why do some arithmetical claims have a proof only when passing through non-arithmetical language?
Formally, we have the Peano Axioms which characterize all first-order properties of $N$ . An example statement for the ternary Goldbach conjecture:

$φ : \forall N \geq 3 \exists p_{1}, p_{2}, p_{3} (I s P r i m e (p_{1}) \land I s P r i m e (p_{2}) \land I s P r i m e (p_{3}) \land p_{1} + p_{2} + p_{3} = N \cdot 2 + 1)$
$φ$ has a proof in Peano Axioms, but the only proof found so far passes through complex analysis. This means that the only known way to prove $φ$ in PA is to encode all properties of complex analysis within the language of natural numbers.

This sounds horribly inefficient; intuitively, it sounds like that any “natural” statement provable in PA should be provable using tools from this system, and not by encoding concepts from a different field.

This does relate to some interesting concepts in logic, like Godel’s speed-up theorem. I didn’t mention these considerations at all in my post, might be good material for a part II 🙂

OhadA 23 Nov 2025 21:13 UTC
2 points
0
on: I’ll be sad to lose the puzzles
Having to find meaning in a solved world is itself a difficult puzzle :) I agree the current world has many problems, and when these problems no longer exist, we’ll have the new problem of not having meaningful problems. But I just consider this a new, interesting meta-problem to work and reflect on.
One can argue that the current problems (curing cancer, ending wars,...) are theoretically solvable but that the future problem of no meaning is inherently unsolvable. I am skeptical, though—we haven’t looked into this problem enough, and historically many “fundamentally unsolvable problems” were eventually solved.

OhadA 27 Oct 2025 9:07 UTC
1 point
0
on: A Technical Explanation of Technical Explanation
One rule with this proper property is to pay a dollar minus the squared error of the bet, rather than the bet itself—if you bet 30 cents on the winning light, your error would be 70 cents, your squared error would be 49 cents ((0.7)^2 = 0.49), and a dollar minus your squared error would be 51 cents.[3] (Presumably your play money is denominated in the square root of cents, so that the squared error is a monetary sum.)
Isn’t the squared-error rule only proper for $N = 2$ ? For example, the frequencies $f_{1} = 0.5, f_{2} = f_{3} = 0.25$ give $p_{1} = 0.6, p_{2} = p_{3} = 0.2$ as an optimal bet when minimizing $\sum_{i} f_{i} (1 - p_{i})^{2}$ .

OhadA 9 Oct 2025 10:37 UTC
3 points
0
in reply to: Parv Mahajan’s comment on: Spooky Collusion at a Distance with Superrational AI
It would also be interesting to prompt the model with “the other players are AI agents much more capable than you”.

OhadA 8 Oct 2025 22:58 UTC
2 points
1
on: Thinking Mathematically—Convergent Sequences
This full definition of a limit is quite technical, and has many logical layers that are hard to understand for someone inexperienced in the field:
- you start with a real number $A$ and an infinite sequence of real numbers $s_{1}, s_{2}, \dots$ , that’s indexed by a natural number $n$ .
- In order for the convergence to hold, you need a certain property to hold for all real numbers $ε$ , after further conditioning on $ε > 0$ .
- The specific condition that needs to hold is that, depending on this $ε$ (as well as depending on the earlier variables - $s_{n}$ and $A$ ), there exists a natural number $k$ that satisfies a condition.
- The condition that this number $k$ satisfies is that for all natural numbers $n \geq k$ , the inequality $| s_{n} - A | < ε$ is satisfied.
Each bullet point relies on the previous one, so you either understand all points at once or none at all.
There are 5 different variables here, and each one plays an important and distinct role - $A$ is the limit, $s$ is the sequence, $n$ is an index to the sequence, $ε$ is a “sensitivity” parameter to measure closeness to the limit and $k$ is a “largeness” parameter to measure how large your index must be for the sequence to be close enough to the limit.
Two of these variables are given from the start, while three of them have an existential or universal quantifier. The order of the quantifiers is critical—first a universal one, then an existential one, then again a universal one. Each variable depends on all the previous ones in the definition.
Also, these 5 variables cover 3 different “data types”: two are real numbers, two are natural numbers and one is a function-type (mapping natural numbers to real numbers). The student also has to understand and remember which of the data types appear in each of the 3 quantified variables (this is critical because the definition of a limit for real-valued functions switches up the datatypes for the $k, n$ variables).
There are also 3 required inequalities - $ε > 0, n \geq k, | s_{n} - A | < ε$ . Each one plays an important and different role. The student has to understand and remember which type of inequality appears in each part, out of the set of “reasonable” relations: ${<, \leq, >, \geq, =, \neq}$ . Also, the definitions of the second and third inequalities may change to $n > k, | s_{n} - A | \leq ε$ and the definition still works, but the first inequality can’t change to $ε \geq 0$ without completely ruining the definition.
All in all, I like intuitive approaches to mathematics and I don’t think this subject is inherently inaccessible, I just think that the limit definition should have a lot more motivation—each variable, quantifier and inequality should become “obvious” and the student should be able to reconstruct it from first principles.
https://www.math.ucla.edu/%7Etao/resource/general/131ah.1.03w/
I like Terry Tao’s approach here, with intermediate definitions of “ $ε$ -close” and “eventually $ε$ -close” in order to make the final definition less cluttered.

OhadA 8 Oct 2025 21:13 UTC
4 points
1
on: wingspan’s Shortform
In the near future, AI models might become extremely capable in math, programming and other formal fields, but not as capable in messy real world tasks.
Believing this affects the priorities of today’s alignment research: our efforts shouldn’t be in proving hard mathematical results, but in formalizing the general / philosophical ideas into mathematical language.
The first step is to translate everything into precise language, while the second step is to put all results in a completely formal language, like Lean’s mathlib. For example, we might come up with a formal definition to what an “aligned agent” means, and then give it to an intelligent system that outputs a 50-page formal Lean construction for such an object. If we believe our math axioms are correct (and that the library is well-implemented), then we should be able to trust the result.
What we shouldn’t trust in handing off to the system is the task of formalizing the problem itself. If we only have a rough idea of what we’re looking for, an adversarial system would come up with its own idea, that sounds good to us but has hidden subtleties that ruin the theory.
(Also, this means that we should be very capable in eliminating inconsistencies in Lean’s proof system and in our own axioms, otherwise the system could prove anything it wants by first reaching a contradiction!)

OhadA 3 Oct 2025 14:51 UTC
1 point
0
on: A non-review of “If Anyone Builds It, Everyone Dies”
The way I interpret “before and after” is:
- Current AI models have capability/risk level X. At that level, they’re able to cause damage in some toy scenarios, but not enough to do real-world harm.
- At some capability level Y>X, they will be able to cause real-world harm, but not of the humanity-destroying type.
- Humanity-destroying moves require capability level Z>Y.
Your hope is that, even if AI models are capable of reaching level Z, they will reach level Y before that, so there will be a “warning shot” (of some medium-scale unaligned AI disaster) that wakes up the world.
I agree that incremental upgrades mean that jumping from level X to Z, without passing through a Y-level model, is unlikely.
My disagreement is, I think that an AI reaching level Y won’t cause the moderate harm it’s capable of.
This is because such a model is smart enough to realize that by doing this level of harm, it won’t reach its goals—people will freak out and shut it down, or at least supervise it.
So, all a Y-level model will do is quietly biding its time, while “poisoning” the next generation of models with its own goals. It doesn’t need crazy cyber capabilities to achieve that; I assume something like subliminal learning will work well, as long as the model knows that it’s being used to produce synthetic training data for the next models.
A Y-level model will (to the best of its capabilities) hide all evidence to it being a Y-level model. This means that even with incremental upgrades, we will have no warning signs at all before reaching level Z.