The Fatal AGI Hardware Gap

[jrincayc]

Restricting computer hardware has been proposed as a solution to preventing the dangers of building superintelligent AI. For example, MIRI suggests restricting hardware fabricated on 28 nanometer process node processes or smaller or hardware with 15,840 TFLOP/​s or greater computing capacity ^[1]

There is a catch with restricting hardware however, that I have not seen discussed elsewhere. Essentially, there is a minimum level of hardware required to create an AGI which we can call . If the hardware is below this level (which will actually be a frontier depending on multiple things like memory, compute speed and available I/​O), the hardware will be incapable of AGI, if it is below the limit. If it is above, the hardware is capable of AGI.

Now, let us suppose that the AGI is ethical, in that it can correctly choose actions that do not harm other beings in the universe. We can list another limit, for the minimum level of hardware required to create an ethical AGI.

Now compare them. Imagine we have a minimum ethical AGI. We can make the AGI smaller and simpler by deleting things like how to figure out which atoms are currently being used by other beings, and instead just grab whatever atoms we can. So it seems quite likely that . ^[2] This means that there is a gap between the two, and if a hardware limit is set in this gap, the probability that an AGI is unethical is 100% ^[3] , because ethical AGIs are not possible below .

In short, if we try and prevent AGI by using hardware limits, we need to make sure we are not in the gap between and because that would actually move into a region where any AGI is always fatal. ^[4] Above the probability of an AGI being unethical is less, but I am not sure what direction the probability goes with increasing computational power (increases, decreases, or remains constant seem possible to me). ^[5]

1. ↩︎

   https://​​ifanyonebuildsit.com/​​treaty

2. ↩︎

   seems impossible, since an ethical AGI is an AGI. might be possible if all (or at least the lowest resource using) AGI’s figure out ethics more or less automatically. might also be possible if ethics are easy to add an existing AGI.

3. ↩︎

   At least if the hardware limit is completely enforced.

4. ↩︎

   Note that the MIRI limits seem more likely to be set to prevent superintelligence, rather than AGI. Eliezer Yudkowsky, Steve Byrnes, and myself have all estimated that current high end personal computers (or lower) probably can do AGI: https://​​intelligence.org/​​2022/​​03/​​01/​​ngo-and-yudkowsky-on-scientific-reasoning-and-pivotal-acts/​​ , https://​​www.lesswrong.com/​​posts/​​LY7rovMiJ4FhHxmH5/​​thoughts-on-hardware-compute-requirements-for-agi , https://​​www.researchgate.net/​​publication/​​388398902_Memory_and_FLOPS_Hardware_limits_to_Prevent_AGI

5. ↩︎

   For clarification, this is P(unethical AGI | AGI created). P(AGI created) seems very likely to increase with higher compute being available.

[Davidmanheim]

I don’t understand the argument. If the limit is above that needed for ASI, we’re dead. The fact that it could also be below or above the level of aligned ASI doesn’t change anything.

[jrincayc]

If you are suggesting that P(unethical ASI | random ASI created) = ~1, then yes, in that case, I agree, this does not matter.

If however, there is some probability of surviving a randomly created ASI, say 10%, then in this case, there is a difference between setting the administrative limit above and below it. If it is above, then we have a chance of surviving the creation of the AGI, but if the administrative limit is below , then we have no chance of surviving.

The problem is, we don’t actually know where is (or for that matter ). So we make a guess, and set an administrative limit. If we want to be safer, we ordinarily would think that lower administrative limits are safer, but based on the arguments in my post, I think there are times where lowering the administrative limit can make it less safe (if we lower the limit below but not below we would have been safer with the higher administrative limit).

[Davidmanheim]

If however, there is some probability of surviving a randomly created ASI...

But that’s the claim that’s obviously not correct. If we are drawing randomly from the distribution, the odds are infinitesimal.

[StanislavKrym]

The main problem is that rapidly DECREASES over time. As far as I understand MIRI’s position, MIRI believesthat current tech has an infinite^[1] because nobody can (yet?) make the AGI ethical. Suppose, alternatively, that current tech is such that is a thousand times more than , while the developments of mechinterp allow us to have reach just twice . Then a blanket ban wouldn’t be THAT useless.

1. ^

   Or impractically large, like simulating literal human brains. But MIRI didn’t mention such a pathway, opting for calls to make the humans smarter.

[jrincayc]

Hm, I think we are using in different ways. I think of as an actual physical constant. Unfortunately, we don’t know what it actually is, and determining it may not be possible.

I agree that when we actually know how to create an ethical AGI, things would be quite different.

Update: As in is the minimum possible, no matter what technology is used. This does make it rather hard to determine.