How useful would uncopyable source code be for limiting non super-intelligent AIs? Something like how humans are limited by only being able to copy resulting ideas, not the actual process that produced them. Homomorphic encryption is sort of similar, but only in the sense that the code is unreadable—I’m more interested in mechanisms to enforce the execution to be a sort of singleton. This of course wouldn’t prevent unboxing etc., but might potentially limit the damage.
I haven’t put much thought into this, but was wondering if anyone else has already gone into the feasibility of it.
Mass-self-copying is one of the mechanisms by which an infrahuman AGI might scale up its compute and turn itself into a superintelligence. Fine-tuning its own weights to produce a more-powerful successor is another strategy it might use. So if you can deny it the ability to do these things, this potentially buys a little time.
This only helps if AGI spends significant time in the infrahuman domain, if one of these strategies would been the first-choice strategy for how to foom (as opposed to social engineering or nanobots or training a successor without using its own weights), and if humanity can do something useful with time that’s bought. So it’s probably a little bit helpful, but in the best case it’s still only a small modifier on a saving throw, not a solution.
How useful would uncopyable source code be for limiting non super-intelligent AIs? Something like how humans are limited by only being able to copy resulting ideas, not the actual process that produced them. Homomorphic encryption is sort of similar, but only in the sense that the code is unreadable—I’m more interested in mechanisms to enforce the execution to be a sort of singleton. This of course wouldn’t prevent unboxing etc., but might potentially limit the damage.
I haven’t put much thought into this, but was wondering if anyone else has already gone into the feasibility of it.
Mass-self-copying is one of the mechanisms by which an infrahuman AGI might scale up its compute and turn itself into a superintelligence. Fine-tuning its own weights to produce a more-powerful successor is another strategy it might use. So if you can deny it the ability to do these things, this potentially buys a little time.
This only helps if AGI spends significant time in the infrahuman domain, if one of these strategies would been the first-choice strategy for how to foom (as opposed to social engineering or nanobots or training a successor without using its own weights), and if humanity can do something useful with time that’s bought. So it’s probably a little bit helpful, but in the best case it’s still only a small modifier on a saving throw, not a solution.
Pretty significant, and probably one big component of limiting the damage, though doesn’t rule out all failure modes.