Ian Klatzco
I suspect physics sidechannels[0] will be possible for AGI to exploit until we completely solve physics, and that it may be always possible to implement weird machines[1] on physics or biology. Consider physical / biological stenography of computation. Seeking feedback / instruction / comments from physicists / biologists.
I am skeptical that security is solvable. Even if you fix memory corruption, even if you fix business logic by creating programming languages that enable you to mathematically / formally specify the behavior of your application, the interaction of your application with reality, across the silicon/reality boundary, will almost always have leaky abstractions until we thoroughly understand physics and will always fail at the human behavior / game theory / social deception / hidden preferences level.
The current economic / systemic incentives for the construction of our computer / noncomputer systems do not reward doing things “correctly” / “securely” for most use cases (notable exception: aviation but c.f. Boeing 737 MAX[2]). This is a tremendous economic liability regardless of whether or not AGI exists. There are probably useful concrete actions (design a logic programming language that is usable by most existing developers to encode business logic by writing something that resembles math, or push forward static analysis / fuzzing research to eliminate entire classes of software vulnerability).
[0] https://en.wikipedia.org/wiki/Tempest_(codename)
[1] https://en.wikipedia.org/wiki/Weird_machine
[2] https://en.wikipedia.org/wiki/Boeing_737_MAX#Grounding_and_recertification
Perhaps some of the failure modes of traditional bug bounty programs:
Underpaying bugfinders (“gig economy-ification”, versus hiring someone into a consulting firm)
Liability avoidance by firms
Deeper, more serious bugs/malicious prompts are overlooked