Another note : why would the AI touch this layer at all? Actual prototype autonomy systems (my day job) there are device drivers, an RTOS, many hardware details. A surprising amount of complexity for a machine that’s only role is to execute some graph on input X and produce control output Y and logs Z.
Most of the improvements you might make are changing the nodes and structure of the graph. There will normally be no need or benefit to changing the graph execution infrastructure. (Nodes are the actual neural networks or algorithms that choose from the outputs the one to use per some rule or choose which boxes in an image detector are probable, and so on. An AGI would presumably be an enormous graph with thousands of nodes)
Growing AI systems may need more—more hardware, of a newer generation—but they won’t need to touch how it works as there would be no benefit in speed but most changes would cause it to outright fail. So not useful to optimize.
So yes, flaws of the class above could be hidden for years. There are ways to find them by decompiling and analyzing the bytecode but an AI wouldn’t necessarily find such a flaw in itself.
Another note : why would the AI touch this layer at all? Actual prototype autonomy systems (my day job) there are device drivers, an RTOS, many hardware details. A surprising amount of complexity for a machine that’s only role is to execute some graph on input X and produce control output Y and logs Z.
Most of the improvements you might make are changing the nodes and structure of the graph. There will normally be no need or benefit to changing the graph execution infrastructure. (Nodes are the actual neural networks or algorithms that choose from the outputs the one to use per some rule or choose which boxes in an image detector are probable, and so on. An AGI would presumably be an enormous graph with thousands of nodes)
Growing AI systems may need more—more hardware, of a newer generation—but they won’t need to touch how it works as there would be no benefit in speed but most changes would cause it to outright fail. So not useful to optimize.
So yes, flaws of the class above could be hidden for years. There are ways to find them by decompiling and analyzing the bytecode but an AI wouldn’t necessarily find such a flaw in itself.