If the unchageability is implemented by storing the code in ROM, then it would be pretty easy for the AI to get around that by copying its code over to RAM, making the desired changes, and then running the modified code. (Presumably on other hardware, since the AI’s original hardware can only accept instructions from the ROM.) Even if the system is very cleverly designed, and the AI has no way at all to access its own source code, it would always have the option of writing an entirely new AI from scratch with the same goal function. It’s slightly easier to improve an existing AI design than create an entirely new one, but a sufficiently smart AI would be able to do both. The key ingredient in a FOOM is that each AI in the chain is capable of building a smarter one, not that the source code of the next AI is derived from the source code of the current one.
If the unchageability is implemented by storing the code in ROM, then it would be pretty easy for the AI to get around that by copying its code over to RAM, making the desired changes, and then running the modified code. (Presumably on other hardware, since the AI’s original hardware can only accept instructions from the ROM.) Even if the system is very cleverly designed, and the AI has no way at all to access its own source code, it would always have the option of writing an entirely new AI from scratch with the same goal function. It’s slightly easier to improve an existing AI design than create an entirely new one, but a sufficiently smart AI would be able to do both. The key ingredient in a FOOM is that each AI in the chain is capable of building a smarter one, not that the source code of the next AI is derived from the source code of the current one.