Yes, and that’s OK. I suspect you can’t do qualitatively better than that (viz ambient set-theoretic universe for second-order logic), but it’s still possible (necessary?) to work under this apparent lack of absolute control over what it is you are dealing with. Even though (first order) PA doesn’t know what “integers” are, it’s still true that the statements it believes valid are true for “integers”, it’s useful that way (just as AIs or humans are useful for making the world better). It is a device that perceives some of the properties of the object we study, but not all, not enough to rebuild it completely. (Other devices can form similarly imperfect pictures of the object of study and its relationship with the device perceiving it, or of themselves perceiving this process, or of the object of study being affected by behavior of some of these devices.)
Likewise, we may fail to account for all worlds that we might be affecting by our decisions, but we mostly care about (or maybe rather have non-negligible consequentialist control over) “real world” (or worlds), whatever this is, and it’s true that our conclusions capture some truth about this “real world”, even if it’s genuinely impossible for us to ever know completely what it is. (We of course “know” plenty more than was ever understood, and it’s a big question how to communicate to a FAI what we do know.)
Yes, and that’s OK. I suspect you can’t do qualitatively better than that (viz ambient set-theoretic universe for second-order logic), but it’s still possible (necessary?) to work under this apparent lack of absolute control over what it is you are dealing with. Even though (first order) PA doesn’t know what “integers” are, it’s still true that the statements it believes valid are true for “integers”, it’s useful that way (just as AIs or humans are useful for making the world better). It is a device that perceives some of the properties of the object we study, but not all, not enough to rebuild it completely. (Other devices can form similarly imperfect pictures of the object of study and its relationship with the device perceiving it, or of themselves perceiving this process, or of the object of study being affected by behavior of some of these devices.)
Likewise, we may fail to account for all worlds that we might be affecting by our decisions, but we mostly care about (or maybe rather have non-negligible consequentialist control over) “real world” (or worlds), whatever this is, and it’s true that our conclusions capture some truth about this “real world”, even if it’s genuinely impossible for us to ever know completely what it is. (We of course “know” plenty more than was ever understood, and it’s a big question how to communicate to a FAI what we do know.)