gworley
Karma: 2,244
NewTop
A developmentally-situated approach to teaching normative behavior to AI
It seems plausible, yes. Certainly I expect militaries to build stronger surveillance capabilities and those will in part be based on using AI to make sense of the large corpus of data; whether or not they will use those for AI policing purposes seems highly uncertain to me at the present but seems like something people in the AI policy space could reasonable push for if they think it will be a worthwhile intervention (I try to refrain from speculating on beneficial policy directions too much myself since it’s not a space I feel I understand enough about to not make recommendations that will blow up in my face for unexpected reasons).
I have no reason to believe that they are since they are primarily focused on signal intelligence and until very recently the key insights here all seemed to come from number theory, measure theory, linear algebra, and a few other fields of mathematics. Maybe the situation has changed some in the last 5 years, but last I checked they were still primarily interested in recruiting talented pure mathematicians, not the sorts of folks with engineering and programming expertise necessary to build AI.
Thinking about your goal of improving how people evaluate randomized trials, a powerful way to gain attention and avoid a similar fate to Sheps’s work would be to apply it to produce a surprising and useful result that would otherwise not be possible. An equally good approach might be to use it to overturn what is currently conventional wisdom by reinterpreting the existing data to show it actually proves the opposite of what has currently been concluded.
I don’t know enough about the relevant fields to make any recommendations in that area, but maybe another reader here will have some suggestions. I think asking for suggestions in the wider EA community might also be a good approach since folks working on global poverty, for example, may already know a problem where your approach would be useful.
Such a secret government AI organization could take advantage of the openness in the field of AI, as it could absorb information about the advances of others, but would not be not legally obliged to share its own achievements. Thus, it would always outperform the current state of public knowledge. Governmental organizations have used this type of advantage before to dominate in cryptography.
This is a small issue that doesn’t affect your overall point, but I previously worked in cryptography so know a bit about what was happening with the state of the art, and generally we consider the NSA to be 20-30 years ahead of publicly available research and that the NSA mathematicians mostly ignore academia because it’s behind (although a few folks watch to keep tabs on what’s become public and so no longer has a plausible secrecy advantage).
Further, the NSA and CIA have a history of both directly discouraging publication of results that will make known math that was previously powering secret cryptographic technology (see the history of RSA for an example) and encouraging research in directions that they already know to be flawed but will result in exploitable technology for a 20-30 year period (see elliptic-curve cryptography).
So as it turns out the NSA doesn’t seem to benefit at all from academia, and instead mostly tries aggressively to stay ahead of it, keep its results secret, and do what it can to interfere with public knowledge to assure it maintains a strategic advantage.
If I read this right it seems to come down, at it’s core, to building a narrow AI with advanced surveillance capabilities that can be used to direct the application of conventional force to gain a generic strategic advantage that could be used, among other things, to police AGI development.
Does that seem like a fair summary for cramming it all into a single sentence?
This seems to cut through a lot of confusion present in decision theory, so I guess the obvious question to ask is why don’t we already work things this way instead of the way they are normally approached in decision theory?
Alternatively, you could say that the AI is an abstract computational process which is trying achieve certain results in another abstract computational process (the toy world) that it is interfacing with.
I expect there to be significant difficulties in achieving this in practice. We can design theoretical models that provide isolation, but implementation details often result in leaks in the abstraction. To give a simple example I know about, quantum cryptography is theoretically “perfect”, but the first implementations had a fatal flaw due to the realities of successfully transmitting qubits over the wire which introduced opportunities for attacks because the qubits were not be represented as single particles but by multiple particles, permitting a “siphoning” attack. We see similar issues around classical computer memory (in the form of row-hammer attacks) and spinning disks (in the form of head-skip attacks). Given even a relatively low-powered AI was able to accidentally create a radio receiver (although in a seemingly not well sandboxed environment, to be fair), we should expect something aiming for superintelligence to not be much hindered by trying to present it with a toy version of the world since the toy world will almost certainly be leaky even if we don’t notice the leaks ourselves.
Ultimately I think you’ll encounter a difficulty here due to epistemic circularity: you’ll eventually need to know about a distribution you can’t go more meta on because it would be functionally equivalent to solving the problem of the criterion, discovering completely the universal prior, grounding induction in general, etc.. Not that we don’t always have to deal with it, just that in particular I don’t expect going meta to help much beyond reducing the number of free variables you have to consider. That being said, getting the number of free variables you have to think about down is helpful, but you’ll still be left with them.
Even worse, in some sense we are hoping that the AI isn’t smart enough to learn the true explanation for the training examples, which is that humans picked them.
Ah right, I don’t often think about this but it’s a good point and a likely source of Goodharting as systems become more capable.
What is happening in this post is that we are taking the expected value of events, by way of the conditional expected value of the random variable (conditioning on the event).
...and I was enlightened. Assuming this is correct (it fits with how I read this post and a couple others), this seems like a much better way to explain what’s going on with probutility.
I suspect dealing adequately with contradictions will be significantly more complicated than you propose, but haven’t written about that in depth yet. When I get around to addressing what I view as necessary in this area (practicing moral particularism that will be robust to false positives) I definitely look forward to talking with you more about it.
Thinking a bit more, it seems a big problem we may face in using model fragments is that they are fragments and we will have to find a way to stitch them together so that they fill the gaps between the models, perhaps requiring something like model interpolation. Of course, maybe this isn’t necessary if we think of fragments as mostly overlapping (although probably inconsistent in the overlaps) or of new fragments to fill gaps as available on demand if we discover we need them and don’t have them.
Herein I’m thinking about this and the sequel post and trying to understand why you might be interested in this since it doesn’t feel to me like you spell it out.
It seems we might care about model fragments if we think we can’t build complete models of other agents/things but can instead build partial models. The “we” building these models might be literally us, but also an AI or a composite agent like humanity. Having a theory of what to do with these model fragments is useful if we want to address at least two questions, then, that we might be worried about around these parts: how do we decide an AI is safe based on our fragmentary models of it, and how does an AI model humanity based on its fragmentary models of humans.
To me this sort of approach feels like a non-starter because you’re ignoring the thing that generates the policy in favor of the policy itself, which would seem to expose you to Goodharting that would be even worse than the Goodharting we expect in terms of values since policy is a grosser instrument. Is there some way in which you think this is not that case, namely that focusing on policy alignment would help us better avoid Goodharting than is possible with value alignment?
Perhaps, but I think that kind of use of the word “objective” only makes sense in a context where we can reclaim it from it’s normal meaning. I realize such a thing has happened within Bayesianism, but it causes significant confusion for the uninitiated reader.
Further there is someone to do this observing and know that they are seeing a librarian or a sailor. There is no “objective” unless you shove the observer outside the frame of reference so you can pretend to get objectivity.
Robustness to fundamental uncertainty in AGI alignment
What I read Qiaochu as saying is that the IRL model doesn’t have an ontology, and the world it lives in is one created by the ontology the programmer implicitly constructs for it based on choices about training data. Thus this problem doesn’t come up because the IRL model isn’t interacting with the whole world; only the parts of it the programmer thought relevant to solving the problem, and the model succeeds in part by how good a job the programmer did in picking what’s relevant.
By the way, Rohin, feel free to mention me by name if you want instead of user name in these things. I’d like my name on LW to be the new style that allows your name with spaces but I’m for now stuck with my username from the beginning off time so I don’t break my history and karma.