Independent AI safety researcher
alexflint
Karma: 2,211
Knowledge is not just mutual information
Thank you for this comment duck_master.
I take your point that it is possible to extract knowledge about human affairs, and about many other things, from the quantum structure of a rock that has been orbiting the Earth. However, I am interested in a definition of knowledge that allows me to say what a given AI does or does not know, insofar as it has the capacity to act on this knowledge. For example, I would like to know whether my robot vacuum has acquired sophisticated knowledge of human psychology, since if it has, and I wasn’t expecting it to, then I might choose to switch it off. On the other hand, if I merely discover that my AI has recorded some videos of humans then I am less concerned, even if these videos contain the basic data necessary to constructed sophisticated knowledge of human psychology, as in the case with the rock. Therefore I am interested not just in information, but something like action-readiness. I am referring to that which is both informative and action-ready as “knowledge”, although this may be stretching the standard use of this term.
Now you say that we might measure more abstract kinds of knowledge by looking at what an AI is willing to bet on. I agree that this is a good way to measure knowledge if it is available. However, if we are worried that an AI is deceiving us, then we may not be willing to trust its reports of its own epistemic state, or even of the bets it makes, since it may be willing to lose money now in order to convince us that it is not particularly intelligent, in order to make a treacherous turn later. Therefore I would very much like to find a definition that does not require me to interact with the AI through its input/output channels in order to find out what it knows, but rather allows me to look directly at its internals. I realize this may be impossible, but this is my goal.
So as you can see, my attempt at a definition of knowledge is very much wrapped up with the specific problem I’m trying to solve, and so any answers I arrive at may not be useful beyond this specific AI-related question. Nevertheless, I see this as an important question and so am content to be a little myopic in my investigation.
Thank you for this thoughtful comment itaibn0.
Matter and energy and also approximately homogeneously distributed in our own physical universe, yet building a small device that expands its influence over time and eventually rearranges the cosmos into a non-trivial pattern would seem to require something like an AI.
It might be that the same feat can be accomplished in Life using a pattern that is quite unintelligent. In that case I am very interested in what it is about our own physical universe that makes it different in this respect from Life.
Now it could actually be that in our own physical universe it is also possible to build not-very-intelligent machines that begin small but eventually rearrange the cosmos. In this case I am personally more interested in the nature of these machines than in “intelligent machines”, because the reason I am interested in intelligence in the first place is due to its capacity to influence the future in a directed way, and if there are simpler avenues to influence in the future in a directed way then I’d rather spend my energy investigating those avenues than investigating AI. But I don’t think it’s possible to influence the future in a directed way in our own physical universe without being intelligent.
to solve the control problem in an environment full of intelligence only requires marginally more intelligence at best
What do you mean by this?
the solution to the control problem may even be less intelligent than the structures it competes against, and make up for that with hard-coded solutions to NP-hard problems in military strategy.
But if one entity reliably outcompetes another entity, then on what basis do you say that this other entity is the more intelligent one?
Thank you for the kind words Jemist.
Yeah I’m open to improvements upon the use of the word “knowledge” because you’re right that what I’m describing here isn’t quite what either philosophers or cognitive scientists refer to as knowledge.
Yes knowledge-accumulating systems do seem to be a special case of optimizing systems. It may be that among all optimizing systems, it is precisely the ones that accumulate knowledge in the process of optimization that are of most interest to us from an alignment perspective, because knowledge-accumulating optimizing systems are (perhaps) the most powerful of all optimizing systems.
Dang, the images in this post are totally off. I have a script that converts a google doc to markdown, then I proofread the markdown, but the images don’t show up in the editor, and it looks like my script is off. Will fix tomorrow.
Update: fixed
Romeo I listened to your dialog. I hear that you are working on the is/ought gap from the ought side. If I may ask, what is the purpose of this work?
Knowledge is not just map/territory resemblance
Yes it does seem challenging to compute realistic complexity measures for such small functions. Perhaps we could just look at the mappings ordered by their volume in parameter space, and check whether the mappings at the top of that ordering “seem” less complex than the mappings at the bottom.
Yeah I had the sense that the project could have been intended as a compression mechanism since compressing in terms of CA rules kind of captures the spatial nature of image information quite well.
Well said, friend, but there is a difference between understanding love as a concept and listening directly to love. Where is it within ourselves that we can listen to love?
Yes—I found that work totally wild. Yes they are setting up a cellular automata in such a way that it evolves towards and then fixates at a target state, but iirc what they are optimizing over is the rules of the automata itself, rather than over a construction within the automata.
Jalen I would love to support you to to turn this into a standalone post, if that is of interest to you. Perhaps we could jump on a call and discuss the ideas you are pointing at here.
Problems facing a correspondence theory of knowledge
Yeah this seems right to me.
Thank you for all the summarization work you do, Rohin.
Somewhat ironically, some of these failures from thinking of oneself or others as agents causes a lack of agency! Maybe this is just a trick of language, but here’s what I have in mind from thinking about some of the pitfalls:
Yeah right, I agree with those three bullet points very much. Could also say “thinking of oneself or others as Cartesian agents causes a lack of power”. Does agency=power? I’m not sure what the appropriate words are but I agree with your point.
I see a certain amount of connection here to, for example, developmental psychology, where greater levels of development come from less thinking of oneself as a subject (an agent) and more thinking of the things one does as object and so thinking of oneself less as subject/agent results in greater agency because the range of things one allows oneself to consider mutable is greater.
Yeah, that seems well said to me. This gradual process of taking more things as object seems to lead towards very good things. Circling, I think, has a lot to do with taking the emotions we are used to treating as subject and getting a bit more of an object lens on them just by talking about them. Gendlin’s focussing seems to have a lot to do with this, too.
This also suggests an end state where there’s no agent model at all, or at least not one that is held to tightly
Yeah right, it’s a great lens to pick up and use when its helpful. But nice to know that it’s there and also to be able to put it down by choice.
Thank you for this comment Chantiel.
Consider inanimate biological substances, like a leaf. From a wide range of initial configurations of a leaf, effectively all make the leaf evolve towards being dirt, because leafs decompose eventually. Are leaves optimizers?
People tend to get older and wrinklier when aging. From a wide range of states, people would tend to “evolve” towards being aged. Are people optimizers for aging?
It’s a good question. However, the decomposition of a leaf and of the body are both examples of increases in entropy over time, but actually if you look at the size of the “target configuration set” you find that it’s almost as big as the whole configuration space, because most of the configurations of a system are high entropy configurations. So I don’t think a leaf or an aging body qualify as optimizing systems according to the definition in this post. See also this section.
If the rock is hotter than the surrounding aid, virtually any initial configuration of the rock would tend towards the rock being somewhere around the temperature of the surrounding air. Are rocks optimizers?
Well you really have to look at the whole system. It’s true that if you have a system that consists of a hot part and cold part, the system overall will evolve towards configurations in which the parts are the same temperature. But this is again an example of entropy increasing. Most of the configurations of the joint rock+environment system have the rock and the environment at approximately the same temperature, since if you randomly sample a temperature for each particle, the large number of particles in the rock and the environment mean that the average temperature of all the particles in the rock will be very similar to the average temperature of all the particles in the environment, with high probability.
What about taking the target configuration to be any state in which all the billiard balls are stationary? A wide range of states of billiards bouncing around on a table would result in all the balls ending up stationary, so I don’t see how it wouldn’t be classified as an optimization process.
Yes, just like a ball rolling down a hill qualifies as an optimizing system, a table with with billiard balls qualifies as an optimizing system in the sense that you point out.
But the whole point of this post is to get past the notion of “optimizer” and “optimization” to the extent that these concepts imply that there is some “agent” performing optimization, and some thing “being optimized”, which sneaks the agent model into all our thinking and leads to a very confused picture of things.
Also, I’ve made my own attempt at defining “optimizer” here, in case you’re interested.
Thank you for the pointer!
Thank you friend
What fear is that, friend?
I very much agree with the emphasis on actionability. But what is it about a physical artifact that makes the knowledge it contains actionable? I don’t think it can be simplicity alone. Suppose I record the trajectory of the moon over many nights by carving markings into a piece of wood. This is a very simple representation, but it does not contain actionable knowledge in the same way that a textbook on Newtonian mechanics does, even if the textbook were represented in a less simple way (say, as a PDF on a computer).