Alex_Altair

• Alex_Altair 6 Jul 2023 20:07 UTC

  4 points

  2

  in reply to: Alex_Altair’s comment on: A Defense of Work on Math­e­mat­i­cal AI Safety

  Unrelatedly, why not make this a cross-post rather than a link-post?

• Alex_Altair 6 Jul 2023 20:06 UTC

  18 points

  8

  on: A Defense of Work on Math­e­mat­i­cal AI Safety

  I think it would help a lot to provide people with examples. For example, here

  Many machine learning research agendas for safety are investigating issues identified years earlier by foundational research, and are at least partly informed by that research.

  You say that, but then don’t provide any examples. I imagine readers just not thinking of any, and then moving on without feeling any more convince.

  Overall, I think that it’s hard for people to believe agent foundations will be useful because they’re not visualizing any compelling concrete path where it makes a big difference.

• Alex_Altair 29 Jun 2023 17:17 UTC

  2 points

  0

  in reply to: aysja’s comment on: Towards Mea­sures of Optimisation

  The first is what Garrett points out, that probabilities are map things, and it’s a bit… weird for our measure of a (presumably) territory thing to be dependent on them. It’s the same sort of trickiness that I don’t feel we’ve properly sorted out in thermodynamics—namely, that if we take the existence of macrostates to be reflections of our uncertainty (as Jaynes does), then it seems we are stuck saying something to the effect of “ice cubes melt because we become more uncertain of their state,” which seems… wrong.

  For this part, my answer is Kolmogorov complexity. An ice cube has lower K-complexity than the same amount of liquid water, which is a fact about the territory and not our maps. (And if a state has lower K-complexity, it’s more knowable; you can observe fewer bits, and predict more of the state.)

  One of my ongoing threads is trying to extend this to optimization. I think a system is being objectively optimized if the state’s K-complexity is being reduced. But I’m still working through the math.

• Alex_Altair 29 Jun 2023 17:09 UTC

  2 points

  0

  in reply to: aysja’s comment on: Towards Mea­sures of Optimisation

  Yeah… so these are reasonable thoughts of the kind that I thought through a bunch when working on this project, and I do think they’re resolvable, but to do so I’d basically be writing out my optimization sequence.

  I agree with Alexander below though, a key part of optimization is that it is not about utility functions, it is only about a preference ordering. Utility functions are about choosing between lotteries, which is a thing that agents do, whereas optimization is just about going up an ordering. Optimization is a thing that a whole system does, which is why there’s no agent/​environment distinction. Sometimes, only a part of the system is responsible for the optimization, and in that case you can start to talk about separating them, and then you can ask questions about what that part would do if it were placed in other environments.

• Alex_Altair 29 Jun 2023 16:08 UTC

  4 points

  1

  in reply to: Matthew Barnett’s comment on: My re­search agenda in agent foundations

  Yeah, this is why we need a better explainer for agent foundations. I won’t do it justice in this comment but I’ll try to say some helpful words. (Have you read the Rocket Alignment Problem?)

  Do you expect there will be a whole new paradigm, and that current neural networks will be nothing like future AIs?

  I can give an easy “no” to this question. I do not necessarily expect future AIs to work in a whole new paradigm.

  My understanding is that you’re trying to build a model of what actual AI agents will be like.

  This doesn’t really describe what I’m doing. I’m trying to help figure out what AIs we should build, so I’m hoping to affect what actual AI agents will be like.

  But more of what I’m doing is trying to understand what the space of possible agents looks like at all. I can see how that could sound like someone saying, “it seems like we don’t know how to build a safe bridge, so I’m going to start by trying to understand what the space of possible configurations of matter looks like at all” but I do think it’s different than that.

  Let me try putting it this way. The arguments that AI could be an existential risk were formed before neural networks were obviously useful for anything. So the inherent danger of AIs does not come from anything particular to current systems. These arguments use specific properties about the general nature of intelligence and agency. But they are ultimately intuitive arguments. The intuition is good enough for us to know that the arguments are correct, but not good enough to help us understand how to build safe AIs. I’m trying to find the formalization behind those intuitions, so that we can have any chance at building a safe thing. Once we get some formal results about how powerful AIs could be safe even in principle, then we can start thinking about how to build versions of existing systems that have those properties. (And yes, that’s a really long feedback loop, so I try to recurringly check that my trains of ideas could still in principle apply to ML systems.)

My re­search agenda in agent foundations

• Alex_Altair 23 Jun 2023 23:45 UTC

  2 points

  0

  in reply to: Jonathan Richens’s comment on: Causal­ity: A Brief Introduction

  I’d agree that the bits of output are not independent in some physical sense. But they’re definitely independent in my mind! If I hear that the 100th binary digit of pi is 1, then my subjective probability over the 101st digit does not update at all, and remains at 0.5/​0.5. So this still feels like a frequentism/​Bayesianism thing to me.

  Re: the modified experiment about random strings, you say that “To get the string of random bits we have to sample a coin flip, and then make two copies of the outcome”. But there’s nothing preventing the universe from simply containing to copies of the same random string, created causally independently. But that’s also vanishingly unlikely as the string gets longer.

• Alex_Altair 23 Jun 2023 23:23 UTC

  3 points

  0

  in reply to: tom4everitt’s comment on: Causal­ity: A Brief Introduction

  Yeah, I think I agree that the resolution here is something about how we should use these words. In practice I don’t find myself having to distinguish between “statistics” and “probability” and “uncertainty” all that often. But in this case I’d be happy to agree that “all statistical correlations are due to casual influences” given that we mean “statistical” in a more limited way than I usually think of it.

  But I don’t think we know how to properly formalise or talk about that yet.

  A group of LessWrong contributors has made a lot of progress on these ideas of logical uncertainty and (what I think they’re now calling) functional decision theory over the last 15ish years, although I don’t really follow it myself, so I’m not sure how close they’d say we are to having it properly formalized.

• Alex_Altair 21 Jun 2023 20:16 UTC

  3 points

  0

  in reply to: Jonathan Richens’s comment on: Causal­ity: A Brief Introduction

  Thanks for writing that out! I’ve enjoyed thinking this through some more.

  I agree that, if you instantiated many copies of the program across the universe as your sampling method, or somehow otherwise “ran them many times”, then their outputs would be independent in the sense that P(A, B) = P(A, B). This also holds true if, on each run, there was some “local” error to the program’s otherwise deterministic output.

  I had intended to be using the program’s output as a time series of bits, where we are considering the bits to be “sampling” from A and B. Let’s say it’s a program that outputs the binary digits of pi. I have no idea what the bits are (after the first few) but there is a sense in which P(A) = 0.5 for either A = 0 or A = 1, and at any timestep. The same is true for P(B). So P(A)P(B) = 0.25. But clearly P(A = 0, B = 0) = 0.5, and P(A = 0, B = 1) = 0, et cetera. So in that case, they’re not probabilistically independent, and therefore there is a correlation not due to a causal influence.

  But this is in a Bayesian framing, where the probability isn’t a physical thing about the programs, it’s a thing inside my mind. So, while there is a common source of the correlation (my uncertainty over what the digits of pi are) it’s certainly not a “causal influence” on A and B.

  This matters to me because, in the context of agent foundations and AI alignment, I want my probabilities to be representing my state of belief (or the agent’s state of belief).

• Alex_Altair 20 Jun 2023 17:33 UTC

  24 points

  14

  on: Causal­ity: A Brief Introduction

  Ultimately, all statistical correlations are due to casual influences.

  As a regular LW reader who has never been that into causality, this reads as a blisteringly hot take to me. My first thought is, what about acausal correlations? You could have two instances of the same program running on opposite sides of the universe, and their outputs would be the same, but there is clearly no causal influence there. The next example that comes to mind is two planets orbiting their respective stars which just so happen to have the same orbital period; their angular offset over time will correlate, and again their is no common cause.

  (In both cases you could say that the common cause is something like the laws of physics allowing two copies of similar systems to come into existence, but I would say that stretches the concept of causality beyond usefulness.)

  I also notice that there’s no wikipedia page for “Reichenbach’s Common Cause Principle”, which makes me think it’s not a particularly widely accepted idea. (In any case I don’t think this has an effect on the value of the rest of this sequence.)

• Alex_Altair 26 May 2023 16:43 UTC

  2 points

  0

  in reply to: Viliam’s comment on: Sup­port Struc­tures for Nat­u­ral­ist Study

  Oh, this thing I identify more with doing (and getting results from).

• Alex_Altair 24 May 2023 22:07 UTC

  2 points

  0

  on: Boundaries vs Frames

  I only vaguely understand both Cartesian frames and «boundaries», but I want to take a shot at explaining what feels to me like some confusions in this post. Also a bunch of this might be invalidated by your disclaimer that this post was published before the final version of Crtich’s boundaries sequence.

  It seems to me that your work in Cartesian frames is about finding an ontology in which we can make progress on agent foundations in general, whereas Critch’s concept of boundaries is about formalizing a necessary condition for identifying when an agent exists in a particular system.

  While every boundary can be recast as a frame, the converse is not also true.

  Indeed! We wouldn’t want a theory of agents (or boundaries) to be applicable to a state in thermal equilibrium, for example, whereas we definitely want our ontology to be able to handle it.

  Getting Past the Physical Frame

  This whole section is actually a little confusing. It sounds like you’re interpreting Critch’s sequence to be saying that boundaries are inherently physical. But aren’t all the definitions in terms of information theory? The nodes in the network are course-grained, but aren’t we free to decide what about the world the are abstracted from?

  This also further motivates frames over boundaries, as we might want to reason about multiple different frames simultaneously with no underlying factorization that multiplicatively-refines both of them.

  It seems entirely possible to me for a formalization of boundaries to allow for two different but equally-valid boundaries to be drawn within the state.

  Actually, now that I think about it, is there any reason that Critch’s definition of boundaries isn’t fully compatible with the ontology of factored sets?

• Alex_Altair 24 May 2023 20:56 UTC

  3 points

  1

  on: «Boundaries», Part 3b: Align­ment prob­lems in terms of bound­aries

  Boundaries are information-theoretic and more objective in that they are (often) inter-subjectively observable just by counting bits of mutual information between variables, whereas preferences a subjective and observable only indirectly through behavior.

  This feels wrong to me, but the feeling is a little fuzzy. Maybe I disagree with the emphasis or the framing, or something.

  Preferences are instantiated in the world just like anything else. And they might be entirely visible; there’s no reason you couldn’t have a system whose preferences are on display (like reading the source code of a program). But I would grant that they’re not usually on display, especially for humans, and so you do have to infer them through behavior (although that can include the person telling you what they are, or otherwise agreeably taking actions that are strong evidence of their preferences).

  In contrast, boundaries are usually evident from the outside, even if they obscure what’s happening on the inside. And if preferences are not directly visible, then they’re probably inside a boundary, in which case the boundary is easier to detect and verify than the preferences.

• Alex_Altair 24 May 2023 19:32 UTC

  3 points

  0

  on: «Boundaries», Part 3a: Definitions

  Here I’m going to log some things I notice while reading, mostly as a way to help check my understanding, but also to help find possible errata.

  In Definition part (a), you’ve got a whole lot of W-type symbols, and I’m not 100% sure I follow each of their uses. You use $\omega$ a couple times which is legit, but it looks a lot like $w$, so maybe it could be replaced with $N$?

  See this comment for two errata with the different w’s.

  $Fut\left(w\right)$ denotes, for any world state $w\in W$ , the future of the Dirac (100% concentrated) distribution on the world state $w\in W$.

  Maybe you could just say, $Fut\left(w\right)$ is shorthand for $Fut\left(T_W\right(w\left)\right)$, since $T_W$ will map $w$ to the right thing of type $\Delta W$. Then you can avoid bringing in the somewhat exotic Dirac delta function. Of course, that now means that $w$ itself is not the first item in the resulting sequence. I’m not sure if you need that to be the case for later. But also, everything above is ambiguous about whether the argument to $Fut$ was in the sequence anyway.

  The character ⫫ doesn’t render for me. (I could figure out what it was by pasting the unicode into google, but maybe it could be done with LaTeX instead?)

  To formalize this, I want a collection of state spaces and maps, like so:

  Is the following bulleted list missing an entry for $E$?

  Each of these factorizations are assumed to be bijective, in the sense of accounting for everything that matters and not double-counting anything

  I was wondering if you were going to say something like $W=V\times B\times E$ and $B=A\times P$. It sounds like that’s almost right, except that you allow the factors to pass through arbitrary functions first, as long as they’re bijective. Is that right?

  We say $r_{\theta }$ is a good fit

  You bring back $r_{\theta }$ here, but I don’t see the $\theta$ doing anything yet. Might be better not to introduce it until later, to free up a bit of the reader’s working memory.

  See this comment for a broken link.

• Alex_Altair 16 May 2023 16:22 UTC

  9 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Towards Mea­sures of Optimisation

  My best guess about the core difference between optimization and agency is the thing I said above about, “a utility function, which they need for picking actions with probabilistic outcomes”.

  An agent wants to move the state up an ordering (its optimization criterion). But an agent also has enough modelling ability to know that any given action has (some approximation of) a probability distribution over outcomes. (Maybe this is what you mean by “counterfactuality”.) Let’s say you’ve got a toy model where your ordering over states is A < B < C < D < E and you’re starting out in state C. The only way to decide between [a 30% chance of B + a 70% chance of D] and [a 40% chance of A + a 60% change of E] is to decide on some numerical measure for how much better E is than D, et cetera.

  Gradient descent doesn’t have to do this at all. It just looks at the gradient and is like, number go down? Great, we go in the down direction. Similarly, natural selection isn’t doing this either. It’s just generating a bunch of random mutations and then some of them die.

  (I’m not totally confident that one couldn’t somehow show some way in which these scenarios can be mathematically described as calculating an expected utility. But I haven’t needed to pull in these ideas for deconfusing myself about optimization.)

• Alex_Altair 15 May 2023 2:56 UTC

  2 points

  0

  in reply to: mattmacdermott’s comment on: Towards Mea­sures of Optimisation

  Mostly it’s that I’ve found that, while trying to understand optimization, I’ve never needed to put “weights” on the ordering. (Of course, you always could map your ordering onto a monotonically increasing function.)

  I think the concept of “trying” mostly dissolves under the kind of scrutiny I’m trying to apply. Or rather, to well-define “trying”, you need a whole bunch of additional machinery that just makes it a different thing than (my concept of) optimization, and that’s not what I’m studying yet.

  I’ve also been working entirely in deterministic settings, so there’s no sense of “how often” a thing happens, just a single trajectory. (This also differentiates my thing from Flint’s.)

  I haven’t stopped working on the overall project. I do seem to have stopped writing and editing that particular sequence, though. I’m considering totally changing the way I present the concept (such that the current Intro post would be more like a middle-post) so I decided to just pull the trigger on publishing the current state of it. I’m also trying to get more actual formal results, which is more about stuff from the end of that sequence. But I’m pretty behind on formal training, so I’m also trying to generally catch up on math.

• Alex_Altair 15 May 2023 2:38 UTC

  2 points

  0

  in reply to: the gears to ascension’s comment on: Towards Mea­sures of Optimisation

  I mean, that makes sense according to their definition, I think I’m just defining the word differently. Personally I think defining “agent” such that gradient descent is an agent seems pretty off from the colloquial use of the word agent.

• Alex_Altair 15 May 2023 2:02 UTC

  4 points

  0

  on: Sup­port Struc­tures for Nat­u­ral­ist Study

  I’m curious how universal the value of “Undirected Time” is. I’ve found that I pretty much just… don’t have “shower thoughts”, at least not ones that are more frequently about what I’ve been working on. Similarly, I don’t relate to the phenomenon of taking a break from a hard problem, coming back, and then suddenly making faster progress on it. And I especially don’t relate to the phenomenon of having ideas come to me in dreams.

  It’s not obvious to me whether this is better or worse. On one hand I could be saving time by not having to wait for background processes (I don’t feel like I get particularly stuck on problems anyway), but on the other hand maybe I’m just failing to use the background processes to do the cognition for me?

  On the other other hand, I have to wonder if maybe I just do the equivalent of “undirected time”, like, all the time? I feel like I pretty often switch what I’m thinking about between unrelated topics, or change scope, or wonder about whether things are connected in a certain way, etc. It feels like this might be satisfying the intended meaning of “undirected time”, even if it doesn’t feel undirected to me.

[Question] Why don’t quan­tiliz­ers also cut off the up­per end of the dis­tri­bu­tion?

• Alex_Altair 12 May 2023 19:52 UTC

  2 points

  0

  in reply to: Garrett Baker’s comment on: Towards Mea­sures of Optimisation

  Okay, that’s an interesting comparison. Maybe this will help; Yudkowsky’s measure of optimization is a measure, like of how much it’s happening, rather than the definition. Then the definition is “when a system’s state moves up an ordering”. Analogously, objects have length, and you can tell “how much of an object” there is by how long it is. And if there’s no object, then it will have zero length. But that doesn’t make the definition of “object” be “a thing that has length”. Does that make sense?