Master’s student in applied mathematics, funded by Center on Long-Term Risk to investigate the cheating problem in safe pareto-improvements. Former dovetail fellow with @Alex_Altair.
Daniel C
Karma: 168
I also talked to Aram recently & he’s optimistic that there’s an algorithmic version of the generalized heat engine where the hot vs cold pool correspond to high vs low k-complexity strings. I’m quite interested in doing follow-up work on that
The continuous state-space is coarse-grained into discrete cells where the dynamics are approximately markovian (the theory is currently classical) & the “laws of physics” probably refers to the stochastic matrix that specifies the transition probabilities of the discrete cells (otherwise we could probably deal with infinite precision through limit computability)
As in, take a set of variables X, then search for some set of its (non-overlapping?) subsets such that there’s a nontrivial natural latent over it? Right, it’s what we’re doing here as well.
I think the subsets can actually be partially overlapping, for instance you may have a that’s approximately deterministic w.r.t and but not alone, weak redundancy (approximately deterministic w.r.t ) is also an example of redunds across overlapping subsets
Mm, this one’s shaky. Cross-hypothesis abstractions don’t seem to be a good idea, see here.
yea so I think the final theory of abstraction will have a weaker notion of equivalence espeically when we incorporate ontology shifts. E.g. we want to say that water is the same concept before and after we discover water is H2O, but the discovery obviously breaks predictive agreement (Indeed, the solomonoff version of natural latent is more robust to the agreement condition)Also, you can totally add new information/abstraction that is not shared between your current and new hypothesis, & that seems consistent with the picture you described here (you can have separate ontologies but you try to capture the overlap as much as possible)
My guess is that there’s something like a hierarchy of hypotheses, with specific high-level hypotheses corresponding to several lower-level more-detailed hypotheses, and what you’re pointing at by “redundant information across a wide variety of hypotheses” is just an abstraction in a (single) high-level hypothesis which is then copied over into lower-level hypotheses. (E. g., the high-level hypothesis is the concept of a tree, the lower-level hypotheses are about how many trees are in this forest.)
yes I think that’s the right picture
But we don’t derive it by generating a bunch of low-level hypotheses and then abstracting over them, that’d lead to broken ontologies.
I agree that we don’t do that practically as it’d be slower (instead we simply generate an abstraction & use future feedback to determine whether it’s a robust one), but I think if you did generate a bunch of low-level hypotheses and look for redundant computation among them, then an adequate version of it would just recover the “high-level low-level hypotheses” picture you’ve described?
In particular, with cross-hypothesis abstraction we don’t have to separately define what the variables are, so we can sidestep dataset-assembly entirely & perhaps simplify the shifting structures problem
Nice, I’ve gestured at similar things in this comment, conceptually the main thing you want to model is variables that control the relationships between other variables, the upshot is you can continue the recursion indefinitely: Once you have second order variables that control the relationships between other variables, you can then have variables that control the relationship among second order variables and so on.
Using function calls as an analogy: When you’re executing a function that itself makes a lot of function calls, there are two main ways these function calls can be useful:
The results of these function calls might be used to compute the final output
The results of these function calls can tell you what other function calls would be useful to make (e.g. if you want to find the shape of a glider, the position tells you which cells to look at to determine that)
an adequate version of this should also be turing complete which means it can accomodate shifting structures, & function calls seem like a good way to represent hierarchies of abstractions
CSI in bayesian networks also deals with the idea that the causal structure between variables changes over time/depending on context (you’re probably more interested in how relationships between levels of abstraction changes with context, but the two directions seem linked). I plan to explore the following variant at some point(not sure if it’s already in the literature):
Suppose that there is a variable that “controls” the causal structure of , we use the good-old KL approximation to represent the error conditional on a particular value of under a particular diagram
You can imagine that the conditional distrbution initially approximately satisfies a diagram , but as you change the value of , the error for goes up while the error for some other diagram goes to 0
In particular, if is a continuous variable, and the conditional distribution changes continuously with , then changes continuously with which is quite nice
So this is a formalism that deals with “context-dependent structure” in a way that plays well with continuity, and if you have discrete variables controlling the causal structure, you can use it to accommodate uncertainty over the discrete outcomes (that determine causal structure).
But note that synergistic information can be defined by referring purely to the system we’re examining, with no “external” target variable. If we have a set of variables , we can define the variable s such that is maximized under the constraint of . (Where is the set of all subsets of except itself.)
That’s a nice formulation of synergistic information, it’s independent with redundant info via the data-processing inequality so somewhat promising that it can add up to total entropy.You might be interested in this comment if distinguishing betweeen synergistic and redundant information is not your main objective: You can simply define redunds over collections of subsets, such that e.g. “dogness” is a redund over every subset of atoms that allows you to conclude you’re looking at a dog. In particular, the redundancy lattice approach seems simpler when the latent depends on not just synergistic but also redundant and unique information
One issue with PID worth mentioning is that they haven’t figured out what measure to use for quantifying multivariate redundant information. It’s the same problem we seem to have. But it’s probably not a major issue in the setting we’re working in (the well-abstracting universes).
Recent impossibility result seems to rule out general multivariate PID that guarantees non-negativity of all components, though partial entropy decomposition may be more tractable
If there’s a pair of , such that , then necessarily contains all information in . Re-define , removing all information present in .
This seems similar to capturing unique information, where the constructive approach is probably harder in PID than PED. E.g. in BROJA it involves an optimization problem over distributions with some constraints on marginals, but it only estimates the magnitude of unique info, not an actual random variable that represents unique info
Nice post!
Some frames about abstractions & ontology shifts I had while thinking through similar problems (which you may have considered already):
The dual of “abstraction as redundant information across a wide variety of agents in the same environment” is “abstraction as redundant information/computation across a wide variety of hypotheses about the environment in an agent’s world model” (E.g. a strawberry is a useful concept to model for many worlds that I might be in). I think this is a useful frame when thinking about “carving up” the world model into concepts, since a concept needs to remain invariant while the hypothesis keeps being updated
The semantics of a component in a world model is partly defined by its relationship with the rest of the components (e.g. move a neuron to a different location and its activation will have a different meaning), so if you want a component to have stable semantics over time, you want to put the “relational/indexical information” inside the component itself
In particular, this means that when an agent acquires new concepts, the existing concepts should be able to “specify” how it should relate to that new concept (e.g. learning about chemistry then using it to deduce macro-properties of strawberries from molecular composition)
happy to discuss more via PM as some of my ideas seem exfohazardous
Neat idea, I’ve thought about similar directions in the context of traders betting on traders in decision markets
A complication might be that a regular deductive process doesn’t discount the “reward” of a proposition based on its complexity whereas your model does, so it might have a different notion of logical induction criterion. For instance, you could have an inductor that’s exploitable but only for propositions with larger and larger complexities over time, such that with the complexity discounting the cash loss is still finite (but the regular LI loss would be infinite so it wouldn’t satisfy regular LI criterion)
(Note that betting on “earlier propositions” already seems beneficial in regular LI since if you can receive payouts earlier you can use it to place larger bets earlier)
There’s also some redundancy where each proposition can be encoded by many different turing machines, whereas a deductive process can guarantee uniqueness in its ordering & be more efficient that way
Are prices still determined using Brouwer’s fixed point theorem? Or do you have a more auction-based mechanism in mind?
Yes I agree
I think it’s similar to CIRL except less reliant on the reward function & more reliant on the things we get to do once we solve ontology identification
An alternative to pure imitation learning is to let the AI predict observations and build its world model as usual (in an environment containing humans), then develop a procedure to extract the model of a human from that world model.
This is definitely harder than imitation learning (probably requires solving ontology identification+ inventing new continual learning algorithms) but should yield stronger guaranteees & be useful in many ways:
It’s basically “biometric feature conditioning” on steroids, (with the right algorithms) the AI will leverage whatever it knows about physics, psychology, neuroscience to form its model of the human, and continue to improve its human model as it learns more about the world (this will require ontology identification)
We can continue to extract the model of the current human from the current world model & therefore keep track of current preferences. With pure imitation learning it’s hard to reliably sync up the human model with the actual human’s current mental state (e.g. the actual human is entangled with the environment in a way that the human model isn’t unless the human wears sensors at all times). If we had perfect upload tech this wouldn’t be much of an issue, but seems significant especially at early stages of pure imitation learning
In particular, if we’re collecting data of human actions under different circumstances, then both the circumstance and the human’s brain state will be changing, & the latter is presumably not observable. It’s unclear how much more data is needed to compensate for that
We often want to run the upload/human model on counterfactual scenarios: Suppose that there is a part of the world that the AI infers but doesn’t directly observe, if we want to use the upload/human model to optimize/evaluate that part of the world, we’d need to answer questions like “How would the upload influence or evaluate that part of the world if she had accurate beliefs about it?”. It seems more natural to achieve that when the human model was originally already entangled with the rest of the world model than if it resulted from imitation learning
(Was in the middle of writing a proof before noticing you did it already)
I believe the end result is that if we have , with ( upstream of , upstream of , upstream of both),
then maximizing is equivalent to maximizing .
& for the proof we can basically replicate the proof for additivity except substituting the factorization as assumption in place of independence, then both directions of inequality will result in .
[EDIT: Forgot term due to marginal dependence ]
I think a subtle point is that this is saying we merely have to assume predictive agreement of distributions marginalized over the latent variables , but once we assume that & the naturality conditions, then even as each agent receive more information about to update their distributions & latent variables , the deterministic constraints between the latents will continue to hold.
Or if a human and AI start out with predictive agreement over some future observables, & the AI’s latent satisfy mediation while human’s latent satisfy redundancy, then we could send the AI out to update on information about those future observables, and humans can (in principle) estimate the redundant latent variable they care about from the AI’s latent without observing the observables themselves. The remaining challenge is that humans often care about things that are not approximately deterministic w.r.t observables from typical sensors.
With normal science, there’s a phenomenon that we observe, and what we want is to figure out the underlying laws. With AI systems, it’s more accurate to say that we know the underlying laws (such as the mathematics of computation, and the “initial conditions” of learning algorithms) and we’re trying to figure out what phenomena will occur (e.g. what fraction of them will undergo instrumental convergence).
I’d say part of agent foundations is the reverse: We know what phenomena will probably occur (extreme optimization by powerful agent) and what phenomena we want to cause (alignment). And we’re trying to understand the underlying laws that could cause those phenomena (algorithms behind general intelligence that have not been invented yet) so that we can steer them towards the outcomes we want.
Congrats!
Some interesting directions I think this opens up: Intuitively, given a set of variables , we want natural latents to be approximately deterministic across a wide variety of (collections of) variables, and if a natural latent is approximately deterministic w.r.t a subset of variables , then we want to be as small as possible (e.g. strong redundancy is better than weak redundancy when the former is attainable)
The redundancy lattice seems natural for representing this: Given an element of the redundancy lattice , we say is a redund over if it’s approximately deterministic w.r.t each subset in . E.g. is weakly redundant over if it’s a redund over (approximately deterministic function of each ), and strongly redundant if it’s a redund over . If is a redund over , our intuitive desiderata for natural latents correspond to containing more subsets (more redundancy), and each subset being small (less “synergy”). Combine this with the mediation condition can probably give us a notion of pareto-optimality for natural latents.
Another thing we could do is when we construct pareto-optimal natural latents over , we add them to the original set of variables to augment the redundancy lattice, so that new natural latents can be approximately deterministic functions over (collections of) existing natural latents, and this naturally allows us to represent the “hierarchical nature of abstractions” where lower-level abstractions makes it easier to compute higher-level ones.
A concrete setting where this can be useful is where a bunch of agents receive different but partially overlapping sets of observations and aims to predict partially overlapping domains. Having a fine grained collection of natural latents redundant across different elements of the redundancy lattice means we get to easily zoom in on the smaller subset of latent variables that’s (maximally) redundantly represented by all of the agents (& be able to tell which domains of predictions these latents actually mediate).
There’s a chicken-and-egg problem here[...] and then using that assumption to prove that markets are causal.
That argument was more about accomodating “different traders with different beliefs”, but here’s an independent argument for market being causal:
When I cause a particular effect/outcome, that means I mediate the influence between the cause of my action and the effect/outcome of my action, the cause of my action is conditionally independent of the effect of my action given me
Futarchy is a similar case: There may be many causes that influence market prices, which in turn determines the decision chosen, & market prices mediate the influence between the cause of market prices (e.g. different traders’ beliefs) and the decision chosen. Any information can only influence what decision will be chosen through influencing the market prices. This seems like what it means for market to be causal (In a bayesnet, the decision chosen will literally only have market prices as the parent, assuming we commit to using futarchy to choose decisions).
The first expectation needs to be conditioned on the market activating. (That is not conditionally independent of u given d1 in general.)
If we commit to using futarchy to choose decision, then market 1 activating will have exactly the same truth conditions as executing d1, so “market activating and d1” would be the exact same thing as “d1“ itself (commiting to use futarchy to choose decision means we assign 0 probability to “first market activating & execute d2” or “Second market activating & execute d1”)
Different people have different beliefs, so the expectations are different for different traders. You can’t write “E” without specifying for which trader.
Yes, we can replace with E_i, and then argue that traders with accurate beliefs will accumulate more money over time, making market estimates more accurate in the limit
My main objection to this logic is that there doesn’t seem to be any reflection of the idea that different traders will have different beliefs.[...] All my logic is based on a setup where different traders have different beliefs.
Over time, traders who have more accurate beliefs (& act rationally according to those beliefs) will accumulate more money in expectation (& vice versa), so in the limit we can think of futarchy as aggregating the beliefs of different traders weighted by how accurate their beliefs were in the pastSo I don’t think the condition “p1>E[u|d1]” really makes sense? [...]and this makes it unlikely that the market will converge to E[u|d1].
If I pay p1 for a contract in market 1, my expected payoff is:
(since I get my money back if d2/market 2 is activated)
this is negative iff and positive iff
and if we commit to using futarchy to choose the decision, then is chosen iff market 1 activates, so E_i[u|d1, market 1 activates] should equal E_i[u|d1]
We want you to pay more for a contract for coin A, since that’s the coin you think is more likely to be heads (60% vs 59%). But if you like money, you’ll pay more for a contract on coin B. You’ll do that because other people might figure out if it’s an always-heads coin or an always-tails coin. If it’s always heads, great, they’ll bid up the market, it will activate, and you’ll make money. If it’s always tails, they’ll bid down the market, and you’ll get your money back.
Let’s call “Bidding on B, hoping that other people will figure out if B is an always-head or always-tails coin” strategy X, and call “Figure out if B is an always-head or always-tail myself & bid accordingly, or if I can’t, bid on A because it’s better in expectation” strategy Y.
If I believe that sufficient number of people in the market are using strategy Y, then it’s beneficial for me to use strategy X, and insofar as my beliefs about the market are accurate, this is okay, because sufficient number of people using strategy Y means the market will actually figure out if B is always-head or always-tail, then bid accordingly. So the market selects the right decision, insofar as my beliefs about the market is correct (Note that I’m never incentivized to place a bid on B so large that it causes B to activate, since I don’t actually know if B is always-head).
On the other hand, if I believe that the vast majority of people in the market are using strategy X instead of strategy Y, then it’s no longer beneficial for me to use strategy X myself, I should instead use strategy Y because the market doesn’t actually do the work of finding out if coin B is always-head for me. Other traders who have accurate beliefs about the market will switch to strategy Y as well, until there is a sufficient number of trader to push the market towards the right decision.
So insofar as people have accurate beliefs about the market, the market will end up selecting the right decision (either sufficient number of people use strategy Y, in which case it’s robust for me to use strategy X, or not enough people are using strategy Y, in which case people are incentivized to switch to Y)
More generally, what’s the argument that the market will always select the decision that leads to he higher expected payout?
“Always” might be too strong, but very informally:
Suppose that we have we have decision d1 d2, with outcome/payoff u & conditional market prices p1 (corresponds to d1) p2 (corresponds to d2)
if p1>E[u|d1], then traders are incentivized to sell & drive down p1. Similarly they will be incentivized to bid up p1 if p1<E[u|d1]. So p1 will tend toward E[u|d1]. We can argue similar for p2 tending towards E[u|d2]
Since we choose the decision with the higher price, and prices tend towards the expected payoff given that decision, the market end up choosing the decision that leads to the higher expected payoff.
I think I’m claiming 3, namely all we want from futarchy is for it to select the decision with the highest expected payout, and for that the property isn’t necessary.
Ex: For the two coin two market case, the first market’s price estimates the expected payout if we flip coin one (& similarly for the second market), & while neither market satisfies the property (E[f]=E[z] always), we would still select the decision that leads to the higher expected payout (as we select the higher price), and that’s all that’s needed
The current theory is based on classical hamiltonian mechanics, but I think the theorems apply whenever you have a markovian coarse-graining. Fermion doubling is a problem for spacetime discretization in the quantum case, so the coarse-graining might need to be different. (E.g. coarse-grain the entire hilbert space, which might have locality issues but probably not load-bearing for algorithmic thermodynamics)
On outside view, quantum reduces to classical (which admits markovian coarse-graining) in the correspondence limit, so there must be some coarse-graining that works