I guess we can try studying Troll Bridge using infra-Bayesian modal logic, but atm I don’t know what would result.
From a radical-probabilist perspective, the complaint would be that Turing RL still uses the InfraBayesian update rule, which might not always be necessary to be rational (the same way Bayesian updates aren’t always necessary).
Ah, but there is a sense in which it doesn’t. The radical update rule is equivalent to updating on “secret evidence”. And in TRL we have such secret evidence. Namely, if we only look at the agent’s beliefs about “physics” (the environment), then they would be updated radically, because of secret evidence from “mathematics” (computations).
Ah, but there is a sense in which it doesn’t. The radical update rule is equivalent to updating on “secret evidence”. And in TRL we have such secret evidence. Namely, if we only look at the agent’s beliefs about “physics” (the environment), then they would be updated radically, because of secret evidence from “mathematics” (computations).
I agree that radical probabilism can be thought of as bayesian-with-a-side-channel, but it’s nice to have a more general characterization where the side channel is black-box, rather than an explicit side-channel which we explicitly update on. This gives us a picture of the space of rational updates. EG, the logical induction criterion allows for a large space of things to count as rational. We get to argue for constraints on rational behavior by pointing to the existence of traders which enforce those constraints, while being agnostic about what’s going on inside a logical inductor. So we have this nice picture, where rationality is characterized by non-exploitability wrt a specific class of potential exploiters.
Here’s an argument for why this is an important dimension to consider:
Human value-uncertainty is not particularly well-captured by Bayesian uncertainty, as I imagine you’ll agree. One particular complaint is realizability: we have no particular reason to assume that human preferences are within any particular space of hypotheses we can write down.
One aspect of this can be captured by InfraBayes: it allows us to eliminate the realizability assumption, instead only assuming that human preferences fall within some set of constraints which we can describe.
However, there is another aspect to human preference-uncertainty: human preferences change over time. Some of this is irrational, but some of it is legitimate philosophical deliberation.
And, somewhat in the spirit of logical induction, humans do tend to eventually address the most egregious irrationalities.
Therefore, I tend to think that toy models of alignment (such as CIRL, DRL, DIRL) should model the human as a radical probabilist; not because it’s a perfect model, but because it constitutes a major incremental improvement wrt modeling what kind of uncertainty humans have over our own preferences.
Recognizing preferences as a thing which naturally changes over time seems, to me, to take a lot of the mystery out of human preference uncertainty. It’s hard to picture that I have some true platonic utility function. It’s much easier to interpret myself as having some preferences right now (which I still have uncertainty about, but which I have some introspective access of), but, also being the kind of entity who shifts preferences over time, and mostly in a way which I myself endorse. In some sense you can see me as converging to a true utility function; however, this “true utility function” is a (non-constructive) consequence of my process of deliberation, and the process of deliberation takes a primary role.
I recognize that this isn’t exactly the same perspective captured by my first reply.
So we have this nice picture, where rationality is characterized by non-exploitability wrt a specific class of potential exploiters.
I’m not convinced this is the right desideratum for that purpose. Why should we care about exploitability by traders if making such trades is not actually possible given the environment and the utility function? IMO epistemic rationality is subservient to instrumental rationality, so our desiderata should be derived from the later.
Human value-uncertainty is not particularly well-captured by Bayesian uncertainty, as I imagine you’ll agree… It’s hard to picture that I have some true platonic utility function.
Actually I am rather skeptical/agnostic on this. For me it’s fairly easy to picture that I have a “platonic” utility function, except that the time discount is dynamically inconsistent (not exponential).
I am in favor of exploring models of preferences which admit all sorts of uncertainty and/or dynamic inconsistency, but (i) it’s up to debate how much degrees of freedom we need to allow there and (ii) I feel that the case logical induction is the right framework for this is kinda weak (but maybe I’m missing something).
I’m not convinced this is the right desideratum for that purpose. Why should we care about exploitability by traders if making such trades is not actually possible given the environment and the utility function? IMO epistemic rationality is subservient to instrumental rationality, so our desiderata should be derived from the later.
This does make sense to me, and I view it as a weakness of the idea. However, the productivity of dutch-book type thinking in terms of implying properties which seem appealing for other reasons speaks heavily in favor of it, in my mind. A formal connection to more pragmatic criteria would be great.
But also, maybe I can articulate a radical-probabilist position without any recourse to dutch books… I’ll have to think more about that.
Actually I am rather skeptical/agnostic on this. For me it’s fairly easy to picture that I have a “platonic” utility function, except that the time discount is dynamically inconsistent (not exponential).
I’m not sure how to double crux with this intuition, unfortunately. When I imagine the perspective you describe, I feel like it’s rolling all dynamic inconsistency into time-preference and ignoring the role of deliberation.
My claim is that there is a type of change-over-time which is due to boundedness, and which looks like “dynamic inconsistency” from a classical bayesian perspective, but which isn’t inherently dynamically inconsistent. EG, if you “sleep on it” and wake up with a different, firmer-feeling perspective, without any articulable thing you updated on. (My point isn’t to dogmatically insist that you haven’t updated on anything, but rather, to point out that it’s useful to have the perspective where we don’t need to suppose there was evidence which justifies the update as Bayesian, in order for it to be rational.)
Actually I am rather skeptical/agnostic on this. For me it’s fairly easy to picture that I have a “platonic” utility function, except that the time discount is dynamically inconsistent (not exponential).
I am in favor of exploring models of preferences which admit all sorts of uncertainty and/or dynamic inconsistency, but (i) it’s up to debate how much degrees of freedom we need to allow there and (ii) I feel that the case logical induction is the right framework for this is kinda weak (but maybe I’m missing something).
It’s clear that you understand logical induction pretty well, so while I feel like you’re missing something, I’m not clear on what that could be.
I think maybe the more fruitful branch of this conversation (as opposed to me trying to provide an instrumental justification for radical probabilism, though I’m still interested in that) is the question of describing the human utility function.
The logical induction picture isn’t strictly at odds with a platonic utility function, I think, since we can consider the limit. (I only claim that this isn’t the best way to think about it in general, since Nature didn’t decide a platonic utility function for us and then design us such that our reasoning has the appropriate limit.)
For example, one case which to my mind argues in favor of the logical induction approach to preferences: the procrastination paradox. All you want to do is ensure that the button is pressed at some point. This isn’t a particularly complex or unrealistic preference for an agent to have. Yet, it’s unclear how to make computable beliefs think about this appropriately. Logical induction provides a theory about how to think about this kind of goal. (I haven’t thought much about how TRL would handle it.)
Agree or disagree: agents can sensibly pursue Δ2 objectives? And, do you think that question is cruxy for you?
I lean towards some kind of finitism or constructivism, and am skeptical of utility functions which involve unbounded quantifiers. But also, how does LI help with the procrastination paradox? I don’t think I’ve seen this result.
What I’m referring to is that LI given a notion of rational uncertain expectation for the procrastination paradox—so, less a positive result, more a framework for thinking about what behavior is reasonable.
However, I also think LIDT solves the problem in practical terms:
In the pure procrastination-paradox problem, LIDT will eventually push the button if its logic is sound. If it did not, it would mean the conditional probability of ever pressing the button given not pressing it today remains forever higher than the conditional probability of ever pressing it today. However, the expectation can be split into the probability it gets pushed today, and the probability that it gets pushed on any day later than today. The LI should eventually know that the conditional probability of ever pressing the button given pressing it today is arbitrarily close to 1. So in order to never press the button, the conditional probability of ever pressing it in the future (given not pressing today) would have to go to 1 (faster than the probability of it ever being pressed given pressing it today). I don’t think this can happen, since there will be some nonzero limit probability that the button will never be pressed (that is, there will be supposing the button is in fact never pressed).
In a situation where there is some actual reason to procrastinate (there are other sources of utility), but we place very high value on eventually pressing the button, it may be that the button will never be pressed? However, this will only happen if we’re subjectively confident that it will eventually be pressed, and always have something better to do in the mean time. The second part seems pretty difficult. So maybe we can also prove that we eventually press the button in this case, as well.
My basic argument is we can model this sort of preference, so why rule it out as a possible human preference? You may be philosophically confident in finitist/constructivist values, but are you so confident that you’d want to lock unbounded quantifiers out of the space of possible values for value learning?
However, I also think LIDT solves the problem in practical terms:
What is LIDT exactly? I can try to guess but I rather make sure we’re both talking about the same thing.
My basic argument is we can model this sort of preference, so why rule it out as a possible human preference? You may be philosophically confident in finitist/constructivist values, but are you so confident that you’d want to lock unbounded quantifiers out of the space of possible values for value learning?
I agree inasmuch as we actually can model this sort of preferences, for a sufficiently strong meaning of “model”. I feel that it’s much harder to be confident about any detailed claim about human values than about the validity of a generic theory of rationality. Therefore, if the ultimate generic theory of rationality imposes some conditions on utility functions (while still leaving a very rich space of different utility functions), that will lead me to try formalizing human values within those constraints. Of course, given a candidate theory, we should poke around and see whether it can be extended to weaken the constraints.
I agree inasmuch as we actually can model this sort of preferences, for a sufficiently strong meaning of “model”. I feel that it’s much harder to be confident about any detailed claim about human values than about the validity of a generic theory of rationality. Therefore, if the ultimate generic theory of rationality imposes some conditions on utility functions (while still leaving a very rich space of different utility functions), that will lead me to try formalizing human values within those constraints. Of course, given a candidate theory, we should poke around and see whether it can be extended to weaken the constraints.
Right, I agree with this. The situation as I see it is that there’s a concrete theory of rationality (logical induction) which I’m using in this way, and it is suggesting to me that your theory (InfraBayes) can still be extended somewhat.
My argument that we want this particular extension is basically as follows: human values can be thought of as the endpoint of human philosophical deliberation about values. (I am thinking of logical induction as a formalization of philosophical deliberation over time.) This endpoint seems limit-computable, but not necessarily computable. Now, it’s also possible that at this endpoint, humans would have a more compact (ie, computable) representation of values. However, why assume this?
(My hope is that by appealing to deliberation like this, my argument has more force than if I was only relying on the strength of logical induction as a theory of rationality. The idea of deliberation gives us a general reason to expect that limit-computable is the right place to look.)
What is LIDT exactly?
I’m not sure details matter very much here, but I’m provisionally happy to spell out LIDT as:
Specify some (bounded-value) LUV to use as “utility”
Make decisions by looking at conditional expectations of that LUV given actions.
I would be convinced if you had a theory of rationality that is a Pareto improvement on IB (i.e. has all the good properties of IB + a more general class of utility functions). However, LI doesn’t provide this AFAICT. That said, I would be interested to see some rigorous theorem about LIDT solving procrastination-like problems.
As to philosophical deliberation, I feel some appeal in this point of view, but I can also easily entertain a different point of view: namely, that human values are more or less fixed and well-defined whereas philosophical deliberation is just a “show” for game theory reasons. Overall, I place much less weight on arguments that revolve around the presumed nature of human values compared to arguments grounded in abstract reasoning about rational agents.
I don’t believe that LI provides such a Pareto improvement, but I suspect that there’s a broader theory which contains the two.
Overall, I place much less weight on arguments that revolve around the presumed nature of human values compared to arguments grounded in abstract reasoning about rational agents.
Ah. I was going for the human-values argument because I thought you might not appreciate the rational-agent argument. After all, who cares what general rational agents can value, if human values happen to be well-represented by infrabayes?
But for general rational agents, rather than make the abstract deliberation argument, I would again mention the case of LIDT in the procrastination paradox, which we’ve already discussed.
Or, I would make the radical probabilist argument against rigid updating, and the ‘orthodox’ argument against fixed utility functions. Combined, we get a picture of “values” which is basically a market for expected values, where prices can change over time (in a “radical” way that doesn’t necessarily spring from an update on a proposition), but which follow some coherence rules like an expectation of an expectation equals an expectation. One formalization of this is Skyrms’. Another is your generalization of LI (iirc).
So to sum it up, my argument for general rational agents is:
In general, we need not update in a rigid way; we can develop a meaningful theory of ‘fluid’ updates, so long as we respect some coherence constraints. In light of this generalization, restriction to ‘rigid’ updates seems somewhat arbitrary (ie there does not seem to be a strong motivation to make the restriction from rationality alone).
Separately, there is no need to actually have a utility function if we have a coherent expectation.
Putting the two together, we can study coherent expectations where the notion of ‘coherence’ doesn’t assume rigid updates.
However, this argument of course does not account for InfraBayes. I suspect your real crux is the plausibility of coming up with a unifying theory which gets both radical-probabilism stuff and InfraBayes stuff. This does seem challenging, but I strongly suspect it to be possible. Indeed, it seems like it might have to do with the idea of a market which maintains a buy/sell spread rather than giving one price for a good.
I’m not convinced this is the right desideratum for that purpose. Why should we care about exploitability by traders if making such trades is not actually possible given the environment and the utility function? IMO epistemic rationality is subservient to instrumental rationality, so our desiderata should be derived from the later.
So, one point is that the InfraBayes picture still gives epistemics an important role: the kind of guarantee arrived at is a guarantee that you won’t do too much worse than the most useful partial model expects. So, we can think about generalized partial models which update by thinking longer in addition to taking in sense-data.
I suppose TRL can model this by observing what those computations would say, in a given situation, and using partial models which only “trust computation X” rather than having any content of their own. Is this “complete” in an appropriate sense? Can we always model a would-be radical-infrabayesian as a TRL agent observing what that radical-infrabayesian would think?
Even if true, there may be a significant computational complexity gap between just doing the thing vs modeling it in this way.
Yes, I’m pretty sure we have that kind of completeness. Obviously representing all hypotheses in this opaque form would give you poor sample and computational complexity, but you can do something midway: use black-box programs as components in your hypothesis but also have some explicit/transparent structure.
The abstract theory of InfraBayes (like the abstract theory of Bayes) elides computational concerns.
In reality, all of ML can more or less be thought of as using a big search for good models, where “good” means something approximately like MAP, although we can also consider more sophisticated variational targets. This introduces two different types of approximation:
The optimization target is approximate.
The optimization itself gives only approximate maxima.
What we want out of InfraBayes is a bounded regret guarantee (in settings where we previously didn’t know how to get one). What we have is a picture of how to get that if we can actually do the generalized Bayesian update. What we might want is a picture of how to do that more generally, when we can’t actually compute the full update.
Can we get such a thing with InfraBayes?
In other words, search is a very basic type of logical uncertainty. Currently, we don’t have much of a model of that, except “Bayesian Search” (which does not provide any nice regret bounds that I know of, although I may be ignorant). We might need such a thing in order to get nice guarantees for systems which employ search internally. Can we get it?
Obviously, we can do the bayesian-search thing with InfraBayes substituted in, which already probably provides some kind of guarantee which couldn’t be gotten otherwise. However, the challenge is to get the guarantee to carry all the way through to the end result.
My hope is that we will eventually have computationally feasible algorithms that satisfy provable (or at least conjectured) infra-Bayesian regret bounds for some sufficiently rich hypothesis space. Currently, even in the Bayesian case, we only have such algorithms for poor hypothesis spaces, such as MDPs with a small number of states. We can also rule out such algorithms for some large hypothesis spaces, such as short programs with a fixed polynomial-time bound. In between, there should be some hypothesis space which is small enough to be feasible and rich enough to be useful. Indeed, it seems to me that the existence of such a space is the simplest explanation for the success of deep learning (that is, for the ability to solve a diverse array of problems with relatively simple and domain-agnostic algorithms). But, at present I only have speculations about what this space looks like.
To further elaborate, this post discusses ways a Bayesian might pragmatically prefer non-Bayesian updates. Some of them don’t carry over, for sure, but I expect the general idea to translate: InfraBayesians need some unrealistic assumptions to reflectively justify the InfraBayesian update in contrast to other updates. (But I am not sure which assumptions to point out, atm.)
In particular, it’s easy to believe that some computation knows more than you.
Yes, I think TRL captures this notion. You have some Knightian uncertainty about the world, and some Knightian uncertainty about the result of a computation, and the two are entangled.
I guess we can try studying Troll Bridge using infra-Bayesian modal logic, but atm I don’t know what would result.
Ah, but there is a sense in which it doesn’t. The radical update rule is equivalent to updating on “secret evidence”. And in TRL we have such secret evidence. Namely, if we only look at the agent’s beliefs about “physics” (the environment), then they would be updated radically, because of secret evidence from “mathematics” (computations).
I agree that radical probabilism can be thought of as bayesian-with-a-side-channel, but it’s nice to have a more general characterization where the side channel is black-box, rather than an explicit side-channel which we explicitly update on. This gives us a picture of the space of rational updates. EG, the logical induction criterion allows for a large space of things to count as rational. We get to argue for constraints on rational behavior by pointing to the existence of traders which enforce those constraints, while being agnostic about what’s going on inside a logical inductor. So we have this nice picture, where rationality is characterized by non-exploitability wrt a specific class of potential exploiters.
Here’s an argument for why this is an important dimension to consider:
Human value-uncertainty is not particularly well-captured by Bayesian uncertainty, as I imagine you’ll agree. One particular complaint is realizability: we have no particular reason to assume that human preferences are within any particular space of hypotheses we can write down.
One aspect of this can be captured by InfraBayes: it allows us to eliminate the realizability assumption, instead only assuming that human preferences fall within some set of constraints which we can describe.
However, there is another aspect to human preference-uncertainty: human preferences change over time. Some of this is irrational, but some of it is legitimate philosophical deliberation.
And, somewhat in the spirit of logical induction, humans do tend to eventually address the most egregious irrationalities.
Therefore, I tend to think that toy models of alignment (such as CIRL, DRL, DIRL) should model the human as a radical probabilist; not because it’s a perfect model, but because it constitutes a major incremental improvement wrt modeling what kind of uncertainty humans have over our own preferences.
Recognizing preferences as a thing which naturally changes over time seems, to me, to take a lot of the mystery out of human preference uncertainty. It’s hard to picture that I have some true platonic utility function. It’s much easier to interpret myself as having some preferences right now (which I still have uncertainty about, but which I have some introspective access of), but, also being the kind of entity who shifts preferences over time, and mostly in a way which I myself endorse. In some sense you can see me as converging to a true utility function; however, this “true utility function” is a (non-constructive) consequence of my process of deliberation, and the process of deliberation takes a primary role.
I recognize that this isn’t exactly the same perspective captured by my first reply.
I’m not convinced this is the right desideratum for that purpose. Why should we care about exploitability by traders if making such trades is not actually possible given the environment and the utility function? IMO epistemic rationality is subservient to instrumental rationality, so our desiderata should be derived from the later.
Actually I am rather skeptical/agnostic on this. For me it’s fairly easy to picture that I have a “platonic” utility function, except that the time discount is dynamically inconsistent (not exponential).
I am in favor of exploring models of preferences which admit all sorts of uncertainty and/or dynamic inconsistency, but (i) it’s up to debate how much degrees of freedom we need to allow there and (ii) I feel that the case logical induction is the right framework for this is kinda weak (but maybe I’m missing something).
This does make sense to me, and I view it as a weakness of the idea. However, the productivity of dutch-book type thinking in terms of implying properties which seem appealing for other reasons speaks heavily in favor of it, in my mind. A formal connection to more pragmatic criteria would be great.
But also, maybe I can articulate a radical-probabilist position without any recourse to dutch books… I’ll have to think more about that.
I’m not sure how to double crux with this intuition, unfortunately. When I imagine the perspective you describe, I feel like it’s rolling all dynamic inconsistency into time-preference and ignoring the role of deliberation.
My claim is that there is a type of change-over-time which is due to boundedness, and which looks like “dynamic inconsistency” from a classical bayesian perspective, but which isn’t inherently dynamically inconsistent. EG, if you “sleep on it” and wake up with a different, firmer-feeling perspective, without any articulable thing you updated on. (My point isn’t to dogmatically insist that you haven’t updated on anything, but rather, to point out that it’s useful to have the perspective where we don’t need to suppose there was evidence which justifies the update as Bayesian, in order for it to be rational.)
It’s clear that you understand logical induction pretty well, so while I feel like you’re missing something, I’m not clear on what that could be.
I think maybe the more fruitful branch of this conversation (as opposed to me trying to provide an instrumental justification for radical probabilism, though I’m still interested in that) is the question of describing the human utility function.
The logical induction picture isn’t strictly at odds with a platonic utility function, I think, since we can consider the limit. (I only claim that this isn’t the best way to think about it in general, since Nature didn’t decide a platonic utility function for us and then design us such that our reasoning has the appropriate limit.)
For example, one case which to my mind argues in favor of the logical induction approach to preferences: the procrastination paradox. All you want to do is ensure that the button is pressed at some point. This isn’t a particularly complex or unrealistic preference for an agent to have. Yet, it’s unclear how to make computable beliefs think about this appropriately. Logical induction provides a theory about how to think about this kind of goal. (I haven’t thought much about how TRL would handle it.)
Agree or disagree: agents can sensibly pursue Δ2 objectives? And, do you think that question is cruxy for you?
I lean towards some kind of finitism or constructivism, and am skeptical of utility functions which involve unbounded quantifiers. But also, how does LI help with the procrastination paradox? I don’t think I’ve seen this result.
What I’m referring to is that LI given a notion of rational uncertain expectation for the procrastination paradox—so, less a positive result, more a framework for thinking about what behavior is reasonable.
However, I also think LIDT solves the problem in practical terms:
In the pure procrastination-paradox problem, LIDT will eventually push the button if its logic is sound. If it did not, it would mean the conditional probability of ever pressing the button given not pressing it today remains forever higher than the conditional probability of ever pressing it today. However, the expectation can be split into the probability it gets pushed today, and the probability that it gets pushed on any day later than today. The LI should eventually know that the conditional probability of ever pressing the button given pressing it today is arbitrarily close to 1. So in order to never press the button, the conditional probability of ever pressing it in the future (given not pressing today) would have to go to 1 (faster than the probability of it ever being pressed given pressing it today). I don’t think this can happen, since there will be some nonzero limit probability that the button will never be pressed (that is, there will be supposing the button is in fact never pressed).
In a situation where there is some actual reason to procrastinate (there are other sources of utility), but we place very high value on eventually pressing the button, it may be that the button will never be pressed? However, this will only happen if we’re subjectively confident that it will eventually be pressed, and always have something better to do in the mean time. The second part seems pretty difficult. So maybe we can also prove that we eventually press the button in this case, as well.
My basic argument is we can model this sort of preference, so why rule it out as a possible human preference? You may be philosophically confident in finitist/constructivist values, but are you so confident that you’d want to lock unbounded quantifiers out of the space of possible values for value learning?
What is LIDT exactly? I can try to guess but I rather make sure we’re both talking about the same thing.
I agree inasmuch as we actually can model this sort of preferences, for a sufficiently strong meaning of “model”. I feel that it’s much harder to be confident about any detailed claim about human values than about the validity of a generic theory of rationality. Therefore, if the ultimate generic theory of rationality imposes some conditions on utility functions (while still leaving a very rich space of different utility functions), that will lead me to try formalizing human values within those constraints. Of course, given a candidate theory, we should poke around and see whether it can be extended to weaken the constraints.
Right, I agree with this. The situation as I see it is that there’s a concrete theory of rationality (logical induction) which I’m using in this way, and it is suggesting to me that your theory (InfraBayes) can still be extended somewhat.
My argument that we want this particular extension is basically as follows: human values can be thought of as the endpoint of human philosophical deliberation about values. (I am thinking of logical induction as a formalization of philosophical deliberation over time.) This endpoint seems limit-computable, but not necessarily computable. Now, it’s also possible that at this endpoint, humans would have a more compact (ie, computable) representation of values. However, why assume this?
(My hope is that by appealing to deliberation like this, my argument has more force than if I was only relying on the strength of logical induction as a theory of rationality. The idea of deliberation gives us a general reason to expect that limit-computable is the right place to look.)
I’m not sure details matter very much here, but I’m provisionally happy to spell out LIDT as:
Specify some (bounded-value) LUV to use as “utility”
Make decisions by looking at conditional expectations of that LUV given actions.
Concrete enough?
I would be convinced if you had a theory of rationality that is a Pareto improvement on IB (i.e. has all the good properties of IB + a more general class of utility functions). However, LI doesn’t provide this AFAICT. That said, I would be interested to see some rigorous theorem about LIDT solving procrastination-like problems.
As to philosophical deliberation, I feel some appeal in this point of view, but I can also easily entertain a different point of view: namely, that human values are more or less fixed and well-defined whereas philosophical deliberation is just a “show” for game theory reasons. Overall, I place much less weight on arguments that revolve around the presumed nature of human values compared to arguments grounded in abstract reasoning about rational agents.
I don’t believe that LI provides such a Pareto improvement, but I suspect that there’s a broader theory which contains the two.
Ah. I was going for the human-values argument because I thought you might not appreciate the rational-agent argument. After all, who cares what general rational agents can value, if human values happen to be well-represented by infrabayes?
But for general rational agents, rather than make the abstract deliberation argument, I would again mention the case of LIDT in the procrastination paradox, which we’ve already discussed.
Or, I would make the radical probabilist argument against rigid updating, and the ‘orthodox’ argument against fixed utility functions. Combined, we get a picture of “values” which is basically a market for expected values, where prices can change over time (in a “radical” way that doesn’t necessarily spring from an update on a proposition), but which follow some coherence rules like an expectation of an expectation equals an expectation. One formalization of this is Skyrms’. Another is your generalization of LI (iirc).
So to sum it up, my argument for general rational agents is:
In general, we need not update in a rigid way; we can develop a meaningful theory of ‘fluid’ updates, so long as we respect some coherence constraints. In light of this generalization, restriction to ‘rigid’ updates seems somewhat arbitrary (ie there does not seem to be a strong motivation to make the restriction from rationality alone).
Separately, there is no need to actually have a utility function if we have a coherent expectation.
Putting the two together, we can study coherent expectations where the notion of ‘coherence’ doesn’t assume rigid updates.
However, this argument of course does not account for InfraBayes. I suspect your real crux is the plausibility of coming up with a unifying theory which gets both radical-probabilism stuff and InfraBayes stuff. This does seem challenging, but I strongly suspect it to be possible. Indeed, it seems like it might have to do with the idea of a market which maintains a buy/sell spread rather than giving one price for a good.
So, one point is that the InfraBayes picture still gives epistemics an important role: the kind of guarantee arrived at is a guarantee that you won’t do too much worse than the most useful partial model expects. So, we can think about generalized partial models which update by thinking longer in addition to taking in sense-data.
I suppose TRL can model this by observing what those computations would say, in a given situation, and using partial models which only “trust computation X” rather than having any content of their own. Is this “complete” in an appropriate sense? Can we always model a would-be radical-infrabayesian as a TRL agent observing what that radical-infrabayesian would think?
Even if true, there may be a significant computational complexity gap between just doing the thing vs modeling it in this way.
Yes, I’m pretty sure we have that kind of completeness. Obviously representing all hypotheses in this opaque form would give you poor sample and computational complexity, but you can do something midway: use black-box programs as components in your hypothesis but also have some explicit/transparent structure.
OK, so, here is a question.
The abstract theory of InfraBayes (like the abstract theory of Bayes) elides computational concerns.
In reality, all of ML can more or less be thought of as using a big search for good models, where “good” means something approximately like MAP, although we can also consider more sophisticated variational targets. This introduces two different types of approximation:
The optimization target is approximate.
The optimization itself gives only approximate maxima.
What we want out of InfraBayes is a bounded regret guarantee (in settings where we previously didn’t know how to get one). What we have is a picture of how to get that if we can actually do the generalized Bayesian update. What we might want is a picture of how to do that more generally, when we can’t actually compute the full update.
Can we get such a thing with InfraBayes?
In other words, search is a very basic type of logical uncertainty. Currently, we don’t have much of a model of that, except “Bayesian Search” (which does not provide any nice regret bounds that I know of, although I may be ignorant). We might need such a thing in order to get nice guarantees for systems which employ search internally. Can we get it?
Obviously, we can do the bayesian-search thing with InfraBayes substituted in, which already probably provides some kind of guarantee which couldn’t be gotten otherwise. However, the challenge is to get the guarantee to carry all the way through to the end result.
My hope is that we will eventually have computationally feasible algorithms that satisfy provable (or at least conjectured) infra-Bayesian regret bounds for some sufficiently rich hypothesis space. Currently, even in the Bayesian case, we only have such algorithms for poor hypothesis spaces, such as MDPs with a small number of states. We can also rule out such algorithms for some large hypothesis spaces, such as short programs with a fixed polynomial-time bound. In between, there should be some hypothesis space which is small enough to be feasible and rich enough to be useful. Indeed, it seems to me that the existence of such a space is the simplest explanation for the success of deep learning (that is, for the ability to solve a diverse array of problems with relatively simple and domain-agnostic algorithms). But, at present I only have speculations about what this space looks like.
To further elaborate, this post discusses ways a Bayesian might pragmatically prefer non-Bayesian updates. Some of them don’t carry over, for sure, but I expect the general idea to translate: InfraBayesians need some unrealistic assumptions to reflectively justify the InfraBayesian update in contrast to other updates. (But I am not sure which assumptions to point out, atm.)
Yes, I think TRL captures this notion. You have some Knightian uncertainty about the world, and some Knightian uncertainty about the result of a computation, and the two are entangled.