John said “there was not any point at which I thought my views were importantly misrepresented” when I asked him for comment. (I added this note to the top of the post as a parenthetical; thanks.)
I think the argument Nate gave is at least correct for markets of relatively-highly-intelligent agents, and that was a big update for me (thankyou Nate!). I’m still unsure how far it generalizes to relatively less powerful agents.
Nate left out my other big takeaway: Nate’s argument here implies that there’s probably a lot of money to be made in real-world markets! In practice, it would probably look like an insurance-like contract, by which two traders would commit to the “side-channel trades at non-market prices” required to make them aggregate into an expected utility maximizer. (Obviously the contract wouldn’t be phrased in those terms; most of the work of implementation would be to figure out what trades need to occur in practice under what conditions to achieve aggregability, and then figuring out simple approximations of those conditions to write into contracts.)
In the year since this discussion, I’ve also understood better why Nate seems to care mostly about the principles of relatively-highly-intelligent agents, as opposed to e.g. humans. I think that crux was mostly about corrigibility as an alignment target, and I have updated substantially toward that position as well.
My main remaining disagreement, for purposes of applying this argument to superhuman AI, is that an intelligence which originally develops as a market of relatively-weak agents does not obviously choose to self-modify in the way described in the post, in the process of becoming more intelligent. It’s not clear that the component weak subagents themselves “upgrade”.
Analogy for humans: insofar as human values are well thought of as a “market of weak subagents”, it’s not clear to me that making the individual subagents more capable (to the point where they make the sort of trades required by Nate’s argument) is actually the way I’d prefer to upgrade myself. I’m not convinced that that would actually be the right way to reflectively extend my extant values.
Real markets mostly have it covered, because they have something close to [aggregated] utilons—money, and so exchanges between 2 different goods rarely take place.
Also, any business can be seen as a “side-channel trade”—the market value of one individual’s time is often lower than the value they can produce in cooperation with others.
John said “there was not any point at which I thought my views were importantly misrepresented” when I asked him for comment. (I added this note to the top of the post as a parenthetical; thanks.)
More details:
I think the argument Nate gave is at least correct for markets of relatively-highly-intelligent agents, and that was a big update for me (thankyou Nate!). I’m still unsure how far it generalizes to relatively less powerful agents.
Nate left out my other big takeaway: Nate’s argument here implies that there’s probably a lot of money to be made in real-world markets! In practice, it would probably look like an insurance-like contract, by which two traders would commit to the “side-channel trades at non-market prices” required to make them aggregate into an expected utility maximizer. (Obviously the contract wouldn’t be phrased in those terms; most of the work of implementation would be to figure out what trades need to occur in practice under what conditions to achieve aggregability, and then figuring out simple approximations of those conditions to write into contracts.)
In the year since this discussion, I’ve also understood better why Nate seems to care mostly about the principles of relatively-highly-intelligent agents, as opposed to e.g. humans. I think that crux was mostly about corrigibility as an alignment target, and I have updated substantially toward that position as well.
My main remaining disagreement, for purposes of applying this argument to superhuman AI, is that an intelligence which originally develops as a market of relatively-weak agents does not obviously choose to self-modify in the way described in the post, in the process of becoming more intelligent. It’s not clear that the component weak subagents themselves “upgrade”.
Analogy for humans: insofar as human values are well thought of as a “market of weak subagents”, it’s not clear to me that making the individual subagents more capable (to the point where they make the sort of trades required by Nate’s argument) is actually the way I’d prefer to upgrade myself. I’m not convinced that that would actually be the right way to reflectively extend my extant values.
Real markets mostly have it covered, because they have something close to [aggregated] utilons—money, and so exchanges between 2 different goods rarely take place.
Also, any business can be seen as a “side-channel trade”—the market value of one individual’s time is often lower than the value they can produce in cooperation with others.