it seems like this problem can’t have existed? why does miri think this is a problem? it seems like it’s only a problem if you ever thought infinite aixi was a valid model. it … was never valid, for anything. it’s not a good theoretical model, it’s a fake theoretical model that we used as approximately valid even though we know it’s catastrophically nonsensical; finite aixi begins to work, of course, but at no point could we actually treat alexei as an independent agent; we’re all just physical circuits, alexei as much as emmy. to the degree that a physical system can exhibit intelligence, it’s by budgeting compute. I feel like this article is the kind of thing one writes if your theoretical models don’t even allow neural networks to exist, and I continue to think that the main reason the miri folks come up with nonsense is because they think you can make exact proofs about physics, when like, you can only ever even in principle make margin proofs, and those proofs are only valid to the degree you can trust that your verifier didn’t have a random failure.
like, why do we need to have a model of intelligence as separate from physical systems? can’t we just make our assertions directly about physical systems?
in terms of logical counterfactuals, it seems to me that a counterfactual is when a set of particles that have a representation for a larger set of particles assign their representation to values that the larger set of particles didn’t take. so, being logical counterfactuals isn’t special—all counterfactuals are to some degree logical.
if your decision theory of counterfactuals is a functional counterfactual, then it seems to me that that just means your counterfactual is a statement like “the universe, but any observations that match <x> get overwritten for consideration by attributes <y>”. where <x> is a matcher on a series of physical states; if a system passes through physical states that match, then <x>’s output is <y> instead of the physical output. that doesn’t seem complicated to me.
but maybe it only seems obvious because I’ve read discovering agents and in 2018 nobody had done that. maybe zac kenton solved this and I’m just echoing the solution and thinking that in retrospect the idea of being confused is confusing.
but I still feel this deep frustration with the question, like, why are you even asking that? it’s clearly a necessarily useless question to ask “but aixi”, because, everything was always going to be embedded, what else would it mean to be an agent besides to be a hunk of matter reacting to impulses? this feels like the kind of nonsense written by someone who doesn’t have a mindset of searching for the most powerful algorithm and trying to build it, but instead of someone who has epistemic learned helplessness about the idea that it’s possible to characterize what algorithms are worth running.
have you noticed you’re not one algorithm, but a network of distributed microcomputers we call neurons, none of whom can ever entirely trust that they heard each other correctly?
[edit: yeah on slower reflection, I think this was guessable but not obvious before papers were published that clarify this perspective.]
and they were blindsided by alphago, whereas @jacob_cannell and I could post screenshots of our old google hangouts conversation from january 2016 where we had been following the go ai research and had sketched out the obvious next additions that in fact ended up being a reasonable guess at what would work. we were surprised it worked quite as well as it did quite so soon, and I lost a bet that it wouldn’t beat lee sedol overall, but dang it’s frustrating how completely blindsided the aixi model was by the success, and yet it stuck around.
You mean shouldn’t have existed?
no I mean was always a deeply confused question whose resolution is to say that the question is invalid rather than to answer—not “shouldn’t have been asked”, but “was asking about a problem that could not have been in the territory because the model was invalid”. How do you model embedded agency? by giving up on the idea that there are coherent ways to separate the universe completely. the ideal representation of friendliness can be applied from a god’s-eye perspective to any two arbitrary blocks of matter to ask how friendly they have been to each other over a particular time period.
but maybe that was what they were asking the whole time, and the origin of my frustration was the fact that they thought they had a gold standard to compare to.
yeah it does seem like probably a lot of why this seems so obvious to me is that I was having inklings of the idea that you need smooth representation of agency and friendliness, and then discovering agents dropped and nailed down what I was looking for and now I just think it’s obvious and have a hard time imagining it not being.
or maybe the issue is that I consider physical laws to be things that particles know about each other? that is, your learning system can start with effectively no knowledge about the behavior of other systems; it gains that knowledge by bumping into them, and the knowledge gets squeezed through a series of conditional resonators of some kind (this should be fully general to all possible intelligent hunks of matter!) into a squashed and rotated dynamical system that has matching transition dynamics and equivalences as the external world as demonstrated by observation. even if you include genetics, this is still true—information got into the genome by the aggregate intelligent behavior of the history of evolutionary life!
okay so I’m reading https://intelligence.org/2018/10/29/embedded-agents/.
it seems like this problem can’t have existed? why does miri think this is a problem? it seems like it’s only a problem if you ever thought infinite aixi was a valid model. it … was never valid, for anything. it’s not a good theoretical model, it’s a fake theoretical model that we used as approximately valid even though we know it’s catastrophically nonsensical; finite aixi begins to work, of course, but at no point could we actually treat alexei as an independent agent; we’re all just physical circuits, alexei as much as emmy. to the degree that a physical system can exhibit intelligence, it’s by budgeting compute. I feel like this article is the kind of thing one writes if your theoretical models don’t even allow neural networks to exist, and I continue to think that the main reason the miri folks come up with nonsense is because they think you can make exact proofs about physics, when like, you can only ever even in principle make margin proofs, and those proofs are only valid to the degree you can trust that your verifier didn’t have a random failure.
like, why do we need to have a model of intelligence as separate from physical systems? can’t we just make our assertions directly about physical systems?
in terms of logical counterfactuals, it seems to me that a counterfactual is when a set of particles that have a representation for a larger set of particles assign their representation to values that the larger set of particles didn’t take. so, being logical counterfactuals isn’t special—all counterfactuals are to some degree logical.
if your decision theory of counterfactuals is a functional counterfactual, then it seems to me that that just means your counterfactual is a statement like “the universe, but any observations that match <x> get overwritten for consideration by attributes <y>”. where <x> is a matcher on a series of physical states; if a system passes through physical states that match, then <x>’s output is <y> instead of the physical output. that doesn’t seem complicated to me.
but maybe it only seems obvious because I’ve read discovering agents and in 2018 nobody had done that. maybe zac kenton solved this and I’m just echoing the solution and thinking that in retrospect the idea of being confused is confusing.
but I still feel this deep frustration with the question, like, why are you even asking that? it’s clearly a necessarily useless question to ask “but aixi”, because, everything was always going to be embedded, what else would it mean to be an agent besides to be a hunk of matter reacting to impulses? this feels like the kind of nonsense written by someone who doesn’t have a mindset of searching for the most powerful algorithm and trying to build it, but instead of someone who has epistemic learned helplessness about the idea that it’s possible to characterize what algorithms are worth running.
have you noticed you’re not one algorithm, but a network of distributed microcomputers we call neurons, none of whom can ever entirely trust that they heard each other correctly?
You mean shouldn’t have existed?
Many did back in the day...very vociferously in some cases.
LW/Miri has a foundations problem. The foundational texts weren’t written by someone with knowledge of AI, or the other subjects.
[edit: yeah on slower reflection, I think this was guessable but not obvious before papers were published that clarify this perspective.]
and they were blindsided by alphago, whereas @jacob_cannell and I could post screenshots of our old google hangouts conversation from january 2016 where we had been following the go ai research and had sketched out the obvious next additions that in fact ended up being a reasonable guess at what would work. we were surprised it worked quite as well as it did quite so soon, and I lost a bet that it wouldn’t beat lee sedol overall, but dang it’s frustrating how completely blindsided the aixi model was by the success, and yet it stuck around.
no I mean was always a deeply confused question whose resolution is to say that the question is invalid rather than to answer—not “shouldn’t have been asked”, but “was asking about a problem that could not have been in the territory because the model was invalid”. How do you model embedded agency? by giving up on the idea that there are coherent ways to separate the universe completely. the ideal representation of friendliness can be applied from a god’s-eye perspective to any two arbitrary blocks of matter to ask how friendly they have been to each other over a particular time period.
but maybe that was what they were asking the whole time, and the origin of my frustration was the fact that they thought they had a gold standard to compare to.
yeah it does seem like probably a lot of why this seems so obvious to me is that I was having inklings of the idea that you need smooth representation of agency and friendliness, and then discovering agents dropped and nailed down what I was looking for and now I just think it’s obvious and have a hard time imagining it not being.
or maybe the issue is that I consider physical laws to be things that particles know about each other? that is, your learning system can start with effectively no knowledge about the behavior of other systems; it gains that knowledge by bumping into them, and the knowledge gets squeezed through a series of conditional resonators of some kind (this should be fully general to all possible intelligent hunks of matter!) into a squashed and rotated dynamical system that has matching transition dynamics and equivalences as the external world as demonstrated by observation. even if you include genetics, this is still true—information got into the genome by the aggregate intelligent behavior of the history of evolutionary life!