Yeah, this is a great connection which I learned about earlier in the summer. I think this theory explains what’s going on when they say
They argue that simple mechanical systems that are postulated to follow this rule show features of “intelligence,” hinting at a connection between this most-human attribute and fundamental physical laws.
Basically, since near-optimal agents tend to go towards states of high power, and near-optimal agents are generally ones which are intelligent, observing an agent moving towards a state of high power is Bayesian evidence that it is intelligent. However, as I understand it, they have the causation wrong: instead of physical laws → power-seeking and intelligence, intelligent goal-directed behavior tends to produce power-seeking.
Yeah, this is a great connection which I learned about earlier in the summer. I think this theory explains what’s going on when they say
Basically, since near-optimal agents tend to go towards states of high power, and near-optimal agents are generally ones which are intelligent, observing an agent moving towards a state of high power is Bayesian evidence that it is intelligent. However, as I understand it, they have the causation wrong: instead of physical laws → power-seeking and intelligence, intelligent goal-directed behavior tends to produce power-seeking.
I agree 100% with everything you said.