I think there’s a way to drop the “for humans/agents like humans” part. Like, we could drop our circuit in the middle of the woods, and sometimes random animals would accidentally turn the knob on the supply, or temperature changes would adjust the resistance in the resistor. “Which counterfactuals actually happen sometimes” doesn’t really seem like the right criterion to use as fundamental here, but it does suggest that there’s something more universal in play.
I think another related qualitative intuition is constructive vs. nonconstructive. “Just turn the knob” is simple and obvious enough to you to be regarded as constructive, not leaving any parts unspecified for a planner to compute. “Just set the voltage to 10V” seems nonconstructive—like it would require further abstract thought to make a plan to make the voltage be 10V. But as we’ve learned, turning knobs is a fairly tricky robotics task, requiring plenty of thought—just thought that’s unconscious in humans.
I think there’s a way to drop the “for humans/agents like humans” part. Like, we could drop our circuit in the middle of the woods, and sometimes random animals would accidentally turn the knob on the supply, or temperature changes would adjust the resistance in the resistor. “Which counterfactuals actually happen sometimes” doesn’t really seem like the right criterion to use as fundamental here, but it does suggest that there’s something more universal in play.
I think another related qualitative intuition is constructive vs. nonconstructive. “Just turn the knob” is simple and obvious enough to you to be regarded as constructive, not leaving any parts unspecified for a planner to compute. “Just set the voltage to 10V” seems nonconstructive—like it would require further abstract thought to make a plan to make the voltage be 10V. But as we’ve learned, turning knobs is a fairly tricky robotics task, requiring plenty of thought—just thought that’s unconscious in humans.