Thanks. I think we mainly agree here.
Stuart_Armstrong(Stuart Armstrong)
Karma: 15,492
Why unriggable *almost* implies uninfluenceable
A possible preference algorithm
If you don’t design for extrapolation, you’ll extrapolate poorly—possibly fatally
Which counterfactuals should an AI follow?
Toy model of preference, bias, and extra information
No. But I expect that it would be much more in the right ballpark than other approaches, and I think it might be refined to be correct.
Look at the paper linked for more details ( https://arxiv.org/abs/1712.05812 ).
Basically “humans are always fully rational and always take the action they want to” is a full explanation of all of human behaviour, that is strictly simpler than any explanation which includes human biases and bounded rationality.
Preferences and biases, the information argument
“Aha! We seem to be past the inflection point!”
It’s generally possible to see where the inflection point is, when we’re past it.
Possibly. What would be the equivalent of a dampening term for a superexponential? A further growth term?
Why sigmoids are so hard to predict
Connecting the good regulator theorem with semantics and symbol grounding
But if you are expecting a 100% guarantee that the uncertainty metrics will detect every possible bad situation
I’m more thinking of how we could automate the navigating of these situations. The detection will be part of this process, and it’s not a Boolean yes/no, but a matter of degree.
I agree that once you have landed in the bad situation, mitigation options might be much the same, e.g. switch off the agent.
I’m most interested in mitigation options the agent can take itself, when it suspects it’s out-of-distribution (and without being turned off, ideally).
Thanks! Lots of useful insights in there.
So I might classify moving out-of-distribution as something that happens to a classifier or agent, and model splintering as something that the machine learning system does to itself.
Why do you think it’s important to distinguish these two situations? It seems that the insights for dealing with one situation may apply to the other, and vice versa.
Cheers! My opinion on category theory has changed a bit, because of this post; by making things fit into the category formulation, I developed insights into how general relations could be used to connect different generalised models.
Thanks! Corrected both of those; is a subset of .
Thanks! That’s useful to know.
I like the subagent approach there.