...which also means that they didn’t have an empire to back them up?
Stuart_Armstrong(Stuart Armstrong)
Karma: 22,745
Thanks for your research, especially the Afonso stuff. One question for that: were these empires used to gaining/losing small pieces of territory? ie did they really dedicate all their might to getting these ports back, or did they eventually write them off as minor losses not worth the cost of fighting (given Portuguese naval advantages)?
Comparing reward learning/reward tampering formalisms
Based on what I recall reading about Pizzaro’s conquest, I feel you might be underestimating the importance of horses. It took centuries for European powers to figure out how to break a heavy cavalry charge with infantry; the amerindians didn’t have the time to figure it out (see various battles where small cavalry forces routed thousands of troops). Once they had got more used to horses, later Inca forces (though much diminished) were more able to win open battles against the Spanish.
Maybe this was the problem for these empires: they were used to winning open battles, but were presented with a situation where only irregular warfare or siege defences could win. They reacted as an empire, when they should have been reacting as a recalcitrant province.
Probabilities, weights, sums: pretty much the same for reward functions
Also, giving something more points for killing people than making cake sounds like a bad incentive scheme.
In the original cake-or-death example, it wasn’t that killing got more points, it’s that killing is easier (and hence gets more points over time). This is a reflection of the fact that “true” human values are complex and difficult to maximise, but many other values are much easier to maximise.
Learning and manipulating learning
My main note is that my comment was just about the concept of rigging a learning process given a fixed prior over rewards. I certainly agree that the general strategy of “update a distribution over reward functions” has lots of as-yet-unsolved problems.
Ah, ok, I see ^_^ Thanks for making me write this post, though, as it has useful things for other people to see, that I had been meaning to write up for some time.
On your main point: if the prior and updating process are over things that are truly beyond the AI’s influence, then there will be no rigging (or, in my terms: uninfluenceable->unriggable). But there are many things that look like this, that are entirely riggable. For example, “have a prior 50-50 on cake and death, and update according to what the programmer says”. This seems to be a prior-and-update combination, but it’s entirely riggable.
So, another way of seeing my paper is “this thing looks like a prior-and-update process. If it’s also unriggable, then (given certain assumptions) it’s truly beyond the AI’s influence”.
Reward functions and updating assumptions can hide a multitude of sins
thanks! Changed title (and corrected badly formatted footnote)
I agree that for such a system, the optimal policy of the actor is to rig the estimator, and to “intentionally” bias it towards easy-to-satisfy rewards like “the human loves heroin”.
The part that confuses me is why we’re having two separate systems with different objectives where one system is dumb and the other system is smart.
We don’t need to have two separate systems. There’s two meaning to your “bias it towards” phrase: the first one is the informal human one, where “the human loves heroin” is clearly a bias. The second is some formal definition of what is biasing and what isn’t. And the system doesn’t have that. The “estimator” doesn’t “know” that “the human loves heroin” is a bias; instead, it sees this as a perfectly satisfactory way of accomplishing its goals, according to the bridging function it’s been given. There is no conflict between estimator and actor.
Imagine that you have a complex CIRL game that models the real world well but assumes that the human is Boltzmann-rational. [...] Such a policy is going to “try” to learn preferences, learn incorrectly, and then act according to those incorrect learned preferences, but it is not going to “intentionally” rig the learning process.
The AI would not see any of these actions as “rigging”, even if we would.
It might think “hey, I should check whether the human likes heroin by giving them some”, and then think “oh they really do love heroin, I should pump them full of it”.
It will do this if it can’t already predict the effect of giving them heroin.
It won’t think “aha, if I give the human heroin, then they’ll ask for more heroin, causing my Boltzmann-rationality estimator module to predict they like heroin, and then I can get easy points by giving humans heroin”.
If it can predict the effect of giving humans heroin, it will think something like that. It think: “if I give the humans heroin, they’ll ask for more heroin; my Boltzmann-rationality estimator module confirms that this means they like heroin, so I can efficiently satisfy their preferences by giving humans heroin”.
How should AIs update a prior over human preferences?
Distinguishing logistic curves
Distinguishing logistic curves: visual
Kurzweil’s predictions’ individual scores
Strong upvote for writing your own predictions before seeing the 2019 graph.
Plot visualised:
(this is, obviously, very speculative ^_^ )