I think it’s worth distinguishing between “smallest” and “fastest” circuits.
A note on smallest.
1) Consider a travelling salesman problem and a small program that brute-forces the solution to it. If the “deamon” wants to make a travelling salesman visit a particular city first, then they would simply order the solution space to consider it first. This has no guarantee of working, but the deamon would get what it wants some of the time. More generally, if there is a class of solutions we are indifferent to, but daemons have a preference order over, then nearly all deterministic algorithms could be seen as deamons. That said, this situation may be “acceptable” and it’s worth re-defining the problem to exactly understand what is acceptable and what isn’t.
A note on fastest
2) Consider a prime-generation problem, where we want some large primes between 10^100 and 10^200. A simple algorithm that hardcodes a set of primes and returns them is “fast”. This isn’t the smallest, since it has to store the primes. In a less silly example, a general prime-returning algorithm could only look for primes of particular types, such as Mersenne primes. The general intuition is that optimizations that make algorithms “faster” could come at a cost of forcing a particular probability distribution on the solution.
This is really good, however i would love some additional discussion on the way that the current optimization changes the user.
Keep in mind, when facebook optimizes “clicks” or “scrolls”, it does so by altering user behavior, thus altering the user’s internal S1 model of what is important. This could frequently lead to a distortion of reality, beliefs and self-esteem. There have been many articles and studies correlating facebook usage with mental health. However, simply understanding “optimization” is enough evidence that this is happening.
While, a lot of these issues are pushed under the same umbrella of “digital addiction,” i think facebook is a lot more of a serious problem that, say video games. Video games do not, as a rule, act through the very social channels that are helpful to reducing mental illness. Facebook does.
Also another problem is facebook’s internal culture that, as of 4 years ago was very marked by the cool-aid that somehow promised unbelievable power(1 billion users, horray) without necessarily caring about responsibility (all we want to do is make the world open and connected, why is everyone mad at us).
This problem is also compounded by the fact that facebook get a lot of shitty critiques (like the critique of the fact that they run A/B tests at all) and has thus learned to ignore legitimate questions of value learning.
full disclosure, i used to work at FB.
hmm, looks like the year is wrong and the delete button has failed to work :(
Maybe this have been said before, but here is a simple idea:
Directly specify a utility function U which you are not sure about, but also discount AI’s own power as part of it. So the new utility function is U—power(AI), where power is a fast growing function of a mix of AI’s source code complexity, intelligence, hardware, electricity costs. One needs to be careful of how to define “self” in this case, as a careful redefinition by the AI will remove the controls.
One also needs to consider the creation of subagents with proper utilities as well, since in a naive implementation, sub-agents will just optimize U, without restrictions.
This is likely not enough, but has the advantage that the AI does not have a will to become stronger a priori, which is better than boxing an AI which does.
Well, i get where you are coming from with Goodhart’s Law, but that’s not the question. Formally speaking, if we take the set of all utility functions with complexity < N = FIXED complexity number, then one of them is going to be the “best”, i.e. most correlated with the “true utility” function which we can’t compute.
As you point out, with we are selecting utilities that are too simple, such as straight up life expectancy, then even the “best” function is not “good enough” to just punch into an AGI because it will likely overfit and produce bad consequences. However we can still reason about “better” or “worse” measures of societies. People might complain about un-employment rate, but it’s a crappy metric to base your decision about which societies are over-all better than others, plus it’s easier to game.
The use of at least “trying” to formalize values means we can at least have a set of metrics, that’s not too large that we might care about in arguments like: “but the AGI reduced GDP, well it also reduced suicide rate”? Which is more important? Without a simple guidance of simply something we value, it’s going to be a long and UN-productive debate.
Regarding 2: So, I am a little surprised that step 2: Valuable goals cannot be directly specified is taken as a given.
If we consider an AI as rational optimizer of the ONE TRUE UTILITY FUNCTION, we might want to look for best available approximations of it short term. The function i have in mind is life expectancy(DALY or QALY), since to me, it is easier to measure than happiness. It also captures a lot of intuition when you ask a person the following hypothetical:
if you could be born in any society on earth today, what one number would be most congruent with your preference? Average life expectancy captures very well which societies are good to be born at.
I am also aware of a ton of problems with this, since one has to be careful to consider humans vs human/cyborg hybrids, time spent in cryo-sleep or normal sleep vs experiential mind-moments. However, i’d rather have an approximate starting point for direct specification, rather than give up on the approach all-together.
Regarding 5: There is an interesting “problem” with “do what i would want if i had more time to think” that happens not in the case of failure, but in the case of success. Let’s say we have our happy go lucky life expectancy maximizing death-defeating FAI. It starts to look at society and sees that some widely accepted acts are totally horrifying from its perspective. It’s “morality” surpasses ours, which is just an obvious consequence of it’s intelligence surpassing ours.
Something like the amount of time we make children sit at their desks at school destroys their health to the point of disallowing immortality. This particular example might not be so hard to convince people of, but there could be others. At this point, they would go against a large number of people, to try and create its own schools which teach how bad the other schools are (or something). The governments don’t like this and shut it down because we still can for some reason.
Basically the issue is: this AI behaving in a friendly manner, which we would understand if we had enough time and intelligence. But we don’t. So we don’t have enough intelligence to determine if it is actually friendly or not.
Regarding 6: I feel that you haven’t even begun to approach the problem of a sub-group of people controlling the AI. The issue gets into the question of peaceful transitions of that power over the long term. There is also an issue of if you come up with a scheme of who gets to call the shots around the AI that’s actually a good idea, convincing people that it is a good idea instead of the default “let the government do it” is in itself a problem. It’s similar in principle to 5.
Note: I may be over my head here in math logic world:
For procrastination paradox:
There seems to be a desire to formalize
T proves G ⇒ G, which messes with completeness. Why not straight up try to formalize:
T proves G at time t ⇒ T proves G at time t+1 for all t > 0
G ⇒ button gets pressed at time some time X and wasn’t pressed at X-1
However, If T proves G at X-1, it must also prove G at X, for all X > 1 therefore it won’t press the button, unless X = 1.
Basically instead of reasoning of whether proving something makes it true, reason whether proving something at some point leads to re-proving it again at another point or just formalizing the very intuition that makes us understand the paradox.