Thanks for writing this. I have been fretting for some time and realized that what I needed was a rational take on the war. I appreciate the time you’ve taken you write this out and I’ll check out your other posts on this.
sbenthall
Karma: 365
This seems correct to me. Thank you.
You don’t know anything about how cars work?
It’s possible to predict the behavior of black boxes without knowing anything about their internal structure.
Elaborate?
That says a lot more about your personal values then the general human condition.
I suppose you are right.
The models of worms might be a bit better at predicting worm behavior but they are not perfect.
They are significantly closer to being perfect than our models of humans. I think you are right in pointing out that where you draw the line is somewhat arbitrary. But the point is the variation on the continuum.
Do you think it is something external to the birds that make them migrate?
Norbert Wiener is where it all starts. This book has a lot of essays. It’s interesting—he’s talking about learning machines before “machine learning” was a household word, but envisioning it as electrical circuits.
http://www.amazon.com/Cybernetics-Second-Edition-Control-Communication/dp/026273009X
I think that it’s important to look inside the boxes. We know a lot about the mathematical limits of boxes which could help us understand whether and how they might go foom.
Thank you for introducing me to that Concrete Mathematics book. That looks cool.
I would be really interested to see how you model this problem. I’m afraid that op-amps are not something I’m familiar with but it sounds like you are onto something.
Do you think that rationalism is becoming a religion, or should become one?
Thanks. That criticism makes sense to me. You put the point very concretely.
What do you think of the use of optimization power in arguments about takeoff speed and x-risk?
Or do you have a different research agenda altogether?
That makes sense. I’m surprised that I haven’t found any explicit reference to that in the literature I’ve been looking at. Is that because it is considered to be implicitly understood?
One way to talk about optimization power, maybe, would be to consider a spectrum between unbounded, LaPlacean rationality and the dumbest things around. There seems to be a move away from this though, because it’s too tied to notions of intelligence and doesn’t look enough at outcomes?
It’s this move that I find confusing.
There are people in my department who do work in this area. I can reach out and ask them.
I think Mechanical Turk gets used a lot for survey experiments because it has a built-in compensation mechanism and there are ways to ask questions in ways that filter people into precisely what you want.
I wouldn’t dismiss Facebook ads so quickly. I bet there is a way to target mobile app developers on that.
My hunch is that like survey questions, sampling methods are going to need to be tuned case-by-case and patterns extracted inductively from that. Good social scientific experiment design is very hard. Standardizing it is a noble but difficult task.
Thanks. That’s very helpful.
I’ve been thinking about Stuart Russell lately, which reminds me...bounded rationality. Isn’t there a bunch of literature on that?
http://en.wikipedia.org/wiki/Bounded_rationality
Have you ever looked into any connections there? Any luck with that?
1) This is an interesting approach. It looks very similar to the approach taken by the mid-20th century cybernetics movement—namely, modeling social and cognitive feedback processes with the metaphors of electrical engineering. Based on this response, you in particular might be interested in the history of that intellectual movement.
My problem with this approach is that it considers the optimization process as a black box. That seems particularly unhelpful when we are talking about the optimization process acting on itself as a cognitive process. It’s easy to imagine that such a thing could just turn itself into a superoptimizer, but that would not be taking into account what we know about computational complexity.
I think that it’s this kind of metaphor that is responsible for “foom” intuitions, but I think those are misplaced.
2) Partial differential equations assume continuous functions, no? But in computation, we are dealing almost always with discrete math. What do you think about using concepts from combinatorial optimization theory, since those are already designed to deal with things like optimization resources and optimization efficiency?
Could you please link to examples of the kind of marketing studies that you are talking about? I’d especially like to see examples of those that you consider good vs. those you consider bad.
I am confused. Shouldn’t the questions depend on the content of the study being performed? Which would depend (very specifically) on the users/clients? Or am I missing something?
I would worry about sampling bias due to selection based on, say, enjoying points.
The privacy issue here is interesting.
It makes sense to guarantee anonymity. Participants recruited personally by company founders may be otherwise unwilling to report honestly (for example). For health related studies, privacy is an issue for insurance reasons, etc.
However, for follow-up studies, it seems important to keep earlier records including personally identifiable information so as to prevent repeatedly sampling from the same population.
That would imply that your organization/system needs to have a data management system for securely storing the personal data while making it available in an anonymized form.
However, there are privacy risks associated with ‘anonymized’ data as well, since this data can sometimes be linked with other data sources to make inferences about participants. (For example, if participants provide a zip code and certain demographic information, that may be enough to narrow it down to a very few people.) You may want to consider differential privacy solutions or other kinds of data perturbation.
He then takes whatever steps we decide on to locate participants.
Even if the group assignments are random, the prior step of participant sampling could lead to distorted effects. For example, the participants could be just the friends of the person who created the study who are willing to shill for it.
The studies would be more robust if your organization took on the responsibility of sampling itself. There is non-trivial scientific literature on the benefits and problems of using, for example, Mechanical Turk and Facebook ads for this kind of work. There is extra value added for the user/client here, which is that the participant sampling becomes a form of advertising.
Thanks for your thoughtful response. I’m glad that I’ve been more comprehensible this time. Let me see if I can address the problems you raise:
1) Point taken that human freedom is important. In the background of my argument is a theory that human freedom has to do with the endogeneity of our own computational process. So, my intuitions about the role of efficiency and freedom are different from yours. One way of describing what I’m doing is trying to come up with a function that a supercontroller would use if it were to try to maximize human freedom. The idea is that choices humans make are some of the most computationally complex things they do, and so the representations created by choices are deeper than others. I realize now I haven’t said any of that explicitly let alone argued for it. Perhaps that’s something I should try to bring up in another post.
2) I also disagree with the morality of this outcome. But I suppose that would be taken as beside the point. Let me see if I understand the argument correctly: if the most ethical outcome is in fact something very simple or low-depth, then this supercontroller wouldn’t be able to hit that mark? I think this is a problem whenever morality (CEV, say) is a process that halts.
I wonder if there is a way to modify what I’ve proposed to select for moral processes as opposed to other generic computational processes.
3) A couple responses:
Oh, if you can just program in “keep humanity alive” then that’s pretty simple and maybe this whole derivation is unnecessary. But I’m concerned about the feasibility of formally specifying what is essential about humanity. VAuroch has commented that he thinks that coming up with the specification is the hard part. I’m trying to defer the problem to a simpler one of just describing everything we can think of that might be relevant. So, it’s meant to be an improvement over programming in “keep humanity alive” in terms of its feasibility, since it doesn’t require solving perhaps impossible problems of understanding human essence.
Is it the consensus of this community that finding an objective function in E is an easy problem? I got the sense from Bostrom’s book talk that existential catastrophe was on the table as a real possibility.
I encourage you to read the original Bennett paper if this interests you. I think your intuitions are on point and appreciate your feedback.
I see, that’s interesting. So you are saying that while the problem as scoped in §2 may take a function of arbitrary complexity, there is a constraint in the superintelligence problem I have missed, which is that the complexity of the objective function has certain computational limits.
I think this is only as extreme a problem as you say in a hard takeoff situation. In a slower takeoff situation, inaccuracies due to missing information could be corrected on-line as computational capacity grows. This is roughly business-as-usual for humanity—powerful entities direct the world according to their current best theories; these are sometimes corrected.
It’s interesting that you are arguing that if we knew what information to include in a full specification of humanity, we’d be making substantial progress towards the value problem. In §3.2 I argued that the value problem need only be solved with a subset of the full specification of humanity. The fullness of that specification was desirable just because it makes it less likely that we’ll be missing the parts that are important to value.
If, on the other hand, that you are right and the full specification of humanity is important to solving the value problem—something I’m secretly very sympathetic to—then
(a) we need a supercomputer capable of processing the full specification in order to solve the value problem, so unless there is an iterative solution here the problem is futile and we should just accept that The End Is Nigh, or else try, as I’ve done, to get something Close Enough and hope for slow takeoff, and
(b) the solution to the value problem is going to be somewhere done the computational path from h and is exactly the sort of thing that would be covered in the scope of g*.
It would be a very nice result, I think, if the indirect normativity problem or CEV or whatever could be expressed in terms of the the depth of computational paths from the present state of humanity for precisely this reason. I don’t think I’ve hit that yet exactly but it’s roughly what I’m going for. I think it may hinge on whether the solution to the value problem is something that involves a halting process, or whether really it’s just to ensure the continuation of human life (i.e. as a computational process). In the latter case, I think the solution is very close to what I’ve been proposing.
This point about Ukrainian neo-Nazis is very misunderstood by the West.
During the Maidan revolution in Ukraine in 2014, neo-Nazi groups occupied government buildings and brought about a transition of government.
Why are there neo-Nazis in Ukraine? Because during WWII, the Nazis and the USSR were fighting over Ukraine. Ukraine is today quite ethnically diverse, and some of the ‘western’ Ukrainians who were resentful of USSR rule and, later, Russian influence, have reclaimed nazi ideas as part of a far-right Ukrainian nationalism. Some of these nazi groups that were originally militias have been incorporated into the Ukrainian military.
This is all quite well documented:
https://en.wikipedia.org/wiki/2014_Euromaidan_regional_state_administration_occupations
https://jacobin.com/2022/02/maidan-protests-neo-nazis-russia-nato-crimea
One of the regiments most well known to have Nazi ties was defeated at the Siege of Mariupol
https://en.wikipedia.org/wiki/Azov_Regiment
Naturally, this history is downplayed in presentations of Ukrainian nationalism targeted at the West, and emphasized in Russia depictions of Ukraine.