I find this confusing, since in my understanding and experience, many organizations undergo recursive self-improvement lots of the time.
Could you elaborate your thinking on this? Why is an organization’s intervention into, say, the organizational structure of its own management not effectively recursively self-improving on applied organization theory?
One could argue that the expansion of global capitalism constitutes a ‘corporate singularity’.
Luke, this is my first comment on LessWrong so forgive me if I’m missing some of the zeitgeist. But I was wondering if you could elaborate on a couple points:
You recommend replacing ethics with moral psychology and decision theory. Hearing that, I’m concerned that replacing ethics with moral psychology would be falling for a naive is/ought fallacy: just because most people’s psychological makeup makes them consider morality in a certain way does not make those moral intuitions correct. And replacing ethics with decision theory would be sidestepping the metaethical question about the legitimacy of consequentialism.
You’ve also left out any political theory from your syllabus. That is disappointing, since one of the roles that philosophy plays when performing at its best is uniting epistemology, ethics, and political philosophy. Plato and Kant, for example, were attempting to do that. How do you see your curriculum weighing in on questions like “What is justice?”
As for my background, I studied cognitive science as an undergraduate with a focus on complexity theory and artificial intelligence, but have also spent a lot of time reading and discussing other philosophy. While I think I understand the thrust of your argument (it’s one I would have made myself when I was an undergrad), I’ve been since convinced of the value of other schools of thought.
I’d argue that, say, continental philosophers are not as sloppy as computer scientists or analytically trained philosophers accuse them of. Rather they have a specialized vocabulary (just like other specialists) for some very difficult but powerful concepts. Often these concepts pertain to social and political life. These concepts aren’t easily reducible to a naturalized cognitivist wordview because they deal with transpersonal phenomena. That doesn’t mean they lack utility though.
oh, sorry I missed it. I’ve only started looking at LW recently. Does anyone have a link?
thanks. I’m new to this editor. will fix.
The privacy issue here is interesting.
It makes sense to guarantee anonymity. Participants recruited personally by company founders may be otherwise unwilling to report honestly (for example). For health related studies, privacy is an issue for insurance reasons, etc.
However, for follow-up studies, it seems important to keep earlier records including personally identifiable information so as to prevent repeatedly sampling from the same population.
That would imply that your organization/system needs to have a data management system for securely storing the personal data while making it available in an anonymized form.
However, there are privacy risks associated with ‘anonymized’ data as well, since this data can sometimes be linked with other data sources to make inferences about participants. (For example, if participants provide a zip code and certain demographic information, that may be enough to narrow it down to a very few people.) You may want to consider differential privacy solutions or other kinds of data perturbation.
fixed. much thanks.
There are people in my department who do work in this area. I can reach out and ask them.
I think Mechanical Turk gets used a lot for survey experiments because it has a built-in compensation mechanism and there are ways to ask questions in ways that filter people into precisely what you want.
I wouldn’t dismiss Facebook ads so quickly. I bet there is a way to target mobile app developers on that.
My hunch is that like survey questions, sampling methods are going to need to be tuned case-by-case and patterns extracted inductively from that. Good social scientific experiment design is very hard. Standardizing it is a noble but difficult task.
He then takes whatever steps we decide on to locate participants.
Even if the group assignments are random, the prior step of participant sampling could lead to distorted effects. For example, the participants could be just the friends of the person who created the study who are willing to shill for it.
The studies would be more robust if your organization took on the responsibility of sampling itself. There is non-trivial scientific literature on the benefits and problems of using, for example, Mechanical Turk and Facebook ads for this kind of work. There is extra value added for the user/client here, which is that the participant sampling becomes a form of advertising.
There are many reasons why the intelligence of AI+ greatly dwarfs that of human organizations; see Section 3.1 of the linked paper.
Since an organization’s optimization power includes optimization power gained from information technology, I think that the “AI Advantages” in section 3.1 mostly apply just as well to organizations. Do you see an exception?
This sounds similar to a position of Robin Hanson addressed in Footnote 25 of the linked paper.
Ah, thanks for that. I think I see your point: rogue AI could kill everybody, whereas a dominant organization would still preserve some people and so is less ‘interesting’.
Two responses:
First, a dominant organization seems like the perfect vehicle for a rogue AI, since it would already have all resources centralized and ready for AI hijacking. So, a study of the present dynamics between superintelligent organizations is important to the prediction of hard takeoff machine superintelligence.
Second, while I once again risk getting political at this point, I’d argue that an overriding concern for the total existence of humanity only makes sense if one doesn’t have any skin in the game of any of the other power dynamics going on. I believe there are ethical reasons for being concerned with some of these other games. That is well beyond the scope of this post.
The Singularity Institute is completely aware that there are other existential risks to humanity; its purpose is to deal with one of them.
That’s clear.
This sounds awfully suspicious. Are you sure you don’t have the bottom line precomputed?
Honestly, I don’t follow the line of reasoning in the post you’ve linked to. Could you summarize in your own terms?
My reason for not providing arguments up front is because I think excessive verbiage impairs readability. I would rather present justifications that are relevant to my interlocutor’s objections than try to predict everything up front. Indeed, I can’t predict all objections up front, since this audience has more information than I have available.
However, since I have faith that we are all in the same game of legitimate truth-seeking, I’m willing to pursue dialectical argumentation until it converges.
How long did it take you to come up with this line of reasoning?
I guess over 27 years. But I stand on the shoulders of giants.
I would worry about sampling bias due to selection based on, say, enjoying points.
They can’t use one improvement to fuel another, they would have to come up with the next one independently
I disagree.
Suppose an organization has developers who work in-house on their issue tracking system (there are several that do—mostly software companies).
An issue tracking system is essentially a way for an organization to manage information flow about bugs, features, and patches to its own software. The issue tracker (as a running application) coordinates between developers and the source code itself (sometimes, its own source code).
Taken as a whole, the developers, issue tracker implementation, and issue tracker source code are part of the distributed cognition of the organization.
I think that in this case, an organization’s self-improvement to the issue tracker source code recursively ‘fuels’ other improvements to the organization’s cognition.
The point isn’t that an AGI has or does not have certain skills. It’s that it has the ability to learn those skills. Deep Blue doesn’t have the capacity to learn anything other than playing chess, while humans, despite never running into a flute in the ancestral environment, can learn to play the flute.
Fair enough. But then we should hold organizations to the same standard. Suppose, for whatever reason, an organization needs better-than-median-human flute-playing for some purpose. What then?
Then they hire a skilled flute-player, right?
I think we may be arguing over an issue of semantics. I agree with you substantively that general intelligence is about adaptability, gaining and losing skills as needed.
My point in the OP was that organizations and the hypothetical AGI have comparable kinds of intelligence, so we can think about them as comparable superintelligences.
I find this difficult to follow. Is there a concrete mathematical definition of ‘recursion’ in this sense available anywhere?
I think there’s a lot to this line of thinking. It’s in fact the counterargument I find most threatening to my position.
But I think you are assuming an organization with a particularly autocratic leadership. In some organizations, leadership is broadly distributed.
For example, in many open source software development communities, decisions about how to change the source code are made by a consensus of their developers.
When these developers are using their own software in the process of developing and/or communicating (such as in the case of Git, or Mailman, or Emacs), then I think there’s a case for a genuine, distributed sense of organizational intelligence with recursive self-modification.
So there’s some big problems of picking the right audience here. I’ve tried to make some headway into the community complaining about newsfeed algorithm curation (which interests me a lot, but may be more “political” than would interest you) here:
https://github.com/sbenthall/tweetserve/blob/master/DesigningNetworkedPublicsforCommunicativeAction.docx
which is currently under review. It’s a lot softer that would be ideal, but since I’m trying to convince these people to go from “algorithms, how complicated! Must be evil” to “oh, they could be designed to be constructive”, it’s a first step. More or less it’s just opening up the idea that Twitter is an interesting testbed for ethically motivated algorithmic curation.
I’ve been concerned more generally with the problem of computational asymmetry in economic situations. I’ve written up something that’s an attempt at a modeling framework here. It’s been accepted only as a poster, because it’s results are very slim. It was like a quarter of a semester’s work. I’d be interested in following through on it.
http://arxiv.org/abs/1206.2878
The main problem I ran into was not knowing a good way to model relative computational capacity; the best tool I had was big-O and other basic computational theory stuff. I did a little sort of remote apprenticeship with David Wolpert as Los Alamos; he’s got some really interesting stuff on level-K reasoning and what he calls predictive game theory.
http://arxiv.org/abs/nlin/0512015
(That’s not his most recent version). It’s really great work, but hard math to tackle on ones own. In general my problem is there isn’t much of a community around this at Berkeley, as far as I can tell. Tell me if you know differently. There’s some demand from some of the policy people—the lawyers are quite open-minded and rigorous about this sort of thing. And there’s currently a ton of formal work on privacy, which is important but not quite as interesting to me personally.
My blog is a mess and doesn’t get into formal stuff at all, at least not recently.