Why maintain any secrecy for SI’s research? Don’t we want others to collaborate on and use safety mechanisms? Of course, a safe AGI must be safe from the ground up. But as to implementation, why should we expect that SI’s AGI design could possibly have an lead on the others?
The question of whether to keep research secret must be made on a case-by-case basis. In fact, next week I have a meeting (with Eliezer and a few others) about whether to publish a particular piece of research progress.
Certainly, there are many questions that can be discussed in public because they are low-risk (in an information hazard sense), and we plan to discuss those in public — e.g. Eliezer is right now working on the posts in his Open Problems in Friendly AI sequence.
Why should we expect that SI’s AGI design will have a lead on others? We shouldn’t. It probably won’t. We can try, though. And we can also try to influence the top AGI people (10-40 years from now) to think with us about FAI and safety mechanisms and so on. We do some of that now, though the people in AGI today probably aren’t the people who will end up building the first AGIs. (Eliezer’s opinion may differ.)
Given that proofs can be wrong and that implementations can have their mistakes, and that we can’t predict the challenges ahead with certainty, what is SI’ s layered safety strategy (granted that FAI theory is the most important component)?
That will become clearer as we learn more. I do think several layers of safety will need to be involved. 100% proofs of Friendliness aren’t possible. There are both technical and social layers of safety strategy to implement.
How do we deal with the fact that unsafe AGI projects, without the constraint of safety, will very likely have the lead on SI’s project?
As I said above, one strategy is to build strong relationships with top AGI people and work with them on Friendliness research and make it available to them, while also being wary of information hazards.
Should we [spread] safety ideas to other AGI projects?
Eliezer may disagree, but I think the answer is “Yes.” There’s a great deal of truth in Upton Sinclair’s quip that “It is difficult to get a man to understand something, when his salary depends upon his not understanding it,” but I don’t think it’s impossible to reach people, especially if we have stronger arguments, more research progress on Friendliness, and a clearer impending risk from AI than is the case in early 2013.
That said, safety outreach may not be a very good investment now — it may be putting the cart before the horse. We probably need clearer and better-formed arguments, and more obvious progress on Friendliness, before safety outreach will be effective on even 10% of the most intelligent AI researchers.
The question of whether to keep research secret must be made on a case-by-case basis. In fact, next week I have a meeting (with Eliezer and a few others) about whether to publish a particular piece of research progress.
Certainly, there are many questions that can be discussed in public because they are low-risk (in an information hazard sense), and we plan to discuss those in public — e.g. Eliezer is right now working on the posts in his Open Problems in Friendly AI sequence.
Why should we expect that SI’s AGI design will have a lead on others? We shouldn’t. It probably won’t. We can try, though. And we can also try to influence the top AGI people (10-40 years from now) to think with us about FAI and safety mechanisms and so on. We do some of that now, though the people in AGI today probably aren’t the people who will end up building the first AGIs. (Eliezer’s opinion may differ.)
That will become clearer as we learn more. I do think several layers of safety will need to be involved. 100% proofs of Friendliness aren’t possible. There are both technical and social layers of safety strategy to implement.
As I said above, one strategy is to build strong relationships with top AGI people and work with them on Friendliness research and make it available to them, while also being wary of information hazards.
Eliezer may disagree, but I think the answer is “Yes.” There’s a great deal of truth in Upton Sinclair’s quip that “It is difficult to get a man to understand something, when his salary depends upon his not understanding it,” but I don’t think it’s impossible to reach people, especially if we have stronger arguments, more research progress on Friendliness, and a clearer impending risk from AI than is the case in early 2013.
That said, safety outreach may not be a very good investment now — it may be putting the cart before the horse. We probably need clearer and better-formed arguments, and more obvious progress on Friendliness, before safety outreach will be effective on even 10% of the most intelligent AI researchers.
Thanks, that makes things much clearer.