The organization reported $118,803.00 in theft in 2009 resulting in a year end asset balance lower than expected. The SIAI is currently pursuing legal restitution.
It isn’t much harder to steal code than to steal money from a bank account. Given the nature of research being conducted by the SIAI, one of the first and most important steps would have to be to think about adequate security measures.
If you are a potential donor interested to mitigate risks from AI then before contributing money you will have to make sure that your contribution does not increase those risks even further.
If you believe that risks from AI are to be taken seriously then you should demand that any organisation that studies artificial general intelligence has to establish significant measures against third-party intrusion and industrial espionage that is at least on par with the biosafety level 4 required for work with dangerous and exotic agents.
It might be the case that the SIAI does already employ various measures against the possibility of theft of sensitive information, yet any evidence that hints at the possibility of weak security should be taken seriously. Especially the possibility that there are potentially untrustworthy people who can access critical material should be examined.
Upvoted for raising some important points. Ceteris paribus, one failure of internal controls is nontrivial evidence of future ones.
For these purposes one should distinguish between sections of the organization. Eliezer Yudkowsky and Marcello Herreshoff’s AI work is a separate ‘box’ from other SIAI activities such as the Summit, Visiting Fellows program, etc. Eliezer is far more often said to be too cautious and secretive with respect to that than the other way around.
I think this is a legitimate concern. It’s probably not a significant issue right now, but definitely would be one if SIAI started making dramatic progress towards AGI. I don’t think it deserves the downvotes its getting.
It should be possible for the SIAI to build security measures while also providing some transparency into the nature of that security in a way that doesn’t also compromise it. I would bet that Eliezer has thought about this, or at least thought about the fact that he needs to think about it in more detail. This would be something to look into in a deeper examination of SIAI plans.
I am more concerned about the possibility that random employees at Google will succeed in making an AGI then I am at SIAI constructing one. To begin with, even if there were only 1000 employees at Google that were interested in AGI and they were only interested in it enough to work 1 hour a month each and they were only 80% as effective as Eliezer (as being some of the smartest people in the world doesn’t quite put them on the same level as Eli) then if Eliezer will have AGI in say, 2031 then Google will have it in about 2017.
Personally, I expect even moderately complicated problems—especially novel ones—to not scale or decompose at all cleanly.
So, leaving aside all questions about who is smarter than whom, I don’t expect a thousand smart people working an hour a month on a project to be nearly as productive as one smart person working eight hours a day.
If you could share your reasons for expecting otherwise, I might find them enlightening.
The idea is that they are sharing their information and findings so that while they are less efficient then working constantly on the problem they are able to point out possible solutions to each other that one person working by himself would be less likely to notice except through a longer process. As there would be between 4-5 people working on the project at any one time during the month I assume they would be working in a group and would stagger the times such that a nearly continuous effort is produced. Also, as much of the problem involves thinking about things, by not focusing on the issue constantly they may be more likely to come up with a solution then if they are focusing on it constantly.
This is a hypothetical, I have no idea how many people at Google are interested in AI or how much time they spend on it. I would imagine that there most likely are quite a few people at Google working on AGI as it relates directly to Google’s core business and that they work on it significantly more than one hour a month each.
(edit the comment with intelligence and Eli was a pun.)
I am at least 95% confident that the procedures that govern access to money in SIAI are both different from and far less strict than the procedures that would be used to govern access to code. Even the woefully obsolete “So You Want To Be A Friendly AI Programmer” document contains procedures that are more strict than this for dealing with actual AI code, and my suspicion is that these are likely to have updated only in the direction of more strictness since.
Disclaimer: I am neither an affiliate of nor a donor to the SIAI.
It isn’t much harder to steal code than to steal money from a bank account. Given the nature of research being conducted by the SIAI, one of the first and most important steps would have to be to think about adequate security measures.
This would certainly be a critical consideration if or when the SIAI was actually doing work directly on AGI construction. I don’t believe that is a focus in the near future. There is too much to be done before that becomes a possibility.
(Not that establishing security anyway is a bad thing.)
If you believe that risks from AI are to be taken seriously then you should demand that any organisation that studies artificial general intelligence has to establish significant measures against third-party intrusion and industrial espionage that is at least on par with the biosafety level 4 required for work with dangerous and exotic agents.
What if you believe in openness and transparency—and feel that elaborate attempts to maintain secrecy will cause your partners to believe you are hiding motives or knowledge from them—thereby tarnishing your reputation—and making them trust you less by making yourself appear selfish and unwilling to share?
Surely, then the strategies you refer to could easily be highly counter-productive.
Basically, if you misguidedly impose secrecy on the organisations involved then the good guys have fewer means of honestly signalling their altruism towards each other—and cooperating with each other—which means that their progress is slower and their relative advantage is diminished. That is surely bad news for overall risk.
The “opposite” strategy is much better, IMO. Don’t cooperate with secretive non-sharers. They are probably selfish bad guys. Sharing now is the best way to honestly signal that you will share again in the future.
It isn’t much harder to steal code than to steal money from a bank account. Given the nature of research being conducted by the SIAI, one of the first and most important steps would have to be to think about adequate security measures.
If you are a potential donor interested to mitigate risks from AI then before contributing money you will have to make sure that your contribution does not increase those risks even further.
If you believe that risks from AI are to be taken seriously then you should demand that any organisation that studies artificial general intelligence has to establish significant measures against third-party intrusion and industrial espionage that is at least on par with the biosafety level 4 required for work with dangerous and exotic agents.
It might be the case that the SIAI does already employ various measures against the possibility of theft of sensitive information, yet any evidence that hints at the possibility of weak security should be taken seriously. Especially the possibility that there are potentially untrustworthy people who can access critical material should be examined.
Upvoted for raising some important points. Ceteris paribus, one failure of internal controls is nontrivial evidence of future ones.
For these purposes one should distinguish between sections of the organization. Eliezer Yudkowsky and Marcello Herreshoff’s AI work is a separate ‘box’ from other SIAI activities such as the Summit, Visiting Fellows program, etc. Eliezer is far more often said to be too cautious and secretive with respect to that than the other way around.
I think this is a legitimate concern. It’s probably not a significant issue right now, but definitely would be one if SIAI started making dramatic progress towards AGI. I don’t think it deserves the downvotes its getting.
Note: the comment has been completely rewritten since the original wave of downvoting. It’s much better now.
I agree, this doesn’t deserve to be downvoted.
It should be possible for the SIAI to build security measures while also providing some transparency into the nature of that security in a way that doesn’t also compromise it. I would bet that Eliezer has thought about this, or at least thought about the fact that he needs to think about it in more detail. This would be something to look into in a deeper examination of SIAI plans.
I am more concerned about the possibility that random employees at Google will succeed in making an AGI then I am at SIAI constructing one. To begin with, even if there were only 1000 employees at Google that were interested in AGI and they were only interested in it enough to work 1 hour a month each and they were only 80% as effective as Eliezer (as being some of the smartest people in the world doesn’t quite put them on the same level as Eli) then if Eliezer will have AGI in say, 2031 then Google will have it in about 2017.
Personally, I expect even moderately complicated problems—especially novel ones—to not scale or decompose at all cleanly.
So, leaving aside all questions about who is smarter than whom, I don’t expect a thousand smart people working an hour a month on a project to be nearly as productive as one smart person working eight hours a day.
If you could share your reasons for expecting otherwise, I might find them enlightening.
The idea is that they are sharing their information and findings so that while they are less efficient then working constantly on the problem they are able to point out possible solutions to each other that one person working by himself would be less likely to notice except through a longer process. As there would be between 4-5 people working on the project at any one time during the month I assume they would be working in a group and would stagger the times such that a nearly continuous effort is produced. Also, as much of the problem involves thinking about things, by not focusing on the issue constantly they may be more likely to come up with a solution then if they are focusing on it constantly.
This is a hypothetical, I have no idea how many people at Google are interested in AI or how much time they spend on it. I would imagine that there most likely are quite a few people at Google working on AGI as it relates directly to Google’s core business and that they work on it significantly more than one hour a month each.
(edit the comment with intelligence and Eli was a pun.)
I don’t get it. I can haz Xplanation?
The word Eli can also be used for god, hence the pun.
Oh :-)
You post employs good parallelism of form, yet poor parallelism of substance.
EDIT: no longer relevant, but kept for context
I added to the comment and expanded on what I thought would be an obvious inference.
I didn’t find your inference too oblique, I found it too inaccurate.
I am at least 95% confident that the procedures that govern access to money in SIAI are both different from and far less strict than the procedures that would be used to govern access to code. Even the woefully obsolete “So You Want To Be A Friendly AI Programmer” document contains procedures that are more strict than this for dealing with actual AI code, and my suspicion is that these are likely to have updated only in the direction of more strictness since.
Disclaimer: I am neither an affiliate of nor a donor to the SIAI.
This would certainly be a critical consideration if or when the SIAI was actually doing work directly on AGI construction. I don’t believe that is a focus in the near future. There is too much to be done before that becomes a possibility.
(Not that establishing security anyway is a bad thing.)
What if you believe in openness and transparency—and feel that elaborate attempts to maintain secrecy will cause your partners to believe you are hiding motives or knowledge from them—thereby tarnishing your reputation—and making them trust you less by making yourself appear selfish and unwilling to share?
Surely, then the strategies you refer to could easily be highly counter-productive.
Basically, if you misguidedly impose secrecy on the organisations involved then the good guys have fewer means of honestly signalling their altruism towards each other—and cooperating with each other—which means that their progress is slower and their relative advantage is diminished. That is surely bad news for overall risk.
The “opposite” strategy is much better, IMO. Don’t cooperate with secretive non-sharers. They are probably selfish bad guys. Sharing now is the best way to honestly signal that you will share again in the future.