Many smart people have voiced opinions against such blind following of metrics in government, but almost all organizations, once beyond the grip of the founders fall into some such pattern. Compassionate mystic movements become rigid churches. Political parties for eg. pay more attention to lip service to issues, pomp, show and mind killing than to actual issues. Companies seek to make money at the expense of creating actual value, forgetting that money is only a symbol of value.
It would be useful to see a link to a peer-reviewed study supporting each of these summaries. I would be surprised (and hence informed!) if they all turned out to be clearly true. My understanding of the sociology of religion is actually that over time religious movements become more liberal and cosmopolitan, in opposition to their early doctrines and the needs of their original members and later members similar to the founders, in the sociology of religion this is basically accepted as a driver behind the formation of schismatic religious movements and the theoretical arguments are generally about the causative factors behind the change in membership over time. Some argue that the increasing number of church members who want cosmopolitan doctrines is simply regression towards the mean (away from the atypical needs of early members) and others argue that the doctrines play a causal role in increasing the worldly success of church members. I suspect it is a little of both.
Also, I don’t think companies ever have the purpose of simply creating value “in general” so it can’t be a lost purpose. My understanding is that companies are created to solve the relatively well defined and more limited problem of doing something so unique and positive that they create scarce value whose combined scarcity and value are so dramatic that people are willing to fork money over for the products or services. There are lots of ways to create value, like providing free internet services or picking up garbage by the side of the road. No one does these sorts of things in private industry without some angle. The trick is for the company to do something that is clearly valuable and also that compels people to pay them for it based on their ability to deprive users of the benefits of their work unless users pay them. This sometimes “feels immoral” in small groups of people, but is cybernetically necessary in large anonymous groups where services have high marginal costs, which is one of many reasons that coordination is hard.
The political economy of democracies is even more complicated, with (1) difficult to measure psychological benefits being allocated to radicalized volunteer activists, (2) selfishly rational and highly strategic lobbying efforts coming from all sides, (3) substantial voter apathy, and (4) a gaping free rider problem right at the center because people who don’t contribute to good government cannot be deprived of most of its benefits. I expect people who are experts in this domain to be doing a lot of complicated stuff that I don’t currently understand and so when you casually dismiss “pomp, show, and mind killing” in the outer forms of politics, and offer no supporting citation, I’m tempted to think that you’re expecting inferential distances to be small and making a rookie rationality error because all of politics isn’t “doing what you naively think it should do”.
The same paragraph continued...
A lot of people have bemoaned the brainpower that is moving into finance. And something even more repugnant, there is an entire economy thriving around the war on drugs, with everyone in on the cut.
I’ve heard people bemoaning that fact, so I agree with your front line evidence: yes people do bemoan this fact.
But this sort of begs the question about whether the bemoaners are actually right. I consider myself moderately smart, and I’ve thought about getting a formal degree in economics and going into finance both to learn more about adaptively self-regulating processes and because it seems like an area that needs to be “done right” if the rest of the world is to function well. Most of the reason I never followed through on these impulses is that I wouldn’t plan on staying in finance after I’d learned what I came to learn and the business culture of finance doesn’t seem particularly nurturing, especially for newbies. I can easily imagine people similar to me in some respects but different from me in other respects who go into finance for “basically good reasons” and thereby do enormous good.
Moreover, while it seems to be illiberal to punish people for putting whatever they want into their own bodies, the fact that so many people do engage in pharmaceutical wireheading seems kind of tragic in some respects. I could imagine reasonably good paternalistic arguments for making certain drugs illegal, and I would expect if the paternalistic argument carried the day, then it would be necessary to make the enforcement systems self sustaining if you actually wanted to decrease drug use in the long term. The only way it would be dramatically repugnant for the war on drugs to be self sustaining is if the war on drugs was itself totally repugnant so that its failure to sustain itself would be an obvious boon. Honest policy discussions should not be one-sided.
For what it’s worth, I basically agree with your bottom line conclusion that there is a real potential for a swiftly arising Unfriendly AGI.
I think there are reasonable defenses of the so-called Scary Idea (maybe the “Scary Hypothesis” would be a better name?), and I think that it improves the odds of the world having a positive outcome if “algorithm literate” net activists occasionally descend on AGI researchers and demand that they give an accounting of themselves in light of the Scary Hypothesis. The trick is that the truth value of the Scary Hypothesis probably changes based on the socio-technological context and the details of each specific project. In light of this, a generic policy of publicly sourced demand for a positive theory of particular safety from each AGI project (maybe every year or two?) seems to me like a prudent and reasonably cheap political safety measure :-)
It would be useful to see a link to a peer-reviewed study supporting each of these summaries. I would be surprised (and hence informed!) if they all turned out to be clearly true. My understanding of the sociology of religion is actually that over time religious movements become more liberal and cosmopolitan, in opposition to their early doctrines and the needs of their original members and later members similar to the founders, in the sociology of religion this is basically accepted as a driver behind the formation of schismatic religious movements and the theoretical arguments are generally about the causative factors behind the change in membership over time. Some argue that the increasing number of church members who want cosmopolitan doctrines is simply regression towards the mean (away from the atypical needs of early members) and others argue that the doctrines play a causal role in increasing the worldly success of church members. I suspect it is a little of both.
Also, I don’t think companies ever have the purpose of simply creating value “in general” so it can’t be a lost purpose. My understanding is that companies are created to solve the relatively well defined and more limited problem of doing something so unique and positive that they create scarce value whose combined scarcity and value are so dramatic that people are willing to fork money over for the products or services. There are lots of ways to create value, like providing free internet services or picking up garbage by the side of the road. No one does these sorts of things in private industry without some angle. The trick is for the company to do something that is clearly valuable and also that compels people to pay them for it based on their ability to deprive users of the benefits of their work unless users pay them. This sometimes “feels immoral” in small groups of people, but is cybernetically necessary in large anonymous groups where services have high marginal costs, which is one of many reasons that coordination is hard.
The political economy of democracies is even more complicated, with (1) difficult to measure psychological benefits being allocated to radicalized volunteer activists, (2) selfishly rational and highly strategic lobbying efforts coming from all sides, (3) substantial voter apathy, and (4) a gaping free rider problem right at the center because people who don’t contribute to good government cannot be deprived of most of its benefits. I expect people who are experts in this domain to be doing a lot of complicated stuff that I don’t currently understand and so when you casually dismiss “pomp, show, and mind killing” in the outer forms of politics, and offer no supporting citation, I’m tempted to think that you’re expecting inferential distances to be small and making a rookie rationality error because all of politics isn’t “doing what you naively think it should do”.
The same paragraph continued...
I’ve heard people bemoaning that fact, so I agree with your front line evidence: yes people do bemoan this fact.
But this sort of begs the question about whether the bemoaners are actually right. I consider myself moderately smart, and I’ve thought about getting a formal degree in economics and going into finance both to learn more about adaptively self-regulating processes and because it seems like an area that needs to be “done right” if the rest of the world is to function well. Most of the reason I never followed through on these impulses is that I wouldn’t plan on staying in finance after I’d learned what I came to learn and the business culture of finance doesn’t seem particularly nurturing, especially for newbies. I can easily imagine people similar to me in some respects but different from me in other respects who go into finance for “basically good reasons” and thereby do enormous good.
Moreover, while it seems to be illiberal to punish people for putting whatever they want into their own bodies, the fact that so many people do engage in pharmaceutical wireheading seems kind of tragic in some respects. I could imagine reasonably good paternalistic arguments for making certain drugs illegal, and I would expect if the paternalistic argument carried the day, then it would be necessary to make the enforcement systems self sustaining if you actually wanted to decrease drug use in the long term. The only way it would be dramatically repugnant for the war on drugs to be self sustaining is if the war on drugs was itself totally repugnant so that its failure to sustain itself would be an obvious boon. Honest policy discussions should not be one-sided.
For what it’s worth, I basically agree with your bottom line conclusion that there is a real potential for a swiftly arising Unfriendly AGI.
I think there are reasonable defenses of the so-called Scary Idea (maybe the “Scary Hypothesis” would be a better name?), and I think that it improves the odds of the world having a positive outcome if “algorithm literate” net activists occasionally descend on AGI researchers and demand that they give an accounting of themselves in light of the Scary Hypothesis. The trick is that the truth value of the Scary Hypothesis probably changes based on the socio-technological context and the details of each specific project. In light of this, a generic policy of publicly sourced demand for a positive theory of particular safety from each AGI project (maybe every year or two?) seems to me like a prudent and reasonably cheap political safety measure :-)