Is this very different from founding a pharmaceutical company?
Larks
Karma: 4,341
Critch wrote a related paper:
Existing multi-objective reinforcement learning (MORL) algorithms do not account for objectives that arise from players with differing beliefs.Concretely, consider two players with different beliefs and utility functions who may cooperate to build a machine that takes actions on their behalf. A representation is needed for how much the machine’s policy will prioritize each player’s interests over time. Assuming the players have reached common knowledge of their situation, this paper derives a recursion that any Pareto optimal policy must satisfy. Two qualitative observations can be made from the recursion: the machine must (1) use each player’s own beliefs in evaluating how well an action will serve that player’s utility function, and (2) shift the relative priority it assigns to each player’s expected utilities over time, by a factor proportional to how well that player’s beliefs predict the machine’s inputs. Observation (2) represents a substantial divergence from naive linear utility aggregation (as in Harsanyi’s utilitarian theorem, and existing MORL algorithms), which is shown here to be inadequate for Pareto optimal sequential decision-making on behalf of players with different beliefs.
War only happens if two agents don’t have common knowledge about who would win (otherwise they’d agree to skip the costs of war).
They might also have poorly aligned incentives, like a war between two countries that allows both governments to gain power and prestige, at the cost of destruction that is borne by the ordinary people of both countries. But this sort of principle-agent problem also seems like something AIs should be better at dealing with.
In light of this:
Build over-communication into the process.
In particular, don’t let silence carry information. Silence can be interpreted a million different ways (Cramton 2001).
Thanks for writing this! I found it very interesting, and I like the style. I particularly hadn’t properly appreciated how semi-distributed was worth than either extreme. It’s disappointing to hear, but seemingly obvious in retrospect and good to know.
Thanks for sharing, seems like a reasonable take to me.
I definitely being near AI hubs is helpful, and I’d be interested in supporting any credible new groups that started in other hubs.
Thanks for that extra info on CHAI staff. In general my objections to the bay area are partly about the EA/LW culture there, and partly about the broader culture. I did end up donating to CHAI despite this!
Thanks! I have fixed most of the typos.
Especially as in recent years investor preferences for lower risk, low beta, low vol, higher yield stocks (like utilities and staples) has been well documented as strong.
One related fact is that (at least some of the banks) feel like they haven’t gotten credit for the risk reductions they have done. Given their lower leverage now, they feel like they should have higher multiples, whereas actually their relative multiples have compressed vs the market vs pre-crisis. Of course, this may be because their risk was under-estimated pre-crisis, so their relative multiple was too high.
At least in public equity markets the size premium is very weak. My team looked at it and decided not to use it as a factor to guide our investing. It’s closely correlated to an illiquidity premia, which I do believe in.
2018 AI Alignment Literature Review and Charity Comparison
Update: Jeff says they only ever sent him part of his data.
Nick’s space papers are largely about how to harvest large amounts of utility from the galaxy, not about how to increase humanity’s robustness. And yes, there are some Xrisks (including the one I am focused on) that space colonies do not help with, but the reader may not be convinced of these, so it is surely worth mentioning that some risks would be guarded against with interstellar diversification. If nothing else you should probably argue that space colonization is not an adequate solution for these reasons.
A study by the investment-research firm Strategas which was cited in The Economist and the Washington Post compared the 50 firms that spent the most on lobbying relative to their assets, and compared their financial performance against that of the S&P 500 in the stock market; the study concluded that spending on lobbying was a “spectacular investment” yielding “blistering” returns comparable to a high-flying hedge fund, even despite the financial downturn of the past few years.
I think I read this research while I was a Strategas client; if I’m remembering it correctly it was extremely poorly done. Short back test (just a few years), garden of forking paths, etc. Most sell-side research is not epistemically rigourous and Strategas is not one of the better firms. I would not put much weight on this research.
There is widespread agreement that a key ingredient in effective lobbying is money. This view is shared by players in the lobbying industry.
Well of course lobbyists would say they’re worth the money!
This, by the way, explains my intuitive dislike for some types of moral realism. If there are true objective moral facts that humans can access, then whatever process counts as “accessing them” becomes… a local stopping condition for defining value.
I’m not sure I understand what you’re getting at here. Yes, they are both local stopping conditions, but there seems to be a clear dis-analogy. The other local stopping conditions seem to be bad not because they are stopping conditions, but because most contemporary people don’t want to end up as Lotus-Eaters, or as mindless outsourcers. We would oppose such a development even if it wasn’t stable! For example, a future where we oscillate between lotus-eaters and mindless outsourcers seems about as bad as either individual scenario. So it’s not really the stability we object to.
But in that case, it’s not clear why we should be opposed to the moral realism. After all, many people would like to go there, even if we are presently a long way away.
the one academic doing good work in the area is Sheffer, who is running a longitudinal survey which may or may not have enough statistical power to rule out particularly dramatic variances in outcomes. (Sheffer mentions the selection bias problem but seems to have the attitude that it’s not a problem for her work.)
Was there any follow-up here?
The bioterminist’s guide is now 5 years old. Does anyone know of an updated version?
Thanks, I’m honoured! I’ve sent you a private message.
I think you have misunderstood the Wells Fargo case. These fake accounts generally didn’t bring in any material revenue; they were just about internal ‘number of new accounts targets’. It was directly a case of bank employees being incentivised to defraud management and investors, which they then did. If ordinary Wells employees had not behaved fraudulently, all the targets would have been missed, informing management/investors about their mis-calibration, and more appropriate targets would have been set. In this case power didn’t buy distance from the crime, but only in the sense that it meant you couldn’t tell you were being cheated.
For more on this I recommend the prolific Matt Levine: