The idea that most people who can’t do technical AI alignment are therefore able to do effective work in public policy or motivating public change seems unsupported by anything you’ve said. And a key problem with “raising awareness” as a method of risk reduction is that it’s rife with infohazard concerns. For example, if we’re really worried about a country seizing a decisive strategic advantage via AGI, that indicates that countries should be much more motivated to pursue AGI.
And I don’t think that within the realm of international agreements and pursuit of AI regulation, postponement is neglected, at least relative to tractability, and policy for AI regulation is certainly an area of active research.
Davidmanheim
“Will > 50% of AGI researchers agree with safety concerns by 2030?”
From my research, I think they mostly already do, they just use different framings, and care about different time frames.
Strong +1 to this suggestion, at least as an option that people can set.
I don’t think this type of comment is appropriate or needed. (It was funny, but still not a good thing to post.)
On the first argument, I replied that I think a non-AGI safety group could do this, and therefore not hurt the principally unrelated AGI safety efforts. Such a group could even call for reduction of existential risk in general, further decoupling the two efforts.
It sounds like you are suggesting that someone somewhere should do this. Who, and how? Because until there is a specific idea being put forward, I can say that pausing AGI would be good, since misaligned AGI would be bad. I don’t know how you’d do it, but if choosing between two world, the one without misaligned AGI seems likely to be better.
But in my mind, the proposal falls apart as soon as you ask who this group is, and whether this hypothetical groups has any leverage or any arguments that would convince people who are not already convinced. If the answer is yes, why do we need this new group to do this, and would we be better off using this leverage to increase the resources and effort put into AI safety?
This is tractable in sports because there are millions of dollars on the line for each player. In most contexts, the costs of negotiation and running a market for talent doesn’t work as well, and it’s better to use simple metrics despite all the very important problems with poorly aligned metrics. (Of course, the better solution is to design better metrics; https://mpra.ub.uni-muenchen.de/98288/ )
This is great, and it deals with a few points I didn’t, but here’s my tweetstorm from the beginning of last year about the distortion of scoring rules alone:
https://twitter.com/davidmanheim/status/1080458380806893568
If you’re interested in probability scoring rules, here’s a somewhat technical and nit-picking tweetstorm about why proper scoring for predictions and supposedly “incentive compatible” scoring systems often aren’t actually a good idea.
First, some background. Scoring rules are how we “score” predictions—decide how good they are. Proper scoring rules are ones where a predictor’s score is maximized when it give it’s true best guess. Wikipedia explains; en.wikipedia.org/wiki/Scoring_r…
A typical improper scoring rule is the “better side of even” rule, where every time your highest probability is assigned to the actual outcome, you get credit. In that case, people have no reason to report probabilities correctly—just pick a most likely outcome and say 100%.
There are many proper scoring rules. Examples include logarithmic scoring, where your score is the log of the probability assigned to the correct answer, and Brier score, which is the mean squared error. de Finetti et al. lays out the details here; link.springer.com/chapter/10.100…
These scoring rules are all fine as long as people’s ONLY incentive is to get a good score.
In fact, in situations where we use quantitative rules, this is rarely the case. Simple scoring rules don’t account for this problem. So what kind of misaligned incentives exist?
Bad places to use proper scoring rules #1 - In many forecasting applications, like tournaments, there is a prestige factor in doing well without a corresponding penalty for doing badly. In that case, proper scoring rules incentivise “risk taking” in predictions, not honesty.
Bad places to use proper scoring rules #2 - In machine learning, scoring rules are used for training models that make probabilistic predictions. If predictions are then used to make decisions that have asymmetric payoffs for different types of mistakes., it’s misaligned.
Bad places to use proper scoring rules #3 - Any time you want the forecasters to have the option to say answer unknown. If this is important—and it usually is—proper scoring rules can disincentify or overincentify not guessing, depending on how that option is treated.
Using a metric that isn’t aligned with incentives is bad. (If you want to hear more, follow me. I can’t shut up about it.)
Carvalho discusses how proper scoring is misused; https://viterbi-web.usc.edu/~shaddin/cs699fa17/docs/Carvalho16.pdf
Anyways, this paper shows a bit of how to do better; https://pubsonline.informs.org/doi/abs/10.1287/deca.1110.0216
Fin.
That all makes sense—I’m less certain that there is a reachable global maximum that is a Pareto improvement in terms of inputs over the current system. That is, I expect any improvement to require more of some critical resource—human time, capital investment, or land.
No, the claim as written is true—agriculture will ruin soil over time, which has happened in recent scientific memory in certain places in Africa. And if you look at the biblical description of parts of the middle east, it’s clear that desertification had taken a tremendous toll over the past couple thousand years. That’s not because of fertilizer usage, it’s because agriculture is about extracting food and moving it elsewhere, usually interrupting the cycle of nutrients, which happens organically otherwise. Obviously, natural habitats don’t do this in the same way, because the varieties of plants shift over time, fauna is involved, etc.
Yes, in the modern world, where babies are seen as precious, that is true. It clearly wasn’t as big a deal when infant mortality was very high.
This is disingenuous, I think. Of course they don’t exist at the necessary scale yet, because the market is small. If the market grew, and was profitable, scaling would be possible. Rare earths aren’t rare enough to be a real constraint, we’d just need to mine more of them. The only thing needed would be to make more of things we know how to make. (And no, that wouldn’t happen, because the new tech being developed would get developed far faster, and used instead.)
This isn’t critiquing the claim, though. Yes, there are alternatives that are available, but those alternatives—multi-cropping, integrating livestock, etc. are more labor intensive, and will produce less over the short term. And I’m very skeptical that the maximum is only local—have you found evidence that you can use land more efficiently, while keeping labor minimal, and produce more? Because if you did, there’s a multi-billion dollar market for doing that. Does that make the alternatives useless, or bad ideas? Not at all, and I agree that changes are likely necessary for long-term stability—unless other technological advances obviate the need for them. But we can’t pretend that staying at the maximum isn’t effectively necessary.
Agreed—and this reminds me of the observation that all of physics is contained in a single pebble; with enough undesrstnding, you could infer all of physics from close observation of quantum effects, find gravity at a very small scale if you had sensitive enough instruments, know much of natural history, liked the fact that earth has running water that made the stone smooth, that it must be in a universe more than a certain age given its composition, etc. With enough detail, any facet of a story requires effectively unlimited detail to fully understand.
And that makes it clear that we don’t intend for every translation to be of unlimited depth—but the depth of the translation matters, and we trade off between depth of translation and accuracy. Translating Sherbert Lemon as Lemon Sorbet is probably a lack of understanding and an overly direct literal-but-incorrect meaning, while translating it as Crembo might be a reasonable choice because of the context, but is not at all a literal translation.
As the post notes, inferential distance relates to differing worldviews and life experiences. This was written to an audience that mostly understands what inferential distance has to do with different worldviews—how would you explain it to a different audience?
Well, a typical translation doesn’t try to bridge the gap between languages, it just picks something on the far side of the gap that seems similar to the one on the near side. But that leaves something out.An example of this is in translations of Harry Potter, where Dumbledore’s password is translated into a local sweet. The UK versions has “Sherbet Lemon” while the US version has “Lemon drop.” Are these the same? I assumed so, but actually it seems the UK version has a “fizzy sweet powder” on the inside. In Danish and Swedish, it’s (mis?) translated as lemon ice cream—which isn’t the same at all. And in Hebrew, it’s translated as Krembo, which doesn’t even get close to translating the meaning correctly—it’s supposed to be an “equivalent children’s dessert”—but the translation simply doesn’t work, because you can’t carry a Krembo around in your pocket, since it would melt. Does this matter? Well, the difference between a kindly old wizard who carries around a sucking candy, and one who carries around a kind-of-big marshmallow dessert. But that’s beside the point—you don’t translate the life experience that growing up eating sherbert lemons gives you, you find an analogue.
The only way to translate a specific word or term accurately could be to provide so much background that the original point is buried, and the only way to translate an idea is to find an analogue that the reader already understands. And that’s why translation is impossible—but we do it anyways, and just accept that the results are fuzzy equivalents, and accept that worldviews are different enough that bridging the gap is impossible.
Tier 3, I think: Hoplite, on Android.
The free game is basically a roguelike, but it’s full information on each level, with only a little bit of strategy for which abilities to pick, and the Challenge mode available in the paid version, for $3, has a lot more straight puzzles.
To what extent are these dynamics the inevitable result of large organizations?
I want to note that I’ve previously argued that much of the dynamics are significantly forced by structure—but not in this context, and I’m thinking about how much or little of that argument applies here. (I’ll need to see what yo say in later posts in the series.)
I think there needs to be individual decisionmaking (on the part of both organizations and individual researchers, especially in light of the unilateralists’ curse,) alongside a much broader discussion about how the world should handle unsafe machine learning, and more advanced AI.
I very much don’t think that the AI safety community debating and coming up with shared, semi-public guidelines for, essentially, what to withhold from the broader public, done without input from the wider ML / AI and research community who are impacted and whose work is a big part of what we are discussing, would be wise. That community needs to be engaged in any such discussions.
There’s some intermediate options available instead of just “full secret” or “full publish”… and I haven’t seen anyone mention that...
OpenAI’s phased release of GPT2 seems like a clear example of exactly this. And there is a forthcoming paper looking at the internal deliberations around this from Toby Shevlane, in addition to his extant work on the question of how disclosure potentially affects misuse.
The first thing I would note is that stakeholders need to be involved in making any guidelines, and that pushing for guidelines from the outside is unhelpful, if not harmful, since it pushes participants to be defensive about their work. There are also an extensive literature discussing the general issue of information dissemination hazards and the issues of regulation in other domains, such as nuclear weapons technology, biological and chemical weapons, and similar.
There is also a fair amount of ongoing work on synthesizing this literature and the implications for AI. Some of it is even on this site. For example, see: https://www.lesswrong.com/posts/RY9XYoqPeMc8W8zbH/mapping-downside-risks-and-information-hazards and https://www.lesswrong.com/posts/6ur8vDX6ApAXrRN3t/information-hazards-why-you-should-care-and-what-you-can-do
So there is tons of discussion about this already, and there is plenty you should read on the topic—I suspect you can start with the paper that provided the name for your post, and continuing with sections of GovAI’s research agenda.
One key factor in metrics is how the number relates to the meaning. We’d prefer metrics that have scales which are meaningful to the users, not arbitrary. I really liked one example I saw recently.
In discussing this point in a paper entitled “Arbitrary metrics in psychology,” Blanton and Jaccard (doi:10.1037/0003-066X.61.1.27) fist point out that likert scales are not so useful. They then discuss the the (in)famous IAT test, where the scale is a direct measurement of the quantity of interest, but note that: “The metric of milliseconds, however, is arbitrary when it is used to measure the magnitude of an attitudinal preference.” Therefore, when thinking about degree of racial bias, “researchers and practitioners should refrain from making such diagnoses until the metric of the IAT can be made less arbitrary and until a compelling empirical case can be made for the diagnostic criteria used.” They go on to discuss norming measures, and looking at variance—but the base measure being used in not meaningful, so any transformation is of dubious value.
Going beyond that paper, looking at the broader literature on biases, we can come up with harder to measure but more meaningful measures of bias. Using probability of hiring someone based on racially-coded names might be a more meaningful indicator—but probability is also not a clear indicator, and use of names as a proxy obscures some key details about whether the measurement is class-based versus racial. It’s also not as clear how big of an effect a difference in probability makes, despite being directly meaningful.
A very directly meaningful measure of bias that is even easier to interpret is dollars. This is immediately meaningful; if a person pays a different amount for identical service, that is a meaningful indicator of not only the existence, but the magnitude of a bias. Of course, evidence of pay differentials is a very indirect and complex question, but there are better ways of getting the same information in less problematic contexts. Evidence can still be direct, such as how much someone bids for watches, where pictures were taken with the watch on a black or white person’s wrist, are a much more direct and useful way to understand how much bias is being displayed.