Davidmanheim
Karma: 670 (LW), 7 (AF)
NewTop
Minor comment/correction—VoI isn’t necessarily linked to a single decision, but the way it is typically defined in introductory works, it implicit that it is limited to one decision. This is mostly because (as I found out when trying to build more generalized VoI models for my dissertation,) it’s usually quickly intractable for multiple decisions.
I agree, and think work in the area is valuable, but would still argue that unless we expect a correct and coherent answer, any single approach is going to be less effective than an average of (contradictory, somewhat unclear) different models.
As an analogue, I think that effort into improving individual prediction accuracy and calibration is valuable, but for most estimation questions, I’d bet on an average of 50 untrained idiots over any single superforecaster.
Having looked into this, it’s partly that, but mostly that tax codes are written in legalese. A simple options contract for a call, which can easily be described in 10 lines of code, or a one-line equation. But the legal terms are actually this 188 page pamplet; https://www.theocc.com/components/docs/riskstoc.pdf which is (technically but not enforced to be a) legally required reading for anyone who wants to purchase an exchange traded option. And don’t worry—it explicitly notes that it doesn’t cover the actual laws governing options, for which you need to read the relevant US code, or the way in which the markets for trading them work, or any of the risks.
re: #2, VoI doesn’t need to be constrained to be positive. If in expectation you think the information will have a net negative impact, you shouldn’t get the information.
re: #3, of course VoI is subjective. It MUST be, because value is subjective. Spending 5 minutes to learn about the contents of a box you can buy is obviously more valuable to you than to me. Similarly, if I like chocolate more than you, finding out if a cake has chocolate is more valuable for me than for you. The information is the same, the value differs.
This matters because if the Less Wrong view of the world is correct, it’s more likely that there are clean mathematical algorithms for thinking about and sharing truth that are value-neutral (or at least value-orthogonal, e.g. “aim to share facts that the student will think are maximally interesting or surprising”.
I don’t think this is correct—it misses the key map-territory distinction in the human mind. Even though there is “truth” in an objective sense, there is no necessity that the human mind can think about or share that truth. Obviously we can say that experientially we have something in our heads that correlates with reality, but that doesn’t imply that we can think about truth without implicating values. It also says nothing about whether we can discuss truth without manipulating the brain to represent things differently—and all imperfect approximations require trade-offs. If you want to train the brain to do X, you’re implicitly prioritizing some aspect of the brain’s approximation of reality over others.
Maybe I’m reading your post wrong, but it seems that you’re assuming that a coherent approach is needed in a way that could be counter-productive. I think that a model of an individual’s preferences is likely to be better represented by taking multiple approaches, where each fails differently. I’d think that a method that extends or uses revealed preferences would have advantages and disadvantages that none of, say, stated preferences, TD Learning, CEV, or indirect normativity share, and the same would be true for each of that list. I think that we want that type of robust multi-model approach as part of the way we mitigate over-optimization failures, and to limit our downside from model specification errors.
(I also think that we might be better off building AI to evaluate actions on the basis of some moral congress approach using differently elicited preferences across multiple groups, and where decisions need a super-majority of some sort as a hedge against over-optimization of an incompletely specified version of morality. But it may be over-restrictive, and not allow any actions—so it’s a weakly held theory, and I haven’t discussed it with anyone.)
Having tried to play with this, I’ll strongly agree that random functions on R^N aren’t a good place to start. But I’ve simulated random nodes in the middle of a causal DAG, or selecting ones for high correlation, and realized that they aren’t particularly useful either; people have some appreciation of causal structure, and they aren’t picking metrics randomly for high correlation—they are simply making mistakes in their causal reasoning, or missing potential ways that the metric can be intercepted. (But I was looking for specific things about how the failures manifested, and I was not thinking about gradient descent, so maybe I’m missing your point.)
“(Each the same size as the original.)”
I was not expecting to laugh reading this. Well done—I just wish I hadn’t been in the middle of drinking my coffee.
Good finds!
I think they are headed in the right direction, but I’m skeptical of the usefulness their work on complexity. The metrics ignore computational complexity of the model, and assume all the variance is modeled based on sources like historical data and expert opinion. It’s also not at all useful unless we can fully characterize the components of the system, which isn’t usually viable.
It also seems to ignore the (in my mind critical) difference between “we know this is evenly distributed in the range 0-1” and “we have no idea what the distribution of this is over the space 0-1.” But I may be asking for too much in a complexity metric.
I discuss a different reformulation in my new paper, “Systemic Fragility as a Vulnerable World” casting this as an explore/exploit tradeoff in a complex space. In the paper, I explicitly discuss the way in which certain subspaces can be safe or beneficial.
“The push to discover new technologies despite risk can be understood as an explore/exploit tradeoff in a potentially dangerous environment. At each stage, the explore action searches the landscape for new technologies, with some probability of a fatal result, and some probability of discovering a highly rewarding new option. The implicit goal in a broad sense is to find a search strategy that maximize humanity’s cosmic endowment—neither so risk-averse that advanced technologies are never explored or developed, nor so risk-accepting that Bostrom’s postulated Vulnerable World becomes inevitable. Either of these risks astronomical waste. However, until and unless the distribution of black balls in Bostrom’s technological urn is understood, we cannot specify an optimal strategy. The first critical question addressed by Bostrom - ``Is there a black ball in the urn of possible inventions?″ is, to reframe the question, about the existence of negative singularities in the fitness landscape.”
As an extension of Bostrom’s ideas, I have written a draft entitled ” Systemic Fragility as a Vulnerable World ” where I introduce the “Fragile World Hypothesis.”
Abstract:
The possibility of social and technological collapse has been the focus of science fiction tropes for decades, but more recent focus has been on specific sources of existential and global catastrophic risk. Because these scenarios are simple to understand and envision, they receive more attention than risks due to complex interplay of failures, or risks that cannot be clearly specified. In this paper, we discuss a new hypothesis that complexity of a certain type can itself function as a source of risk. This ”Fragile World Hypothesis” is compared to Bostroms ”Vulnerable World Hypothesis”, and the assumptions and potential mitigations are contrasted.
Yes.
But to clarify, I don’t think the Antonine plague is quite the same as modern ones, for the simple reason that it could only spread over a fairly limited geographic region, and it could not become endemic because of population density constraints. Smallpox evolution is driven by selection pressure in humans, and the “500 years old” claim is about that evolution, not about whether it affected humans at any time in the past. That said, it absolutely matters, because if the original source of smallpox was only 500 years ago, where did it come from?
The question is how smallpox evolved, and what variant was present prior to the 1500s. It’s plausible that Horsepox, which was probably the source for the vaccine strain, or Cowpox, spread via intermediate infections in cats, were the source—but these are phylogenetically distant enough that, from my limited understanding, it’s clearly implausible that it first infected humans and turned into modern smallpox at recently as the 1500s. (But perhaps this is exactly the claim of the paper. I’m unclear.) Instead, my understanding is that there must have been some other conduit, and it seems very likely that it’s related to a historically much earlier human pox virus—thousands of years, not hundreds.
I’m definitely not the best person to explain this, since I’m more on the epidemiology side. I understand the molecular clock analyses a bit, and they involve mutation rates plus tracking mutations in different variants, and figuring out how long it should take for the various samples collected at different times to have diverged, and what their common ancestors are.
Thank you! This is a point I keep trying to make, less eloquently, in both bioethics and in AI safety.
We need fewer talking heads making suggestions for how to regulate, and more input from actual experts, and more informed advice going to decision makers. If “professional ethicists” have any role, it should be elicitation, attempting to reconcile or delineate different opinions, and translation of ethical opinions of experts into norms and policies.
I have several short comments about part 3, short not because there is little to say, but because I want to make the points and do not have time to discuss them in depth right now.
1) If multi-agent systems are more likely to succeed in achieving GAI, we should shut up about why they are important. I’m concerned about unilateralist curse, and would ask that someone from MIRI weigh in on this.
2) I agree that multi-agent systems are critical, but for different (non-contradictory) reasons—I think multi-agent systems are likely to be less safe and harder to understand. See draft of my forthcoming article here: https://arxiv.org/abs/1810.10862
3) If this is deemed to be important, the technical research directions point to here are under-specified and too vague to be carried out. I think concretizing them would be useful. (I’d love to chat about this, as I have ideas in this vein. If you are interested in talking, feel free to be in touch—about.me/davidmanheim .)
There is genetic evidence discussed in Hopkins’ “Princes and Peasants: Smallpox in History,” which implies ancient existence of variola viruses, as you note from the Wiki article. The newer paper overstates the case in typical academic fashion in order to sound as noteworthy as possible. The issue with saying that earlier emergence is not the “current” disease of smallpox is that we expect significant evolution to occur once there is sufficient population density, and more once there is selection pressure due to vaccination, and so it is very unsurprising that there are more recent changes. (I discuss this in my most recent paper, https://www.liebertpub.com/doi/pdf/10.1089/hs.2018.0039 )
It’s very clear that a precursor disease existed in humans for quite a while. It’s also very clear that these outbreaks in thin populations would have continued spreading, so I’m unconvinced that the supposed evidence of lack due to Hippocrate’s omission, and the lack of discussion in the old and new testament is meaningful. And regarding the old testament, at least, the books aren’t great with describing “plagues” in detail, and there are plenty of times we hear about some unspecified type of plague or malady as divine punishment.
So the answer depends on definitions. It’s unclear that there is anything like a smallpox epidemic as the disease currently occurs in a population that is not concentrated enough for significant person-to-person spread. If that’s required, we have no really ancient diseases, because we defined them away.
The model implies that if funding and prestige increased, this limitation would be reduced. And I would think we don’t need prestige nearly as much as funding—even if near-top scientists were recruited and paid the way second and third string major league players in most professional sports were paid, we’d see a significant relaxation of the constraint.
Instead, the uniform wage for most professors means that even the very top people benefit from supplementing their pay with consulting, running companies on the side, giving popular lectures for money, etc. - all of which compete for time with their research.
Yes, this might help somewhat, but there is an overhead / deduplication tradeoff that is unavoidable.
I discussed these dynamics in detail (i.e. at great length) on Ribbonfarm here.
The large team benefit would explain why most innovation happens near hubs / at the leading edge companies and universities, but that is explained by the other theories as well.
The problem with fracturing is that you lose coordination and increase duplication.
I have a more general piece that discusses scaling costs and structure for companies that I think applies here as well—https://www.ribbonfarm.com/2016/03/17/go-corporate-or-go-home/
Yes—and this is equivalent to saying that evidence about probability provides Bayesian metric evidence—you need to transform it.