Great long-form interview with Andrew Yang here: Joe Rogan Experience #1245 - Andrew Yang.
Did you make any update regarding the simplicity / complexity of value?
My impression is that theoretical simplicity is a major driver of your preference for NU, and also that if others (such as myself) weighed theoretical simplicity more highly that they would likely be more inclined towards NU.
In other words, I think theoretical simplicity may be a double crux in the disagreements here about NU. Would you agree with that?
Meta-note: I am surprised by the current karma rating of this question. At present, it is sitting at +9 points with 7 votes, but it would be at +2 with 6 votes w/o my strong upvote.
To those who downvoted, or do not feel inclined to upvote—does this question not seem like a good use of LW’s question system? To me it seems entirely on-topic, and very much the kind of thing I would want to see here. I found myself disagreeing with much of the text, but it seemed to be an honest question, sincerely asked.
Was it something about the wording (either of the headline or the explanatory text) that put you off?
Relatedly: shorter articles don’t need to be as well-written and engaging for me to actually read to the end of them.
I suspect, though, that there is wide variation in willingness to read long posts, perhaps explained (in part) by reading speed.
If the rationality and EA communities are looking for a unified theory of value
Are they? Many of us seem to have accepted that our values are complex.
Absolute negative utilitarianism (ANU) is a minority view despite the theoretical advantages of terminal value monism (suffering is the only thing that motivates us “by itself”) over pluralism (there are many such things). Notably, ANU doesn’t require solving value incommensurability, because all other values can be instrumentally evaluated by their relationship to the suffering of sentient beings, using only one terminal value-grounded common currency for everything.
This seems like an argument that it would be convenient if our values were simple. This does not seem like strong evidence that they actually are simple. (Though I grant that you could make an argument that it might be better to try to achieve only part of what we value if we’re much more likely to be successful that way.)
FWIW, I was thinking of the related relationship as a human-defined one. That is, the author (or someone else?) manually links another question as related.
Q&A in particular is something that I can imagine productively scaling to a larger audience, in a way that actually causes the contributions from the larger audience to result in real intellectual progress.
Do you mean scaling it as is, or in the future?
I think there’s a lot of potential to innovate on the Q&A system, and I think it’d be valuable to make progress on that before trying to scale. In particular, I’d like to see some method of tracking (or taking advantage of) the structure behind questions—something to do with how they’re related to each other.
Maybe this is as simple as marking two questions as “related” (as I think you and I have discussed offline). Maybe you’d want more fine-grained relationships.
It’d also be cool to have some way of quickly figuring out what the major open questions are in some area (e.g. IDA, or value learning), or maybe what specific people consider to be important open questions.
Have any posts from LW 2.0 generated new conceptual handles for the community like “the sanity waterline”? If not, maybe it’s because they just aren’t reaching a big enough audience.
Doesn’t this get the causality backwards? I’m confused about the model that would generate this hypothesis.
One way I can imagine good concepts not taking root in “the community” is if not enough of the community is reading the posts. But then why would (most of) the prescriptions seem to be about advertising to the outside world?
And the stories of their students are heartwarming.
Btw, Lambda School twitter is fun to follow. They’re doing some impressive stuff.
2) Which assets will be more scarce/in demand as that happens? Are there currently available opportunities for “shorting” the education bubble and invest in ways which will yield profit when it pops?
Vocational schools seem like a reasonable bet. In particular something like Lambda School, where they’ve aligned incentives by tying tuition to alumni income.
VCs seem to agree, pouring in $14MM in a series A in October 2018, followed by an additional $30MM in a series B just 3 months later.
It seems to me that perhaps your argument about expected utility maximization being a trivial property extends back one step previous in the argument, to non-exploitability as well.
AlphaZero is better than us at chess, and so it is non-exploitable at chess (or you might say that being better at chess is the same thing as being non-exploitable at chess). If that’s true, then it must also appear to us to be an expected utility maximizer. But notably the kind of EU-maximizer that it must appear to be is: one whose utility function is defined in terms of chess outcomes. AlphaZero *is* exploitable if we’re secretly playing a slightly different game, like how-many-more-pawns-do-I-have-than-my-opponent-after-twenty-moves, or the game don’t-get-unplugged.
Going the other direction, from EU-maximization to non-exploitability, we can point out that any agent could be thought of as an EU-maximizer (perhaps with a very convoluted utility function), and if it’s very competent w.r.t. its utility function, then it will be non-exploitable by us, w.r.t. outcomes related to its utility function.
In other words, non-exploitability is only meaningful with respect to some utility function, and is not a property of “intelligence” or “competence” in general.
Would you agree with this statement?
when everything that can go wrong is the agent breaking the vase, and breaking the vase allows higher utility solutions
What does “breaking the vase” refer to here?
I would assume this is an allusion to the scene in The Matrix with Neo and the Oracle (where there’s a paradox about whether Neo would have broken the vase if the Oracle hadn’t said, “Don’t worry about the vase,” causing Neo to turn around to look for the vase and then bump into it), but I’m having trouble seeing how that relates to sampling and search.
For the parenthetical in Proposed Experiment #2,
or you can train a neural net to try to copy U
should this be “try to copy V”, since V is what you want a proxy for, and U is the proxy?
As I was writing the last few paragraphs, and thinking about Wei Dei’s objections, I found it hard to clearly model how CAIS would handle the cancer example.
This link appears to be broken. It directs me to https://www.lesswrong.com/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services-as/comment/gMZes7XnQK8FHcZsu, which does not seem to exist.
Replacing the /comment/ part with a # gives https://www.lesswrong.com/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services-as#gMZes7XnQK8FHcZsu, which does work.
(Also it should be “Dai”, not “Dei”.)
you should actually first try to integrate each technique and get a sense of whether it worked for you (or why it did not).
This could actually be the theme of the podcast. “Each week I try to integrate one technique and then report on how it went.”
Sounds more interesting than just an explanation of what the technique is.
I wanted to get a better sense of the risk, so here is some arithmetic.
Putting together one of the quotes above:
An estimated 300,000 sport-related traumatic brain injuries, predominantly concussions, occur annually in the United States.
And this bit from the recommended Prognosis section:
Most TBIs are mild and do not cause permanent or long-term disability; however, all severity levels of TBI have the potential to cause significant, long-lasting disability. Permanent disability is thought to occur in 10% of mild injuries, 66% of moderate injuries, and 100% of severe injuries.
And this bit from the Epidemiology section:
a US study found that moderate and severe injuries each account for 10% of TBIs, with the rest mild.
We get that there are 300k sport-related TBI’s per year in the US, and of those, 240k are mild, 30k are moderate, and 30k are severe. Those severity levels together result in 24k + 20k + 30k ~= 75k cases of permanent disability per year.
To put that in perspective, we can compare to another common activity that has potential to cause harm:
In 2010, there were an estimated 5,419,000 crashes, 30,296 of with fatalities, killing 32,999, and injuring 2,239,000.
If we say that a fatality and a permanent disability due to brain injury are the same order of magnitude of badness, this suggests that sports and traveling by car expose the average (as in mean, not median) American to roughly the same level of risk.
Would be interesting to dig deeper to see how much time Americans spend in cars vs playing sports on average (and then you’d also want to look at the benefits you get from each), but I’ll stop here for now.
but this seems like reason to doubt that AI has surpassed human strategy in StarCraft
I think Charlie might be suggesting that AlphaStar would be superior to humans, even with only human or sub-human APM, because the precision of those actions would still be superhuman, even if the total number was slightly subhuman:
the micro advantage for 98% of the game isn’t because it’s clicking faster, its clicks are just better
This wouldn’t necessarily mean that AlphaStar is better at strategy.
“Does perfect stalker micro really count as intelligence?”
Love this bit.
the evidence is pretty strong that AlphaStar (at least the version without attention that just perceived the whole map) could beat humans under whatever symmetric APM cap you want
This does not seem at all clear to me. Weren’t all the strategies using micro super-effectively? And apparently making other human-detectable mistakes? Seems possible that AlphaStar would win anyway without the micro, but not at all certain.
Interesting analysis here:
I will try to make a convincing argument for the following:
1. AlphaStar played with superhuman speed and precision.
2. Deepmind claimed to have restricted the AI from performing actions that would be physically impossible to a human. They have not succeeded in this and most likely are aware of it.
3. The reason why AlphaStar is performing at superhuman speeds is most likely due to it’s inability to unlearn the human players tendency to spam click. I suspect Deepmind wanted to restrict it to a more human like performance but are simply not able to. It’s going to take us some time to work our way to this point but it is the whole reason why I’m writing this so I ask you to have patience.