Creating arbitrary animals that are barely alive, don’t need food, water, air, or movement, and made of easily workable material which is also good as armor seems like a good place to start, and also within the bounds of magic. This isn’t as absurd as it seems. Essentially living armor plates. You’d want them to be thin so you could have multiple layers, and to fall off when they die, and various similar things. Or maybe on a different scale, like scale or lamellar armor.
taygetea
Well, I did say it far outweighed it. Even that’s less of an inconvenience in my mind, but that’s getting to be very much a personal preference thing.
the second, cybernetic, industrial revolution “is [bound] to devalue the human brain, at least in its simpler and more routine decisions
It certainly seems like he considered it, at least on a basic level, enough to be extrapolated.
According to Quirrel, yes, they are. “Anything with a brain”. And I notice that you’ve only looked at what we’ve directly seen. The presence of spells like all the ones you mentioned lead me to think that you can do more directed things with spells harry hasn’t come across yet.
Relating to your first point, I’ve read several stories that talk about that in reverse. AIs (F or UF is debatable for this kind) that expand out into the universe and completely ignore aliens, destroying them for resources. That seems like a problem that’s solvable with a wider definition of the sort of stuff it’s supposed to be Friendly to, and I’d hope aliens would think of that, but it’s certainly possible.
From my experiences trying similar things over IRC, I have found that the lack of anything holding you to your promises definitely is a detriment to most people. I have found a few for whom that’s not the case, but that’s very much the exception. That’s definitely a failure mode to look out for, doing this online (especially in text) won’t work for many people. In addition, this discrepancy can create friction between people.
The general structure of the failure tends to be one person feeling vaguely bad about not talking as much, or missing a session. And then when they don’t have many vectors to viscerally receive signals of disapproval, of the kind that would cause them to be uncomfortable and go through with it even when they don’t want to, it becomes easiest to do it the next time. Schelling Fences are easier to break without face to face interaction.
There should be ways to bypass that problem. One of the memes around LW is actively reinforcing positive things, instead of relying on implied approval. If you can create a culture of actively rewarding success, and treating apathy as something to be stamped out at every point, then you can do it. You can also make a point to create norms where one goes out of their way to help someone who falls behind to figure out what the true problem is. If you can manage that, instead of silence or simple berating, then you can make it work. Ideas around Tell Culture can help you with this. Unfortunately, this also requires diverting a lot of focus into preserving those conditions. Creating community norms is hard, but that seems like the way you avoid that problem.
I don’t mean to imply that you want to start a community around this along the lines of the LW study hall, but this is what I have found from my attempts. Maybe someone will find it helpful.
You could construct an argument about needing to reinforce explicitly using system-2 ethics on common situations to make sure that you associate those ethics implicitly with normal situations, and not just contrived edge cases. But that seems to be even a bit too charitable. And also easily fixed if so.
This would rely on a large fraction of pageviews being from Wikipedia editors. That seems unlikely. Got any data for that?
The entire point of “politics is the mind-killer” is that no, even here is not immune to tribalistic idea-warfare politics. The politics just get more complicated. And the stopgap solution until we figure out a way around that tendency, which doesn’t appear reliably avoidable, is to sandbox the topic and keep it limited. You should have a high prior that a belief that you can be “strong” is Dunning-Kruger talking.
Nitpick: BTC can be worth effectively less than $0 if you buy some then the price drops. But in a Pascalian scenario, that’s a rounding error.
More generally, the difference between a Mugging and a Wager is that the wager has low opportunity cost for a low chance of a large positive outcome, and the Mugging is avoiding a negative outcome. So, unless you’ve bet all the money you have on Bitcoin, it maps much better to a Wager scenario than a Mugging. This is played out in the common reasoning of “There’s a low chance of this becoming extremely valuable. I will buy a small amount corresponding to the EV of that chance, just in case”.
Edit: I may have misread, but just to make sure, you were making the gold comparison as a way to determine the scale of the mentioned large positive outcome, correct? And my jump to individual investing wasn’t a misinterpretation?
I can’t say much about the consequences of this, but it appears to me that both democracy and futarchy are efforts to more closely approximate something along the lines of a CEV for humanity. They have the same problems, in fact. How do you reconcile mutually exclusive goals of the people involved?
In any case, that isn’t directly relevant, but linking futarchy with AI caused me to notice that. Perhaps that sort of optimization style, of getting at what we “truly want” once we’ve cleared up all the conflicting meta-levels of “want-to-want”, is something that the same sorts of people tend to promote.
I think I see the problem. Tell me what your response to this article is. Do you see messy self-modification in pursuit of goals at the expense of a bit of epistemic rationality to be a valid option to take? Is Dark == Bad? In your post, you say that it is generally better not to believe falsehoods. My response to that is that things which depend on what you expect to happen are the exception to that heuristic.
Life outcomes are in large part determined by your background that you can’t change, but expecting to be able to change that will lead you to ignore fewer opportunities to get out of that situation. This post about luck is also relevant.
Determining the language to use is a classic case of premature optimization. No matter what the case, it will have to be provably free of ambiguities, which leaves us programming languages. In addition, in terms of the math of FAI, we’re still at the “is this Turing complete” sort of stage in development. So it doesn’t really matter yet. I guess one consideration is that the algorithm design is going to take way more time and effort than the programming, and the program has essentially no room for bugs (Corrigibility is an effort to make it easier to test an AI without it resisting). So in that sense, it could be argued that the lower level the language, the better.
Directly programming human values into an AI has always been the worst option, partially for your reason. In addition, the religious concept you gave can be trivially broken by two different beings having different or conflicting utility functions, and so acting as if they were the same is a bad outcome. A better option is to construct a scheme so that the smarter the AI gets, the better it approximates human values, by using its own intelligence to determine them, as in coherent extrapolated volition.
Typical Mind Fallacy. Allows people to actually cooperate for once. One of the things I’ve been thinking about is how one person’s fundamental mind structure is interpreted by another as an obvious status grab. I want humans to better approximate Aumann’s Agreement Theorem. Solve the coordination problem, solve everything.
I’ve begun to notice discussion of AI risk in more and more places in the last year. Many of them reference Superintelligence. It doesn’t seem like a confirmation bias/Baader-Meinhoff effect, not really. It’s quite an unexpected change. Have others encountered a similar broadening in the sorts of people you encounter talking about this?
Hi. I don’t post much, but if anyone who knows me can vouch for me here, I would appreciate it.
I have a bit of a Situation, and I would like some help. I’m fairly sure it will be positive utility, not just positive fuzzies. Doesn’t stop me feeling ridiculous for needing it. But if any of you can, I would appreciate donations, feedback, or anything else over here: http://www.gofundme.com/usc9j4
This month (and a half), I dropped out of community college, raised money as investment in what I’ll do in the future, moved to Berkeley, got very involved in the rationalist community here, smashed a bunch of impostor syndrome, wrote a bunch of code, got into several extremely promising and potentially impactful projects, read several MIRI papers and kept being urged to involve myself with their research further.
I took several levels of agency.
That’s a pretty large question. I’d love to, but I’m not sure where to start. I’ll describe my experience in broad strokes to start.
Whenever I do anything, I quickly acclimate to it. It’s very difficult to remember that things I know how to do aren’t trivial for other people. It’s way more complex than that… but I’ve been sitting on this text box for a few hours. So, ask a more detailed question?
So, to my understanding, doing this in 2015 instead of 2018 is more or less exactly the sort of thing that gets talked about when people refer to a large-scale necessity to “get there first”. This is what it looks like to push for the sort of first-mover advantage everyone knows MIRI needs to succeed.
It seems like a few people I’ve talked to missed that connection, but they support the requirement for having a first-mover advantage. They support a MIRI-influenced value alignment research community, but then they perceive you asking for more money than you need! Making an effort to remind people more explicitly why MIRI needs to grow quickly may be valuable. Link the effect of ‘fundraiser’ to the cause of ‘value learning first-mover’.
The messiness and potential for really unpleasant sounds, in my mind, far outweighs the need for a specific type of dry-erase marker. Though that might be related to how easily sounds can be unpleasant to me in particular.