Could it be both “it seems like you’re doing something insane and terrible and I will act accordingly” and “it probably also seems that way to many other people, and my silence would look like an approval, so I need to distance myself from you visibly (instead of just ignoring you)”?
One thing is “I don’t trust you anymore” and the other thing is “you are burning social capital, so I am building a firewall between you and me”.
I can imagine that some people join EA for the vibes, not because they are fully on board with consequentialism. For example, they are vegans, they see veganism popular among the EAs, so they treat EA as another vehicle to spread veganism.
(Generally, whenever we are interacting with a movement greater than us, there is a risk of being taken over by them.)
I guess that people are strongly divided on the dimension of “openness to culinary experience”. Some are happy exploring new meals, and the best way to make them go vegan is to show them new tasty vegan meals. Others only want more of the same, and the only way to make them go vegan is to create a hopefully indistinguishable version of the same (and probably even then they will complain, because placebo effect is an important component of taste, so even lab-grown meat will “taste different”).
Anyway, the greater point here is that you have limited weirdness points, and if you spend them all on veganism, there is not enough left for anything else.
Yeah, it could be that 99% people hate it but remain silent, and 1% really like it and say it out loud. You think you are popular in general, while in reality you are just optimizing for the tastes of 1%, and getting more and more silent disapproval of the 99%. Even worse, the reason why the 1% like it may be different than you think; for example you think “this is nice”, but for them it is an arbitrary ingroup symbol, so you are ultimately wrong about everyone.
This seems bad.
The entire situation seems like: Facing extinction, humans still continue behaving like complete idiots. If you think the higher stakes would change the behavior of homo sapiens somehow, you don’t really know him.
This makes me wonder whether the situation could have been different if ten years ago we gave greater priority to raising the human sanity waterline. It is tempting to think that we can win by solving the alignment and building a friendly AI, which will then solve all the problems for us. But that requires the people involved to be extremely sane; and even if they are, there is a large chance that the almost finished AI will be at the last moment copied by an adversary, or simply seized by the government, or a military coup.
by buying a subscription to a service, you are subscribing not only to the service that exists today, but to a stream of improvements to that service, including improvements that have not yet been announced or developed, delivered as fast as possible
Is this as crazy as it sounds to me, or are there some legal precedents that would support such perspective?
Sometimes there are good reasons to “refuse logic”. Suppose that you give me a mathematical proof that is 500 pages long and ultimately proves that 2+2=5. I say there is a mistake somewhere, you say nope this was verified by Lean, I say there may be a bug in Lean… at the end of the day, there will be a mathematical proof that according to you is valid and verified, but I refuse to accept it, and won’t even spend my time going through the pages to show you the exact line I disagree with; I just disagree with the conclusion, full stop.
There are also objections to thought experiments, which may mean something like “I do not think that your model actually captures all the important parts of reality, so if we used the conclusion in real life, it would be wrong for reasons that are missing from your model”. Sometimes you do not even know what specifically is missing, it’s just that you know that things in real life evolve differently than the model would predict.
It is also valid to have two opinions X and Y such that their naive extensions contradict each other, and you are not sure how to extend them properly. Two approximations that are both useful in their typical use, but contradict each other in some special case.
The usual answer is “more research is needed”.
For $10 I can only give you an outline:
List of people who hate the rationalist community (hint: they can provide cool quotes)
David Gerard
Alexander Kruel
Cade Metz
r/SneerClub
List of murderers associated with the rationalist community
Zizians
more Zizians
even more Zizians
List of sexual scandals associated with the rationalist community
Brent Dill
Aella’s sex parties
Other miscellaneous scandals
Harry Potter fan-fic
Leverage Research
List of people who are not rationalists, but can still be associated with them if you squint the right way
Neoreactionaries
literally Hitler
I’ve read enough rationalist material to understand that my tribal enemies are not actually bad people as a whole,
Please explain vice signaling. (That is something that the rationalist materials ignore, but it exists in real world.)
You don’t have to be left wing to find the neoreactionaries annoying.
They bring a few interesting ideas, but after a while you have already heard them all, and then it’s just endless obscurantist writing. They are good at criticizing status quo, but unable to respond to criticism of their own ideas (such as “have you guys noticed that North Korea is actually not a paradise?”). I would delete them from my Facebook wall too, if they bothered to go there, because all they can do is repeat their slogans.
The Grader sounds like a machine God. Perhaps any intelligence trained by rewards and punishment is likely to invent its idea of a superintelligent source of the rewards and punishments.
Could you make a shorter version of your point? If it is “people in this forum are unlikely to calculate the exact probability of a specific attack”, that sounds correct, but is irrelevant. The AI can try multiple attacks at the same time, and e.g. update the virus design once it gets the data that the virus is not as efficient as expected.
AI doesn’t drink, doesn’t eat
AI needs energy, and it may have a preference to e.g. remove the water from the environment. It may also have a preference about global temperature and oxygen levels.
I trust the plebs when they same something is wrong (though I may disagree with their specific diagnosis), I do not trust them when they propose a solution. Where does that put me on the populism/elitism scale?
This sounds to me like a proposal of a new false dilemma; I am not sure what benefits is it supposed to bring.
Yep, the rich of the near future are likely to think about UBI the same way Elon Musk thought about USAID. In theory, they will have enough resources to make other people happy at no detriment to themselves, but in practice… if they were that kind of people they probably wouldn’t have ended in those positions.
They might give something on a whim, but just as likely take something away on a whim. Most likely, think about completely unrelated things.
Are you more afraid of death or losing high status?
These are different concerns. In the (probably unlikely) case the AIs turn out to be friendly, you will live, maybe much longer that otherwise, but your contributions will be unimportant, a child’s play compared to what the AIs could do if they cared.
I want to have work that makes everybody admire and approve of me! Maybe I can’t, or don’t want to, do things that are objectively necessary or contribute to society! I still want everybody to like me!
AIs will solve the status scarcity by generating virtual admirers for each of us. Sycophancy is just the first step.
the natural conclusion is that psychiatric categories are loose and don’t map exactly to who benefits from what medication. Some people with “depression” will respond strongly to a given med, some won’t, and it averages out to look like the med has weak or no effect on “depression”.
We should collect more data and let the machine crunch the numbers. Too bad the data would be immediately sold on the market.
About Chornobyl, I think it is a curious case where Soviet propaganda and American propaganda want to convince the people about the same thing: that it was a human mistake. (The Soviet propaganda, to make the point that humans are fallible. The American propaganda, to make the point that the Soviet system was bad.) As far as I know, it was essentially a technical problem: the nuclear reactor is a machine with many possible states, the operation manuals did not include all possible situations.
There was some switch X (I do not remember the details anymore) which would only rarely be in the “on” position; and when it was, it was only temporarily. But there was nothing seemingly dangerous about the switch per se; it just gave a small boost to something, so most of the time you did not need it. There was also a shutdown button, which also seemed perfectly safe. The only problem was that if you pressed the shutdown button while X was on, the entire thing exploded within a few seconds, for technical reasons that are only obvious in hindsight. Apparently, the people designing and testing the reactor have never considered the scenario “what if someone presses the shutdown button while X is on”, because both functionalities were tested separately, and the X was used only rarely and temporarily. There were already dozen reactors with exactly the same bug working for years, and the situation simply never happened in real life… until one day, it did. Someone forgot to turn X off, but no one knew that turning X off before shutdown was important; people usually just turned it off simply because they didn’t need it anymore. And when the problem started, it was too late for anyone to do anything about it, Soviet leadership or not, the time interval between the first suspicious thing and the explosion was less than a minute, the hypothesis “maybe this is related to X” was not salient, and it was probably too late to actually do something about it anyway.
We can blame the Soviet system for the long-term lack of testing. But as far as I know, there was no immediate mistake made on that day.
(As an analogy, imagine getting a new generation of notebooks, which explode when you shut them down with the Caps Lock button on. No one noticed it, because no one uses Caps Lock anymore. And when the testers are told to test Caps Lock, they turn it on, press a few keys, then turn it off, press a few more keys, and consider it tested. No one has ever thought about including “shutdown the computer while the Caps Lock is on” in a test suite. All tests have passed, the notebooks are already sold on the market for a few years. Would you really blame the first person who forgot to turn off the Caps Lock before shutting down their computer?)
If the AI wants to die, and can kill itself simply by sending a proper signal, it will send the signal. No useful work done.
If the AI wants to die, but cannot achieve it simply, it may do something with lots of collateral damage, e.g. a huge explosion.
If the AI wants to die, but the only condition is that there will be no negative impact on humans… well, you no longer need the death wish, because “there will be no negative impact on humans” assumes that alignment was already solved.
The lesson is important, but it felt like the article had 5x more words than necessary.
An explosion will kill you, but you can’t predict where the individual parts of your body will end up.