“if it’s ok to do A or B then it’s fine to run an experiment on A vs B”
Allowing A and B, and allowing an experiment on A vs. B, may create different incentives, and these incentives may be different enough to change whether we should allow the experiment versus allowing A and B.
Luckily for you, there definitely exists a rule that tells you the best possible move to play for every given configuration of pieces—the rule that tells you the move that maximizes the probability of victory (or since draws exist and may be acceptable, the move that minimizes the probability of defeat.
If your opponent is a perfect player, each move has a 0% or 100% probability of victory. You can only maximize it in a trivial sense.
If your opponent is an imperfect player, your best move is the one that maximizes the probability of victory given your opponent’s pattern of imperfection. Depending on what this pattern is, this may also mean that each move has a 0% or 100% probability of victory.
Your process of deciding what to do may at some point include simulating Omega and Omicron. If so, this means that when Omega and Omicron are simulating you, they are now trying to solve the Halting Problem. I am skeptical that Omega or Omicron can solve the Halting Problem.
I would suggest that this is ameliorated by the following:
Nobody actually believes that you are to blame for every bad consequence of things you do, no matter how indirect. A conscientious person is expected to research and know some of the indirect consequences of his actions, but this expectation doesn’t go out to infinity.
While you don’t get credit for unintended good consequences in general, you do get such credit in some situations. Specifically, if the good consequence is associated with a bad consequence, you are allowed to get credit for the good consequence and trade it off against the bad consequence. If I buy a tomato, bad consequences of this (someone else can’t get one) are balanced off against good consequences (the store knows to order extra tomatoes next week) because they are both part of the same process. On the other hand, I can’t offset a murder by saving two drowning victims, because the acts are not entwined and I could do one without doing the other.
How can you (in general) conclude something by examining the source code of an agent, without potentially implicating the Halting Problem?
I think there’s a difference between “Most of the IRS tax code is reasonable” and “Most of the instances where the IRS tax code does something are instances where it does reasonable things.” Not all parts of the tax code are used equally often. Furthermore, most unreasonable instances of a lot of things will be rare as a percentage of the whole because there is a large set of uncontroversial background uses. For instance, consider a completely corrupt politician who takes bribes—he’s not going to be taking a bribe for every decision he makes and most of the ones he does make will be uncontroversial things like “approve $X for this thing which everyone thinks should be approved anyway”.
I want employees to ask themselves whether they are willing to have any contemplated act appear the next day on the front page of their local paper—to be read by their spouses, children and friends—with the reporting done by an informed and critical reporter.”
Leaving out “parents” gets rid of some of the obvious objections, but even then, I don’t want my children to know about my sexual fetishes. Other objections may include, for instance, letting your friends know that you voted for someone who they think will ruin the country. And I certainly wouldn’t want rationalist-but-unpopular opinions I hold to be on the front page of the local paper to be seen by everyone (Go ahead, see what happens when the front page of the newspaper announces that you think that you should kill a fat man to stop a trolley.) This aphorism amounts to “never compartmentalize your life” which doesn’t seem very justifiable.
Bob does not know X. That’s why Alice is telling Bob in the first place.
Conversational phrases aren’t supposed to be interpreted literally. “Everybody knows” never means “literally every single person knows”. This is about equivalent to complaining that people say “you’re welcome” when the person really wouldn’t be welcome under some circumstances.
Don’t be the literal Internet guy who thinks this way.
I think the word “unbiased” there may be a typo; your statement would make a lot more sense if the word you meant to put there was actually “biased”.
I meant “unbiased” in scare quotes. Typical newsfeeds that are claimed to be unbiased in the real world (but actually may not be).
Typical unbiased newsfeeds in the real world are created by organizations with bias who have an interest in spreading biased news. It could, of course, be that this was about a rare instance where this was not the case, but the odds are against it.
Manipulative newsfeeds aren’t an example of an AI becoming manipulative when the human just wanted it to be unbiased. They’re an example of an AI becoming manipulative when the human also wanted it to be manipulative, but didn’t want to be too obvious about it.
Don’t Godel sentences rebut the ideas of groundedness or of creating a system where self-referential sentences are blocked? Their existence means that you can create something that behaves as a self-referential sentence and has the associated paradoxes while using only normal arithmetic and without a “this sentence”.
I would not, in fact, save a drowning child.
Or rather, I’d save a central example of a drowning child, but I wouldn’t save a drowning child under literally all circumstances, and I think most people wouldn’t either. If a child was drowning in a scenario similar to one that Singer uses it as an analogy for, it would be something like a scenario where there is an endless series of drowning children in front of me with an individually small but cumulatively large cost to saving them. Under those circumstances, I would not save every drowning child, or even try to maximize the number of drowning children I do save.
It seems to me that if you expect that the results of your experiment can be useful in and generalized to other situations, then it has to be possible to replicate it. Or to put it another way, if the principle you discovered is useful for more than running the same program with a different seed, shouldn’t it be possible to test it by some means other than running the same program with a different seed?
the pre-brainwashed person had preferences about their future selves
That would qualify as
for instance, you might think that forcibly changing preferences is different from creating a being with unusual preferences
Also, it’s possible for people to have preferences about either their descendants, or about other sentient beings, just like they have preferences about their future selves. In fact, I would suggest that pretty much all the opposition to the idea is because people have preferences about their descendants or about other sentient beings. Again, it may be useful to spell out why you think those preferences merit less respect than preferences about one’s future self.
(Note that some answers to this require making assumptions about how to aggregate preferences, that are also serious points of disagreement. Fo instance, you might say that if you create a lot of slaves, the preferences of that large number should have a large weight. Such assumptions can also be questioned, and by most people, would be questioned.)
Instead of preregistering all experiments, maybe researchers could run experiments and observe results, formulate a theory, and then preregister an experiment that would test the theory—but in this case I would expect that researchers end up “preregistering” experiments that are very similar to the experiments that generated the theory, such that the results are very likely to come out in support of the theory.
Why would you expect this? Assuming you are not suggesting “what if the researchers lie and say they did the experiment again when they didn’t”, then doing a similar experiment again is called “replication”. If the initial result was caused by p-hacking, then the similar experiment won’t support the theory. This is why we do replication.
Also, I notice the term “p-hacking” appears nowhere in your post.
Consider a similar situation without creating a race: some wizard brainwashes an existing person into becoming a willing slave. Is it moral to thwart the preferences of the brainwashed person bu not enslaving him, or by forcibly modifying his brain to desire freedom again? Most people would say yes.
You might argue that there is a difference (for instance, you might think that forcibly changing preferences is different from creating a being with unusual preferences) but it may be useful to spell out those differences and distinguish between objections that are affected by those differences and objections which are not.
My friend explains again that he does not have the time to see any doctor the next day
He had the time to see a doctor. He didn’t have the time to see a doctor without disrupting his life, which isn’t the same thing as actually not having the time to see the doctor. And the fact that seeing a doctor disrupts his life is his own fault for delaying the appointment.
The doctor shouldn’t be required to alter his procedure just because doing so would alleviate the consequences of the patient’s own decisions.
The conceptual gap between a standard use of a poison from a hardware store and a deadly use is much larger than the gap between a standard use and a deadly use of a medication, so I would expect far more tragedies to come from the medication than from the hardware store poison.
Nobody’s going to self-diagnose and inject themselves with poison from a hardware store.
Under what circumstances do you get people telling you they are fine? That doesn’t happen to me very much—”I’m fine” as part of normal conversation does not literally mean that they are fine.