Any disagreement with Robin Hanson (https://twitter.com/robinhanson/status/1022535475410731009)?
It seems to me that subsidy addresses V, II and maybe III, but not so much IV and I. Does that seem right?
(Also, I think he’s suggested non-anonymous trading as a cure for insider trading before)
Section of an interesting talk relating to this by Anna Salamon. Makes the point that if ability to improve its model of fundamental physics is not linear in the amount of Universe it controls, such an AI would be at least somewhat risk-averse (with respect to gambles that give it different proportions of our Universe)
This doesn’t make any sense but people try it anyway, especially when they’re talking to high status people and/or experts. “Oh, okay, I’ll try hard to believe what the expert said, so I look like I know what I’m talking about.”
It seems to me, sometimes you just need to ignore your model’s output and just go with an expert’s opinion. For example, the current expected value of trying to integrate the expert’s model might be low (even zero if it’s just impossible for you to get it!).
Still, I agree that you shouldn’t throw away your model. Looking at your diagrams, it’s easy to think we could take a leaf from Pearl’s Causality (Ch. 3). Pearl is talking about modelling intervention in causal graphs. That does seem kind of similar to modelling an ‘expert override’ in belief networks, but that’s not necessary to steal the idea. We can either just snip the arrows from our model to the output probability and set the output to the expert’s belief, or (perhaps more realistically), add an additional node ‘what I view to be expert consensus’ that modulates how our credence depends on our model.
Your diagrams even have white circles before the credence that could represent our ‘seeming’—i.e. what the probability we get just from the model is—and then that can be modulated by a node representing expert consensus and how confidently we think we can diverge from it.
Here is my attempt at a summary of (a standalone part of) the reasoning in this post.
An agent trying to get a lot of reward can get stuck (or at least waste data) when the actions that seem good don’t plug into the parts of the world/data stream that contain information about which actions are in fact good. That is, an agent that restricts its information about the reward+dynamics of the world to only its reward feedback will get less reward
One way an agent can try and get additional information is by deductive reasoning from propositions (if they can relate sense data to world models to propositions). Sometimes the deductive reasoning they need to do will only become apparent shortly before the result of the reasoning is required (so the reasoning should be fast)
The nice thing about logic is that you don’t need fresh data to produce test cases: you can make up puzzles! As an agent will need fast deductive reasoning strategies, they may want to try out the goodness of their reasoning strategies on puzzles they invent (to make sure they’re fast and reliable (if they hadn’t proved reliability))
In general, we should model things that we think agents are going to do, because that gives us a handle on reasoning about advanced agents. It is good to be able to establish what we can about the behaviour of advanced boundedly-rational agents so that we can make progress on the alignment problem etc
This changed my mind about the parent comment (I think the first paragraph would have done so, but the example certainly helped).
In general, I don’t mind added concreteness even at the cost of some valence-loading. But seeing how well “sanction” works and some other comments that seem to disagree on the exact meaning of “punch”, I guess not using “punch” would have been better
I suppose the problem comes when the AI starts to communicate with us. There would be a lot of information that they could exploit. Even if they don’t get any sense of our physics, if they are able to model us we might be in trouble. And even if we didn’t give them any direct communication (for example manifesting puzzles in their world, the solution of which would allow us to solve our own questions), they might promote simulation to a reasonable hypothesis.
Now let’s imagine a situation where [...] there are two potential chocolate vendors, one of them of standard politeness, the second very rude. Without running the experiment, I’m confident that people will be willing to pay more in the second case than in the first.
Do you mean the first case (standard politeness)? If not, I’m not sure why you think people would be willing to pay more. Is it that the rude vendor is doing some sort of handicapping, and the fact that they’re still in business proves they have great chocolate?
Judging by the structure of your post though, I think you’re pointing out that though the rudeness doesn’t change the quality of the chocolate, it still reasonably (according to most humans) changes preferences about where to buy it
The (apparent) reasoning I am challenging, Eliezer, is this: “Alice is making an invalid argument for side A” → “therefore Alice would not make an invalid argument for side B”. This seems faulty.
As some anecdata, I know several people who have done this precisely for global warming. I suspect Eliezer has spoken to them as well, because the standard response when you point out that they’re using the reverse of the “day of cold weather” argument they’ve derided is:
Well, it’s important to use only valid arguments… but there was a sustained pattern of record highs worldwide over multiple years which does count as evidence, and that particular very hot day was a part of that pattern, so it’s valid evidence for global warming.
Now, what I don’t know is if they actually wouldn’t reason wrongly on both sides, but have already been taught elsewhere the problem with the “day of cold weather” argument. It could still be that the problem is not malice but failure to generalise
This was a great explanation—well-written, clear and not too simple.
I think if there were a flaw, it might be slightly too long. The “common knowledge production in society” section might not be entirely pulling its weight. That said, it was interesting and I think it’s useful to know ways in which we do (try to) solve the problem. (Our civilisation might be inadequate at points, but if the whole point of civilisation is to move to better equilibria, it would be surprising if we had no tech for it!)
I wonder if including a bit of a reader’s guide at the beginning. Something like “if you’re in a hurry and looking to generate the key understanding, try just reading the dictatorship example from the ‘three coordination problems’ section and skip the section on how we generate common knowledge”.
This could be a bad idea, as the post is known to be good (curated, highly voted up) and this has some risk to reduce the quality. Also, perhaps I have not fully internalised the importance of our existing common knowledge institutions and so underrate that section!
I want to echo the other comments thanking you for making this lay-approachable and for the fun format!
I do find myself confused by some of the statements though. It may be that I have a root misunderstanding or that I am misreading some of the more quickly stated sentences.
For example, when you talk about the trees of truth & falsehood and the gap in the middle: am I right in thinking of these trees as provability and non-provability? Rather than perhaps truth & falsehood
Also, in the existence proof for Bs such that and we can prove , you say that if B is a logical truth, A → B must be provable, because anything implies a logical truth. It seems right to me that anything logically implies a logical truth. But surely we can’t prove all logical truths from anything—what if it’s a truth in the grey area such that it can’t be proved at all?
If someone can put me right, that would be great
I wasn’t aware that CFAR had workshops in Europe before this comment. I applied for a workshop off the back of this. Thanks!
As I read through, the core model fit well with my intuition. But then I was surprised when I got to the section on religious schisms! I wondered why we should model the adherents of a religion as trying to join the school with the most ‘accurate’ claims about the religion.
On reflection, it appears to me that the model probably holds roughly as well in the religion case as the local radio intellectual case. Both of those are examples of “hostile” talking up. I wonder if the ways in which those cases diverge from pure information sharing explains the difference between humble and hostile.
In particular, perhaps some audiences are looking to reduce cognitive dissonance between their self-image as unbiased on the one hand and their particular beliefs and preferences on the other. That leaves an opening for someone to sell reasonableness/unbiasedness self-image to people holding a given set of beliefs and preferences.
Someone making reasonable counterarguments is a threat to what you’ve offered, and in that case your job is to provide refutation, counterargument and discredit so it is easy for that person’s arguments to be dismissed (through a mixture of claimed flaws in their arguments and claimed flaws in the person promoting them). This would be a ‘hostile’ talking up.
Also, we should probably expect to find it hard to distinguish between some hostile talking ups and overconfident talking downs. If we could always distinguish, hostile talking up is a clear signal of defeat.
Having the rules in the post made me think you wanted new suggestions in this thread. The rest of the post and habryka’s comment point towards new comments in the old thread.
If you want people to update the old thread, I would either remove the rules from this post, or add a caveat like “Remember, when you go to post in that thread, you should follow the rules below”
I feel a pull towards downvoting this. I am not going to, because I think this was posted in good faith, and as you say, it’s clear a lot of time and effort has gone into these comments. That said, I’d like to unpack my reaction a bit. It may be you disagree with my take, but it may also be there’s something useful in it.
[EDIT: I should disclaim that my reaction may be biased from having recently received an aggressive comment.]
First, I should note that I don’t know why you did these edits. Did sarahconstantin ask you to? Did you think a good post was being lost behind poor presentation? Is it to spell out your other comment in more detail? Knowing the answer to this might have changed my reaction.
My most important concern is why this feedback was public. The only (charitable) reason I can think of is to give space for pushback of the kind that I am giving.
My other major concern is presentation. The sentence ‘I trust that you can see past the basic “I’m being attacked” feeling and can recognise the effort and time that has gone into the comments’ felt to me like a status move and potentially upsetting someone then asking them to say thank you.