Wei_Dai

Karma: 28,507 (LW), 375 (AF)

Wei_Dai 15 May 2019 1:54 UTC
6 points

in reply to: rohinmshah's comment on: Disincentives for participating on LW/AF

I don’t think any of those features strongly disincentivize me from participating on LW/AF; it’s more the lack of people close to my own viewpoint that disincentivizes me from participating.

I see. Hopefully the LW/AF team is following this thread and thinking about what to do, but in the meantime I encourage you to participate anyway, as it seems good to get ideas from your viewpoint “out there” even if no one is currently engaging with them in a way that you find useful.

as well as the focus on expected utility maximization with simple utility functions

I don’t think anyone talks about simple utility functions? Maybe you mean explicit utility functions?

A priori I assign somewhat high probability that I will not find useful a critical comment on my work from anyone holding that perspective, but I’ll feel obligated to reply anyway.

If this feature request of mine were implemented, you’d be able to respond to such comments with a couple of clicks. In the meantime it seems best to just not feel obligated to reply.

Wei_Dai 14 May 2019 14:36 UTC
6 points

in reply to: rohinmshah's comment on: Disincentives for participating on LW/AF

I mean, I’m not sure if an intervention is necessary—I do in fact engage with people who share my viewpoint, or at least understand it well; many of them are at CHAI. It just doesn’t happen on LW/AF.

Yeah, I figured as much, which is why I said I’d prefer having an online place for such discussions so that I would be able to listen in on these discussions. :) Another advantage is to encourage more discussions across organizations and from independent researchers, students, and others considering going into the field.

Maybe I more mean that there’s an emphasis that any particular idea must have a connection via a sequence of logical steps to a full solution to AI safety.

It’s worth noting that many MIRI researchers seem to have backed away from this (or clarified that they didn’t think this in the first place). This was pretty noticeable at the research retreat and also reflected in their recent writings. I want to note though how scary it is that almost nobody has a good idea how their current work logically connects to a full solution to AI safety.

Note that I’m not saying I disagree with all of these points; I’m trying to point at a cluster of beliefs / modes of thinking that I tend to see in people who have viewpoint X.

I’m curious what your strongest disagreements are, and what bugs you the most, as far as disincentivizing you to participate on LW/AF.

Wei_Dai 12 May 2019 22:04 UTC
8 points

in reply to: rohinmshah's comment on: Disincentives for participating on LW/AF
It sounds like you might prefer a separate place to engage more with people who already share your viewpoint. Does that seem right? I think I would prefer having something like that too if it means being able to listen in on discussions of AI safety researchers with perspectives different from myself.

I would be interested in getting a clearer picture of what you mean by “viewpoint X”, how your viewpoint differs from it, and what especially bugs you about it, but I guess it’s hard to do, or you would have done it already.

Wei_Dai 12 May 2019 20:23 UTC
5 points

in reply to: Benquo's comment on: Narcissism vs. social signalling

It seems to me like the first two stages are simple enough that Jessica’s treatment is an adequate formalization, insofar as the “market for lemons” model is well-understood. Can you say a bit more about how you’d expect additional formalization to help here?

In the original “market for lemons” game there was no signaling. Instead the possibility of “lemons” in the market just drives out “peaches” until the whole market collapses.

As I mentioned in my reply to Jessica, the actual model for stage 2 she had in mind seems more complex than any formal model in the literature that I can easily find. I was unsure from her short verbal description in the original comment what model she had in mind (in particular I wasn’t sure how to interpret “convincing lies”), and am still unsure whether the math would actually work out the way she thinks (although I grant that it seems intuitively plausible). I was also unsure whether she is assuming standard unbounded rationality or something else.

It’s in the transition from stage 2 to 3 and 4 that some modeling specific to this framework seems needed, to me.

I was confused/uncertain about stage 2 already, but sure I’d be interested in thoughts about how to model the higher stages too.

Wei_Dai 12 May 2019 20:10 UTC
6 points

in reply to: jessicata's comment on: Narcissism vs. social signalling

I would not be surprised if this model is already in the literature somewhere.

I couldn’t find one after doing a quick search. According to http://www.rasmusen.org/GI/chapters/chap11_signalling.pdf there are separate classes of Audit Games and Signaling Games in the literature. It would seem natural to combine auditing and signaling into a single model but I’m not sure anyone has done so, or how the math would work out.

Wei_Dai 12 May 2019 11:46 UTC
6 points

in reply to: jessicata's comment on: Narcissism vs. social signalling

Stage 2 signalling is this but with convincing lies, which actually are enough to convince a Bayesian evaluator (who may be aware of the adversarial dynamic, and audit sometimes).

Do you have a formal (e.g., game theoretic) model of this in mind, or see an approach to creating a formal model for it?

On the one hand, I don’t want to Goodhart on excess formality / mathematization or not take advantage of informal models where available, but on the other hand, I’m not sure if long-term intellectual progress is possible without using formal models, since informal models seem very lossy in transmission and it seems very easy to talk past each other when using informal models (e.g., two people think they’re discussing one model but actually have two different models in mind). I’m thinking of writing a Question Post about this. If the answer to the above question is “no”, would you mind if I used this as an example in my post?

“UDT2” and “against UD+ASSA”

Wei_Dai

12 May 2019 4:18 UTC

40 points

4 comments7 min readLW link

Wei_Dai 11 May 2019 17:57 UTC
5 points

on: What features of people do you know of that might predict academic success?
Giving more context might help readers to know what kind of answer you’re looking for. What kind of people are you planning to run your predictions on? High school students? College students? Graduate students? AI researchers? People in non-AI fields? What kind of interventions are you planning to do on them?

Wei_Dai 11 May 2019 17:51 UTC
9 points

in reply to: rohinmshah's comment on: Disincentives for participating on LW/AF

The LW/AF audience by and large operates under a set of assumptions about AI safety that I don’t really share. I can’t easily describe this set, but one bad way to describe it would be “the MIRI viewpoint” on AI safety.

Are you seeing this reflected in the pattern of votes (comments/posts reflecting “the MIRI viewpoint” get voted up more), pattern of posts (there’s less content about other viewpoints), or pattern of engagement (most replies you’re getting are from this viewpoint)? Please give some examples if you feel comfortable doing that.

In any case, do you think recruiting more alignment/safety researchers with other viewpoints to participate on LW/AF would be a good solution? Would you like the current audience to consider the arguments for other viewpoints more seriously? Other solutions you think are worth trying?

TurnTrout and I switched to private online messaging at one point

Yeah, I think this is probably being done less than optimally, and I’d like to see LW support or encourage this somehow. One problem with the way people are doing this currently is that the chat transcripts are typically not posted, which prevents others from following along and perhaps getting a similar level of understanding, or asking questions, or spotting errors that both sides are making, or learning discussion skills from such examples.

Wei_Dai 11 May 2019 4:10 UTC
3 points
AF
in reply to: michaelcohen's comment on: Not Deceiving the Evaluator
Thanks for the example, which is really helpful. Am I correct in thinking that in general since the agent will be doing what is optimal according to its own prior, which won’t be the same as what is optimal according to the evaluator’s prior, if the evaluator is a rational agent trying to optimize the world according to its own prior, then it would not actually reward the agent according to what this scheme specifies but instead reward the agent in such a way as to cause the agent to act according to a policy that the evaluator thinks is best? In other words the evaluator has an incentive to deceive the agent as to what its prior and/or utility function actually are?

Wei_Dai 10 May 2019 23:13 UTC
5 points

in reply to: Raemon's comment on: Disincentives for participating on LW/AF

a) Do you have a sense that these people think of LW/AF as a/the primary nexus for discussion alignment-related issues? (but didn’t either because they didn’t expect to get much benefit, or would endure too much cost)

Again I didn’t get a chance to talk much about this topic, but I would guess yes.

b) I don’t actually know if there’s any other actual locus of conversation happening anywhere other than individual private google docs, curious if you know of any such thing? (such as mailing lists. Not asking for the specific details of any such mailing list, just wanting to check if such a thing exists at all).

The only thing I personally know is a low-traffic private mailing list run by FHI which has non-FHI researchers on it but mostly consist of discussions between FHI researchers.

Wei_Dai 10 May 2019 22:56 UTC
19 points

in reply to: Raemon's comment on: Disincentives for participating on LW/AF

I definitely expect there to be lot of room for improvement here – each of the areas you point to is something we’ve talked about.

That’s good to hear.

One quick check (too late in this case, but fairly high priority to figure out IMO) is “do they even think of LW as a place they should have considered reading?”

A lot of them know of LW/AF and at least read some of the posts.

Also it’s sort of awkward to have the two places to hang out be “The AlignmentForum” and “The Alignment Subforum [of LessWrong]”

Agreed this seems really awkward/confusing, but it makes me realize we do need better ways to onboard people who are mainly interested in AI alignment as opposed to rationality and cater to their needs generally. If a new user tries to comment on a post on AF now, it just pops up a message “Log in or go to LessWrong to submit your comment.” and there’s not even a link to the same post on LW. This whole experience probably needs to be reconsidered.

Disincentives for participating on LW/AF

Wei_Dai

10 May 2019 19:46 UTC

67 points

41 comments2 min readLW link

Wei_Dai 9 May 2019 23:30 UTC
3 points
AF
in reply to: michaelcohen's comment on: Not Deceiving the Evaluator
I think I vaguely understand but it would be a lot clearer if you gave a concrete example. Also please update in the direction that people often find it hard to understand things without examples and giving examples preemptively is very cost effective in general.

Wei_Dai 9 May 2019 19:17 UTC
3 points

in reply to: Benquo's comment on: Should Effective Altruism be at war with North Korea?

I don’t think I’ve seen a single clear example of someone taking initiative (where saying something new in public based on engagement with the post’s underlying model would count as taking initiative) as a result of that post, and making different giving decisions would probably count too.

I wrote a post that was in part a response/followup to your GiveWell post although I’m not sure if you’d count that as engagement with your underlying model or just superficially engaging with the conclusions or going off on a tangent or something like that.

I think I have some general confusion about what you’re trying to do. If you think you have ideas that are good enough to, upon vetting by a wider community, potentially be basis for action for others or help change other people’s decisions, or be the basis for further thinking by others, and aren’t getting as much engagement as you hope, it seems like you should try harder to communicate your ideas clearly and to a wide audience. On the other hand if you’re still pretty confused about something and still trying to figure things out to your own satisfaction, then it would make sense to just talk with others who already share your context and not try super hard to make things clear to a wider audience. Or do you think you’ve figured some things out but it doesn’t seem cost effective to communicate to a wider audience but you might as well put them out there in a low-effort way and maybe a few readers will get your ideas.

(So one suggestion/complaint is to make clearer which type of post is which. Just throwing things out there isn’t low cost if it wastes readers’ time! Again maybe you think that should just be obvious from looking at the first few paragraphs of a post but it was not to me, in part because others like Eliezer use dialogs to write the first kind of post. In retrospect he was writing fictionalized dialogs instead of reporting actual dialogs but I think that’s why the post didn’t immediately jump out to me as “maybe this isn’t worthwhile for me to try to understand so I should stop before I invest more time/effort into it”.)

It seems like you’re saying that you rarely or never get enough engagement with the first type of writing, so you no longer think that is cost effective for you, but then what is your motivation for trying to figure these things out now? Just to guide your own actions and maybe a very small group of others? If so, what is your reason for being so pessimistic about getting your ideas into a wider audience if you tried harder? Are there not comparably complex or subtle or counterintuitive ideas that have gotten into a wider audience?

Wei_Dai 9 May 2019 10:19 UTC
12 points

in reply to: Benquo's comment on: Should Effective Altruism be at war with North Korea?

On review, there was already a summary at the beginning of the Authoritarian Empiricism post

I didn’t recognize this as a summary because it seemed to be talking about a specific “pseudo-community” and I didn’t interpret it as making a general point. Even reading it now, knowing that it’s a summary, I still can’t tell what the main point of the article might be. The beginning of Totalitarian ethical systems seems clearer as summary now that you’ve described it as such, but before that I didn’t know if it was presenting the main point or just an example of a more general point or something tangential, etc., since I didn’t understand all of the rest of the post so I couldn’t be sure the main point of the post wasn’t something different.

Also it seems like the point of a summary is to clearly communicate what the main points of the post are so the reader has some reference to know whether they should read the rest of it and also to help understand the rest of the post (since they can interpret it in relation to the main points) and having an unlabeled summary seems to defeat these purposes as the reader can’t even recognize the summary as a summary before they’ve read and understood the rest of the post.

Wei_Dai 9 May 2019 10:00 UTC
12 points

in reply to: Benquo's comment on: Should Effective Altruism be at war with North Korea?
(Sorry, due to attending a research retreat I didn’t get a chance to answer your comments until now.)

I don’t think you should care so much about engagement as opposed to communicating your ideas to your readers. I found your series on GiveWell a lot easier to understand would much prefer writings in that style.

More specific feedback would be helpful to me, like, “I started reading this article because I got the sense that it was about X, and was disappointed because it didn’t cover arguments Y and Z that I consider important.”

I started reading this post because I read some posts from you in the past that I liked (such as the GiveWell one), and on these dialog ones it was just really hard to tell what main points you’re trying to make. I questioned the NK government vs NK people thing because I at least understood that part, and didn’t realize it’s tangential.

Like, before you added a summary, this post started by talking to a friend who used “threatening” with regard to NK, without even mentioning EA, which made me think “why should I care about this?” so I tried to skim the article but that didn’t work (I found one part that seemed clear to me but that turned out to be tangential). I guess just don’t know how to read an article that doesn’t clearly at the outset say what the main points are (and therefore why I should care), and which also can’t be skimmed.

Wei_Dai 8 May 2019 17:04 UTC
6 points
AF
in reply to: rohinmshah's comment on: Not Deceiving the Evaluator
I worked out a toy example that may be helpful. Suppose the setup is that there are states labeled 0-10, actions labeled 0-10, observations labeled 0-10, initial state is 0 and each action takes the system into state with same label and agent/evaluator get observation with same label, and two equally probable utility functions: sum of state labels over time, or the negative of that.

First suppose the policy is to always do the same action, then when you sum over the two utility functions the utilities cancel out so the expected utility is 0. Now suppose the policy is to do any non-zero action (let’s say 5) at the start, and then do 0 if the agent observes a negative reward and 10 if the agent observes a positive reward. Now when you sum over the utility functions, in the negative case the utility is −5 + 0 (this policy implies that conditional on that utility function, with probability 1 the state trajectory is 0, 5, 0), in the positive case it’s 5+10 (state trajectory 0, 5, 10), so EU is .5 * −5 + .5 * 15 = 5 so this is a better policy than the first one and it should be easy to see that it’s optimal.

Hope I understood the idea correct and that helps to explain it?

Is the point you are trying to make different from the one in Learning What to Value? (Specifically, the point about observation-utility maximizers.) If so, how?

It looks closer to the Value Learning Agent in that paper to me and maybe can be considered an implementation / specific instance of that? (Although I haven’t tried to figure out whether that’s mathematically / formally the case.)

Something that confuses me is that since the evaluator sees everything the agent sees/does, it’s not clear how the agent can deceive the evaluator at all. Can someone provide an example in which the agent has an opportunity to deceive in some sense and declines to do that in the optimal policy?

Wei_Dai 6 May 2019 16:31 UTC
13 points

in reply to: Benquo's comment on: Should Effective Altruism be at war with North Korea?

It seems like you need this labeled DISCLAIMER so you can perform ACTION: CONSIDER WHETHER NOT TO READ instead of, well, parsing the information and acting based on the model it gives you.

Model building is cognitively taxing. I usually just go by the general expectation that someone wouldn’t post on LW unless they think most readers would get positive value from reading it. It seems right to disclaim this when you already have a model and your model doesn’t predict this.

Wei_Dai 6 May 2019 15:55 UTC
5 points

in reply to: Benquo's comment on: Should Effective Altruism be at war with North Korea?

but at this point I’ve basically given up on most real engagement by people I’m not in a direct dialogue with and am throwing these things up on a sort of just-in-case basis and trying not to do extra work that I don’t expect to pay off.

Thanks for the clarification, but if I had known this earlier, I probably would have invested less time/effort trying to understand these posts. Maybe you could put this disclaimer on top of your dialog posts in the future for the benefit of other readers?

Wei_Dai

“UDT2” and “against UD+ASSA”

Dis­in­cen­tives for par­ti­ci­pat­ing on LW/​AF

Disincentives for participating on LW/AF