I think my way of thinking about things is often a lot like “draw random samples,” more like drawing N random samples rather than particle filtering (I guess since we aren’t making observations as we go—if I notice an inconsistency the thing I do is more like backtrack and start over with N fresh samples having updated on the logical fact).
The main complexity feels like the thing you point out where it’s impossible to make them fully fleshed out, so you build a bunch of intuitions about what is consistent (and could be fleshed out given enough time) and then refine those intuitions only periodically when you actually try to flesh something out and see if it makes sense. And often you go even further and just talk about relationships amongst surface level features using intuitions refined from a bunch of samples.
I feel like a distinctive feature of Eliezer’s dialog w.r.t. foom / alignment difficulty is that he has a lot of views about strong regularities that should hold across all of these worlds. And then disputes about whether worlds are plausible often turn on things like “is this property of the described world likely?” which is tough because obviously everyone agrees that every particular world is unlikely. To Eliezer it seems obvious that the feature is improbable (because it was just produced by seeing where the world violated the strong regularity he believes in), whereas to the other person it just looks like one of many scenarios that is implausible only in its concrete details. And then this isn’t well-resolved by “just talk about your mainline” because the “mainline” is a distribution over worlds which are all individually improbable (for either Eliezer or for others).
This is all a bit of a guess though / rambling speculation.
I think my way of thinking about things is often a lot like “draw random samples,” more like drawing N random samples rather than particle filtering (I guess since we aren’t making observations as we go—if I notice an inconsistency the thing I do is more like backtrack and start over with N fresh samples having updated on the logical fact).
Oh whoa, you don’t remember your samples from before? [I guess I might not either, unless I’m concentrating on keeping them around or verbalized them or something; probably I do something more expert-iteration-like where I’m silently updating my generating distributions based on the samples and then resampling them in the future.]
To Eliezer it seems obvious that the feature is improbable (because it was just produced by seeing where the world violated the strong regularity he believes in), whereas to the other person it just looks like one of many scenarios that is implausible only in its concrete details. And then this isn’t well-resolved by “just talk about your mainline” because the “mainline” is a distribution over worlds which are all individually improbable (for either Eliezer or for others).
Yeah, this seems likely; this makes me more interested in the “selectively ignoring variables” hypothesis for why Eliezer running this strategy might have something that would naturally be called a mainline. [Like, it’s very easy to predict “number of apples sold = number of apples bought” whereas it’s much harder to predict the price of apples.] But maybe instead he means it in the ‘startup plan’ sense, where you do actually assign basically no probability to your mainline prediction, but still vastly more than any other prediction that’s equally conjunctive.
I think my way of thinking about things is often a lot like “draw random samples,” more like drawing N random samples rather than particle filtering (I guess since we aren’t making observations as we go—if I notice an inconsistency the thing I do is more like backtrack and start over with N fresh samples having updated on the logical fact).
The main complexity feels like the thing you point out where it’s impossible to make them fully fleshed out, so you build a bunch of intuitions about what is consistent (and could be fleshed out given enough time) and then refine those intuitions only periodically when you actually try to flesh something out and see if it makes sense. And often you go even further and just talk about relationships amongst surface level features using intuitions refined from a bunch of samples.
I feel like a distinctive feature of Eliezer’s dialog w.r.t. foom / alignment difficulty is that he has a lot of views about strong regularities that should hold across all of these worlds. And then disputes about whether worlds are plausible often turn on things like “is this property of the described world likely?” which is tough because obviously everyone agrees that every particular world is unlikely. To Eliezer it seems obvious that the feature is improbable (because it was just produced by seeing where the world violated the strong regularity he believes in), whereas to the other person it just looks like one of many scenarios that is implausible only in its concrete details. And then this isn’t well-resolved by “just talk about your mainline” because the “mainline” is a distribution over worlds which are all individually improbable (for either Eliezer or for others).
This is all a bit of a guess though / rambling speculation.
Oh whoa, you don’t remember your samples from before? [I guess I might not either, unless I’m concentrating on keeping them around or verbalized them or something; probably I do something more expert-iteration-like where I’m silently updating my generating distributions based on the samples and then resampling them in the future.]
Yeah, this seems likely; this makes me more interested in the “selectively ignoring variables” hypothesis for why Eliezer running this strategy might have something that would naturally be called a mainline. [Like, it’s very easy to predict “number of apples sold = number of apples bought” whereas it’s much harder to predict the price of apples.] But maybe instead he means it in the ‘startup plan’ sense, where you do actually assign basically no probability to your mainline prediction, but still vastly more than any other prediction that’s equally conjunctive.