There is a theory of “the path of least resistance” that implies the one should go with the flow. With that in mind, how do you continue to nurture the growth resulting from challenges? Does the rationale of the path of least resistance conflict with the challenges of life that are required for change?
Does the rationale of the path of least resistance conflict with the challenges of life that are required for change?
No:
“Life provides enough problems without us making more.”
Yes:
If you’re up to your neck in water, (maybe) it’s time to stop walking, and start swimming.
If you don’t have enough challenges you can make what you’re doing more difficult—or go find something better (harder) to do.
Imagine you are at the beach. If you swim out far enough, you can surf back in. If you get caught in a rip tide/rip current, the current may be too strong to fight.*
Metaphorically, just because you’re going in the direction of the current doesn’t mean you have to just stay afloat** - you can swim.
* - swim perpendicular to the beach, then go back in.
** This behavior also seems characteristic of something having gone wrong. (If this is the case, identifying and addressing the problem may be as important as trying to change tack—which is not a 1d move in the literal world.)
Yes. Therefore you should not do that. “Least resistance” and “going with the flow” are for those who want to remain asleep, to do nothing, to be nothing.
This hotel is 2 star at best, imagine having to pack up your stuff every time the hotel receives a new guest? I’ve decided to fix that. The hotel still has infinite rooms and guests but this time every other room is unoccupied which prepares the hotel for an infinite amount of new visitors without inconveniencing the current residence.
If the number of guests is countable (which is the usual assumption in Hilbert’s setup), then every guest will only have to travel a finite (albeit unboundedly long) distance before they reach their room.
With climate change getting worse by the day we need to switch to sustainable energy sources sooner rather than later. The new Molten salt reactors are small, clean and safe, but still carry the stigma of nuclear energy. Since these reactors (like others) can use old nuclear waste as a fuel source, I suggest we rebrand them to “Nuclear Waste Eaters” and give them (or a company that makes them) a logo in the vein of this quick sketch I made: https://postimg.cc/jWy3PtjJ
Hopefully a rebranding to “thing getting rid of the thing you hate, also did you know it’s clean and safe” will get people more motivated for these kinds of energy sources.
Anything sufficiently far enough away from you is causally isolated from you. Because of the fundamental constraints of physics, information from there can never reach here, and vice versa. you may as well be in separate universes.
The performance of AlphaGo got me thinking about algorithms we can’t access. In the case of AlphaGo, we implemented the algorithm (AlphaGo) which discovered some strategies we could never have created. (Go Master Ke Jie famously said “I would go as far as to say not a single human has touched the edge of the truth of Go.”)
Perhaps we can imagine a sort of “logical causal isolation.” An algorithm is logically causally isolated from us if we cannot discover it (e.g. in the case of the Go strategies that AlphaGo used) and we cannot specify an algorithm to discover it (except by random accident) given finite computation over a finite time horizon (i.e. in the lifetime of the observable universe).
Importantly, we can devise algorithms which search the entire space of algorithms (e.g. generate all permutations all possible strings of bits less than length n as n approaches infinity), but there’s little reason to expect that such a strategy will result in any useful outputs of some finite length (there appear to be enough atoms in the universe (1080) to represent all possible algorithms of lengthlog2(1080)≈265.
There’s one important weakness in LCI (that doesn’t exist in Physical Causal Isolation). We can randomly jump to algorithms of arbitrary lengths. This stipulation gives us the weird ability to pull stuff from outside our LCI-cone into it. Unfortunately, we cannot do so with the expectation of arriving at a useful algorithm. (There’s an interesting question about which I haven’t yet thought about the distribution of useful algorithms of a given length.) Hence we must add the caveat to our definition of LCI “except by random accident.”
We aren’t LCI’d from the strategies AlphaGo used, because we created AlphaGo and AlphaGo discovered those strategies (even if human Go masters may never have discovered them independently). I wonder what algorithms exist beyond not just our horizons, but the horizons of all the algorithms which descend from everything we are able to compute.
2 things necessary for an algorithm to be useful:
If it’s not fast enough, it doesn’t matter how good it is
If we don’t know what it’s good for, it doesn’t matter how good it is (until we figure that out)
Unfortunately, we cannot do so with the expectation of arriving at a useful algorithm.
Part of the issue with this might be programs that don’t work or do anything (Beyond the trivial, it’s not clear how to select for this, outside of something like AlphaGo.)
If it’s not fast enough, it doesn’t matter how good it is
Sure! My brute-force bitwise algorithm generator won’t be fast enough to generate any algorithm of length 300 bits, and our universe probably can’t support any representation of any algorithm of length greater than (the number of atoms in the observable universe) ~ 10^82 bits. (I don’t know much about physics, so this could be very wrong, but think of it as a useful bound. If there’s a better one (e.g. number of Planck volumes in the observable universe), substitute that and carry on, and also please let me know!)
Part of the issue with this might be programs that don’t work or do anything (Beyond the trivial, it’s not clear how to select for this, outside of something like AlphaGo.)
Another class of algorithms that cause problems are those that don’t do anything useful for some number of computations, after which they begin to output something useful. We don’t really get to know if they will halt, so if the useful structure emerges after some number of steps, we may not be committed to or able to run it that long.
I’m not a physicist either, but quantum mechanics might change the limits. (If it scales, though this might leave input and output limits; if the quantum computer can’t store the output in classical mode, then it’s ability to run the program probably doesn’t matter. This might make less efficient crypto systems more secure, by virtue of size.*)
You would hope that people actually saw steelmanning as an ideal to follow. If that was ever true, the corona pandemic and the policy response seem to have killed the demand for this. It seems to become acceptable to attribute just any kind of seemingly-wrong behavior to either incredible stupidity or incredible malice, both proving that all institutions are completely broken.
For the foreseeable future, it seems that anything I might try to say to my UK friends about anything to do with LW-style thinking is going to be met with “but Dominic Cummings”. Three separate instances of this in just the last few days.
Basic skills of decision making under uncertainty have been sorely lacking in this crisis. Oxford University’s Future of Humanity Institute is building up its Epidemic Forecasting project, and needs a project manager.
Response:
I’m honestly struggling with a polite response to this. Here in the UK, Dominic Cummings has tried a Less Wrong approach to policy making, and our death rate is terrible. This idea that a solution will somehow spring from left-field maverick thinking is actually lethal.
Seems like a good discussion could be had about long-term predictions and how much evidence there is to be had in short-term political fluctuations. The Cummings silliness vs unprecedented immigration restrictions—which is likely to have impact 5 years from now?
Surfing Uncertainty is about predictive coding, the theory in neuroscience that each part of your brain attempts to predict its own inputs. Predictive coding has lots of potential consequences. It could resolve the problem of top-down vs bottom-up processing. It cleanly unifies lots of ideas in psychology. It even has implications for the continuum with autism on one end and schizophrenia on the other.
The most promising thing about predictive coding is how it could provide a mathematical formulation for how the human brain works. Mathematical formulations are great because once they let you do things like falsify experiments and simulate things on computers. But while Surfing Uncertainty goes into many of the potential implications of predictive codings, the author never hammers out exactly what “prediction error” means in quantifiable material terms on the neuronal level.
This book is a reiteration of the scientific consensus[1]. Judging by the total absense of mathematical equations on the Wikipedia page for predictive coding, I suspect the book never defines “prediction error” in mathematically precise terms because no such definition exists. There is no scientific consensus.
Perhaps I was disappointed with this book because my expectations were too high. If we could write equations for how the human brain performs predictive processing then we would be significantly closer to building an AGI than where we are right now.
The book contains 47 pages of scientific citations. ↩︎
Has anyone tried to work with neural networks predicting the weights of other neural networks? I’m thinking about that in the context of something like subsystem alignment, e.g. in an RL setting where an agent first learns about the environment, and then creates the subagent (by outputting the weights or some embedding of its policy) who actually obtains some reward
Observation: It should generally be safe to forbid non-termination when searching for programs/algorithms.
In practice, all useful algorithms terminate: If you know that you’re dealing with a semi-decidable thing and doing serious work, you’ll either (a) add a hard cutoff, or (b) structure the algorithm into a bounded step function and a controller that decides whether or not to run for another step. That transformation is not adding significant overhead size-wise, so you’re bound to find a terminating algorithm “near” a non-terminating one!
Sure, that slightly changes the interface – it’s now allowed to abort with “don’t know”, but that’s a transformation that you likely would have applied anyway. Even if you consider that a drawback, not having to deal with potentially non-terminating programs / being able to use a description format that cannot represent non-terminating forms should more than make up for that.
(I just noticed this while thinking about how to best write something in Coq (and deciding on termination by “fuel limit”), after AABoyles’ shortform on logical causal isolation with its tragically simple bit-flip search had recently made me think about program enumeration again…)
I had a very useful conversation with someone about how and why I am rambly. (I rambled a lot in the conversation!).
Disclaimer: I am not making much effort to not ramble in this post.
A couple takeaways:
1. Working Memory Limits
One key problem is that I introduce so many points, subpoints, and subthreads, that I overwhelm people’s working memory (where human working memory limits is roughly “4-7 chunks”).
It’s sort of embarrassing that I didn’t concretely think about this before, because I’ve spent the past year SPECIFICALLY thinking about working memory limits, and how they are the key bottleneck on intellectual progress.
So, one new habit I have is “whenever I’ve introduced more than 6 points to keep track of, stop and and figure out how to condense the working tree of points down to <4.
(Ideally, I also keep track of this in advance and word things more simply, or give better signposting for what overall point I’m going to make, or why I’m talking about the things I’m talking about)
...
2. I just don’t finish sente
I frequently don’t finish sentences, whether in person voice or in text (like emails). I’ve known this for awhile, although I kinda forgot recently. I switch abruptly to a new sentence when I realize the current sentence isn’t going to accomplish the thing I want, and I have a Much Shinier Sentence Over Here that seems much more promising.
But, people don’t understand why I’m making the leap from one half-finished thought to another.
So, another simple habit is “make sure to finish my god damn sentences, even if I become disappointed in them halfway through”
...
3. Use Mindful Cognition Tuning to train on *what is easy for people to follow*, as well as to improve the creativity/usefulness of my thoughts.
I’ve always been rambly. But a thing that I think has made me EVEN MORE rambly in the past 2 years is a mindful-thinking-technique, where you notice all of your thoughts on the less-than-a-second level, so that you can notice which thought patterns are useful or anti-useful.
This has been really powerful for improving my thought-quality. I’m fairly confident that I’ve become a better programmer and better thinker because of it.
But, it introduces even more meta-thoughts for me to notice while I’m articulating a sentence, which distract me from the sentence itself.
What I realized last weekend was: I can use Mindful Cognition to notice what types of thoughts/sentences are useful for *other people’s comprehension of me*, not just how useful m original thought processes are.
The whole point of the technique is to improve your feedback loop (both speed and awareness), which makes it easier to deliberate practice. I think if I just apply that towards Being More Comprehensible, it’ll change from being a liability in rambliness to an asset.
re working memory: never thought of it during conversations, interesting. it seems that we sometime hold the nodes of the conversation tree to go back to them afterward. and maybe if you’re introducing new concepts while you’re talking people need to hold those definitions in working memory as well.
This specific situation looks unrealistic. But it’s not really trying to be too realistic, it’s trying to be a counterexample. In that spirit, you could also just use R2(a,sd)=1000, which is a reward function parametrized by θ that gives the same behavior but stops me from saying “Why Not Just set θ=−1”, which isn’t the point.
How something like this might actually happen: you try to have your R1 be a complicated neural network that can approximate any function. But you butcher the implementation and get something basically random instead, and this R2 cannot approximate the real human reward.
An important insight this highlights well: An off-switch is something that you press only when you’ve programmed the AI badly enough that you need to press the off-switch. But if you’ve programmed it wrong, you don’t know what it’s going to do, including, possibly, its off-switch behavior. Make sure you know under which assumptions your off-switch will still work!
Assigning high value to shutting down is incorrigible, because the AI shuts itself down. What about assigning high value to being in a button state?
The paper considers a situation where the shutdown button is hardcoded, which isn’t enough by itself. What’s really happening is that the human either wants or doesn’t want the AI to shut down, which sounds like a term in the human reward that the AI can learn.
One way to do this is for the AI to do maximum likelihood with a prior that assigns 0 probability to the human erroneously giving the shutdown command. I suspect there’s something less hacky related to setting an appropriate prior over the reward assigned to shutting down.
The footnote on page 7 confuses me a bit—don’t you want the AI to always defer to the human in button states? The answer feels like it will be clearer to me if I look into how “expected reward if the button state isn’t avoided” is calculated.
Also I did just jump into this paper. There are probably lots of interesting things that people have said about MDPs and CIRLs and Q-values that would be useful.
I don’t think it’s quite right to call this an off-switch—the model is fully general to the situation where the AI is choosing between two alternatives A and B (normalized in the paper so that U(B) = 0), and to me an off-switch is a hardware override that the AI need not want you to press.
The wisdom to take away from the paper: An AI will voluntarily defer to a human—in the sense that the AI thinks that it can get a better outcome by its own standards if it does what the human says—if it’s uncertain about the utilities, or if the human is rational.
This whole setup seems to be somewhat superseded by CIRL, which has the AI, uh, causally find UA by learning its value from the human actions, instead of evidentially(?) doing it by taking decisions that happen to land it on action A when UA is high because it’s acting in a weird environment where a human is present as a side-constraint.
Could some wisdom to gain be that the high-variance high-human-rationality is something of an explanation as to why CIRL works? I should read more about CIRL to see if this is needed or helpful and to compare and contrast etc.
Why does the reward gained drop when uncertainty is too high? Because the prior that the AI gets from estimating the human reward is more accurate than the human decisions, so in too-high-uncertainty situations it keeps mistakenly deferring to the flawed human who tells it to take the worse action more often?
The verbal description, that the human just types in a noisily sampled value of UA, is somewhat strange—if the human has explicit access to their own utility function, they can just take the best actions directly! In practice, though, the AI would learn this by looking at many past human actions (there’s some CIRL!) which does seem like it plausibly gives a more accurate policy than the human’s (ht Should Robots Be Obedient).
The human is Boltzmann-rational in the two-action situation (hence the sigmoid). I assume that it’s the same for the multi-action situation, though this isn’t stated. How much does the exact way in which the human is irrational matter for their results?
When I read Partial Agency, I was struck with a desire to try formalizing this partial agency thing. Defining Myopia seems like it might have a definition of myopia; one day I might look at it. Anyway,
Formalization of Partial Agency: Try One
A myopic agent is optimizing a reward function R(x0,y(x0)) where x is the vector of parameters it’s thinking about and y is the vector of parameters it isn’t thinking about. The gradient descent step picks the δx in the direction that maximizes R(x0+δx,y(x0)) (it is myopic so it can’t consider the effects on y), and then moves the agent to the point (x0+δx,y(x0+δx)).
This is dual to a stop-gradient agent, which picks the δx in the direction that maximizes f(x0+δx,y(x0+δx)) but then moves the agent to the point (x0+δx,y(x0)) (the gradient through y is stopped).
For example,
Nash equilibria - x are the parameters defining the agent’s behavior.y(x0) are the parameters of the other agents if they go up against the agent parametrized by x0. R is the reward given for an agent x going up against a set of agents y.
Image recognition with a neural network - x is the parameters defining the network, y(x0) are the image classifications for every image in the dataset for the network with parameters x0, and R is the loss function plus the loss of the network described by x on classifying the current training example.
Episodic agent - x are parameters describing the agents behavior.y(x0) are the performances of the agent x0 in future episodes.R is the sum of y, plus the reward obtained in the current episode.
Partial Agency due to Uncertainty?
Is it possible to cast partial agency in terms of uncertainty over reward functions? One reason I’d be myopic is if I didn’t believe that I could, in expectation, improve some part of the reward, perhaps because it’s intractable to calculate (behavior of other agents) or something I’m not programmed to care about (reward in other episodes).
Let R1 be drawn from a probability distribution over reward functions. Then one could decompose the true, uncertain, reward into R′=R0(x0)+R1(x0) defined in such a way that E(R1(x0+δx)−R1(x0))≈0 for any δx? Then this is would be myopia where the agent either doesn’t know or doesn’t care about R1, or at least doesn’t know or care what its output does to R1. This seems sufficient, but not necessary.
Now I have two things that might describe myopia, so let’s use both of them at once! Since you only end up doing gradient descent on R0, it would make sense to say R′(x)=R(x,y(x)) , R0(x)=R(x,y(x0)) , and hence that R1(x)=R(x,y(x))−R(x,y(x0)).
Since R1(x0+δx)=R1(x0)+δx∂R1∂x for small δx, this means that E(∂R1∂x)=0 , so substituting in my expression for R1 gives E(∂R∂x+∂R∂y∂y∂x−∂R∂x)=0 , so E(∂R∂y∂y∂x)=0 . Uncertainly is only over R, so this is just the claim that the agent will be myopic with respect to y if E(∂R∂y)=0. So it won’t want to include y in its gradient calculation if it thinks the gradients with respect to y are, on average, 0. Well, at least I didn’t derive something obviously false!
But Wait There’s More
When writing the examples for the gradient descenty formalisation, something struck me: it seems there’s a R(x)=r(x)+∑iyi(x) structure to a lot of them, where r is the reward on the current episode, and yi are rewards obtained on future episodes.
You could maybe even use this to have soft episode boundaries, like say the agent receives a reward rt on each timestep so R(x)=r0(x)+r1(x)α+r2(x)α2+∑i=3ri(x)αi , and saying that α3≪1 so that ∂R∂ri≪1 for i≥3, which is basically the criterion for myopia up above.
Unrelated Note
On a completely unrelated note, I read the Parable of Predict-O-Matic in the past, but foolishly neglected to read Partial Agency beforehand. The only thing that I took away from PoPOM the first time around was the bit about inner optimisers, coincidentally the only concept introduced that I had been thinking about beforehand. I should have read the manga before I watched the anime.
So the definition of myopia given in Defining Myopia was quite similar to my expansion in the But Wait There’s More section; you can roughly match them up by saying r(x)=∑ifiri(x) and yi(x)=(1−fi)ri(x) , where fi is a real number corresponding to the amount that the agent cares about rewards obtained in episode i and ri is the reward obtained in episode i. Putting both of these into the sum gives R(x)=∑iri(x), the undiscounted, non-myopic reward that the agent eventually obtains.
In terms of the R=R0+R1 definition that I give in the uncertainty framing, this is R0=R(x,y0)=∑ifiri(x)+∑i(1−fi)ri(x0), and R1=R(x,y)−R(x,y0)=∑i(1−fi)(ri(x)−ri(x0)).
So if you let r be a vector of the reward obtained on each step and f be a vector of how much the agent cares about each step then x→x+ϵ∑ifi∂ri∂x , and thus the change to the overall reward is R→R+ϵ∑i∂ri∂x∑jfj∂rj∂x , which can be negative if the two sums have different signs.
I was hoping that a point would reveal itself to me about now but I’ll have to get back to you on that one.
It’s a crowd I’d come into contact with as a manager of an online bookshop (and most of the reason I quitted). Usually, I can pretend they don’t exist, but… we all know how it goes… and now that they don’t make my blood boil every weekend, I can afford to speak about them.
“Some other people” will play to win—say, a facebook lottery with a book for a prize, and they will mean it. If they don’t win, they will say the lottery was rigged. Public righteous indignation on every player’s behalf is a weapon (and for the manager, a potent vaccine against righteously indignant polemics of many other kinds). Private appeals to the manager’s pity; commenting the rules’ exploitable/exploited loopholes—after the winner is announced; repeating actions which have already been answered elsewhere in the thread. I don’t include ‘filing a complaint’ here, because it’s frankly too straightforward for most of them, most of the time; the bookshop would likely send them a book with an eloquent blessing/apology, just to get them to shut up and earn good PR points for “owning up to mistakes”. But in practice, it still matters too much to be the actual winner, and the brain of the trophy-gatherer works like other brains don’t. At least not for a while.
I’m not unusually out-of-touch with customers; I was recommended for the job after two years in an offline shop. And this was… entirely different. I’d never encountered people with whole profiles dedicated to reposting online lotteries—living people I had to call on the phone. It is another world.
When I read about (simple) “pure” game theoretical problems, in which the players “care only about winning”, I cannot reconcile the image of Worthy Rivals the author has in mind with the actual Really-Want-This-Whatever Whiners who seek out such contests. Get it, not the passively allowing themselves to be drawn into a strategic game kind of players, but the self-sorting to exploit as many offers as possible kind. They will be few, yes. Nobody of them might force their way through every single time.
But they will define the meaning of the rules you think you write.
Not any particular book, but rather some frequent conditions of game theory problems I have seen here and elsewhere (my fb friend keeps posting such pieces). “The players care only about winning” etc. Well, some people actually do.
There are two kinds of pleasurable feelings. The first one is a self-reinforcing loop, where the in-the-moment pleasure leads to craving for more pleasure, such as mindlessly scrolling through social media, or eating highly-processed, highly-palatable food. The second is pleasure gained through either thoughtfully consuming good content, like listening to good music or reading good books, or the fulfillment of a task that’s meaningful, such as getting good grades or getting a promotion for sustained conscientious effort.
The first is pleasure for its own sake, without any “real world rewards” that come with it, ie, distractions. The second isn’t as “addictive” as the first, nor does it cause the same spikes in pleasure, but it comes with real world tangible rewards.
There is no way to completely eliminate the human need for the first pleasure. But the need can be reduced. The ratio of second:first pleasure, is the degree to which a person is able to achieve his goals, the degree to which a person is successful.
There’s a large piece missing from your model—why seek pleasure at all? Wouldn’t it be better to measure success directly based on meeting long-term goals (clippy interjects from the balcony: like number of paperclips created!).
As an add-on, I found this LW article today that captures the essence of “first pleasure”
clippy interjects from the balcony: like number of paperclips created!
I’m not quite sure I got this part, could you please elaborate on it?
why seek pleasure at all? Wouldn’t it be better to measure success directly based on meeting long-term goals
Here, I would argue that the feeling of the second pleasure is essential to meeting long-term goals. Feeling good about accomplishing sub-tasks will keep someone working towards an important goal, especially if it requires a long period of sustained effort.
Thus, meeting short-term goal → success → second pleasure → working on next short-term goal; with enough iterations, that will lead to the meeting of long-term goals and thus success.
https://wiki.lesswrong.com/wiki/Paperclip_maximizer is the canonical example of over-simplified goal optimization. I bring it up mostly as a reminder that getting your motivational model wrong can lead to undesirable actions and results.
Which leads to my main point. You’re recommending one type of pleasure over another, based on it being more aligned with your non-pleasure-measured goals. I’m wondering why you are arguing for this, as opposed to just pursuing the goals directly, without consideration of pleasure.
Ah, now I’ve got what you mean. Thanks for referring me to that thought experiment, I don’t have much prior knowledge on the field of AI so that was definitely a new insight for me.
I see now that my original shortform did not explicitly state that my terminal value was indeed the fulfillment of important goals. I was reflecting more on the distinction between pleasurable feelings that led to distraction & bad habits, vs ones that led to the actual fulfillment of goals. It was a personal reminder to experience the latter in place of the former, as much as I can.
Now, I hold the view that pleasure can be a useful tool in the pursuit of my goals. It is a means to that end. An important caveat that your response reminded me of, though, is that sometimes pursuing goals might not be immediately pleasurable, so it might be wise not to naïvely expect pleasure from every part of that process.
Sentences spoken aloud are a latent space embedding of our thoughts; when trying to move a thought from our mind to another’s, our thoughts are encoded with the aim of minimizing the other person’s decoder error.
There’s a problem at parties where there’ll be a good, high-context conversation happening, and then one-too-many-people join, and then the conversation suddenly dies.
Sometimes this is fine, but other times it’s quite sad.
Things I think might help:
If you’re an existing conversation participant:
Actively try to keep the conversation small. The upper limit is 5, 3-4 is better. If someone looks like they want to join, smile warmly and say “hey, sorry we’re kinda in a high context conversation right now. Listening is fine but probably don’t join.”
If you do want to let a newcomer join in, don’t try to get them up to speed (I don’t know if I’ve ever seen that actually work). Instead, say “this is high context so we’re not gonna repeat the earlier bits, maybe wait to join in until you’ve listened enough to understand the overall context”, and then quickly get back to the conversation before you lose the Flow.
If you want to join a conversation:
If there are already 5 people, sorry, it’s probably too late. Listen if you find it interesting, but if you actively join you’ll probably just kill the conversation.
Give them the opportunity to gracefully keep the conversation small if they choose. (say something like “hey can I join? It sounds like maybe a high context conversation, no worries if you wanna keep it small.”)
Listen for longer before joining. Don’t just wait till you understand the current topic – try to understand the overall vibe, and what previous topics might be informing the current one. Try to get a sense of what each current participant is getting out the conversation. When you do join, do so in a small way that gives them affordance to shift back to an earlier topic if your new contribution turned out to be not-actually-on-topic.
I hosted an online-party using zoom breakout rooms a few weeks ago and ran into similar problems.
Half-way through the party I noticed people were clustering in suboptimal size conversations and bringing high-context conversations to a stop, so I actually brought everybody backed to the lobby then randomly assigned them to groups of 2 or 3 - and when I checked 10 minutes later everyone was in the same two rooms again with groups of 8 − 10 people.
AFAICT this was status/feelings driven—there were a few people at the party who were either existing high-status to the participants, or who were very charismatic, and everyone wanted to be in the same conversation as them.
I think norm-setting around this is very hard, because it’s natural to want to be around high-status and charismatic people, and it’s also natural to want to participate in a conversation you’re listening to.
I’m going to try to add your suggestions to the top of the shared google doc next time I host one of these and see how it goes.
Agreed with the status/feelings cause. And I’m not 100% sure the solution is “prevent people from doing the thing they instinctively want to do” (especially “all the time.”)
My current guess is “let people crowd around the charismatic/and/or/interesting people, but treat it more like a panel discussion or fireside chat, like you might have at a conference, where mostly 2-3 people are talking and everyone else is more formally ‘audience.’”
But doing that all the time would also be kinda bad in different ways.
In this case… you might actually be able to fix this with technology? Can you literally put room-caps on the rooms, so if someone wants to be the 4th or 6th person in a room they… just… can’t?
physically separate the group. Go into another room or at least corner. Signal that you’re not seeking additional participants.
When you notice this, make it explicit—“I’m really enjoying the depth of this conversation, should we move into the lounge for a brandy and a little more quiet?”
Admit (to yourself) that others may feel excluded, because they are. At many gatherings, such discussions/situations are time-bound and really can’t last more than 10-45 minutes. The only solution is to have more frequent, smaller gatherings.
Get good at involved listening—it’s different than 1:1 active listening, but has similar goals: don’t inject any ideas, but do give signals that you’re following and supporting. This is at least 80% as enjoyable as active participation, and doesn’t break the flow when you join a clique in progress.
I wonder what analogs there are to online conversations. I suspect there’s a lot of similarity for synchronous chats—too many people make it impossible to follow. For threaded, async discussions, the limits are probably much larger.
I wonder what analogs there are to online conversations. I suspect there’s a lot of similarity for synchronous chats—too many people make it impossible to follow. For threaded, async discussions, the limits are probably much larger.
FYI, the actual motivating example here was at a party in gather.town, (formerly online.town, formerly town.siempre), which has much more typical “party” dynamics. (i.e people can wander around an online world and video chat with people nearby).
In this case there were actually some additional complexities – I had joined a conversation relatively late, I did lurk for quite awhile, and wait for the current set of topics to die down completely before introducing a new one. And then the conversation took a turn that I was really excited by, and at least 1-2 other people were interested in, but it wasn’t obvious to me that it was interesting to everyone else (I think ~5 people involved total?)
And then a new person came in, and asked what we were talking about and someone filled them in… …and then immediately the conversation ended. And in this case I don’t know if the issue was more like “the newcomer killed the conversation” or “the convo actually had roughly reached it’s natural end, and/or other people weren’t that interested in the first place.”
But, from my own perspective, the conversation had just finished covering all the obvious background concepts that would be required for the “real” conversation to begin, and I was hoping to actually Make Real Progress on a complex concept.
So, I dunno if this counted as “an interesting conversation” yet, and unfortunately the act of asking the question “hey, do we want to continue diving deep into this, or wrap up and transition into some other convo?” also kinda kills the conversation. Conversations are so god damn fragile.
What I really wished was that everyone already had common knowledge of the meta-concept, wherein:
Party conversations are particularly fragile
Bringing a newcomer up to speed is usually costly if the conversation is doing anything deep
We might or might not want to continue delving into the current convo (but we don’t currently have common knowledge of this in either direction)
And if everyone (newcomer included) had those concepts, and new everyone had those concepts, then I feel like I could have asked more gracefully “hey, I’m kinda interested in continuing to hash out some ideas here. Are people up for taking this high context”, and had people give their honest answer.
[EDIT, was intended as a response to Raemon, not Dagon.]
Maybe it’s the way you phrase the responses. But as described, I get the impression that this norm would mainly work for relatively extroverted persons with low rejection sensitivity.
I’d be much less likely to ever try to join a discussion (and would tend to not attend events with such a norm). But maybe there’s a way to avoid this, both from “my side” and “yours”.
Hmm, seems like important feedback. I had specifically been trying to phrase the responses in a way that addressed this specific problem. Sounds like it didn’t work.
There is some intrinsic rejection going on here, which probably no amount of kind wording can alleviate for a rejection-sensitive person.
For my “sorry, we’re keeping the convo small” bit, I suggested:
smile warmly and say “hey, sorry we’re kinda in a high context conversation right now. Listening is fine but probably don’t join.”
The Smile Warmly part was meant to be a pretty active ingredient, helping to reassure them it isn’t personal.
Another thing that seems pretty important, is that this applies to all newcomers, even your friends and High Status People. (i.e. hopefully if Anxious Alex gets turned away, but later sees High Status Bob also get turned away, they get reassured a bit that this wasn’t about them)
I’m looking for an old post where Eliezer makes the basic point that we should be able to do better than intellectual figures of the past, because we have the “unfair” advantage of knowing all the scientific results that have been discovered since then.
I think he cites in particular the heuristics and biases literature as something that thinkers wouldn’t have known about 100 years ago.
I don’t remember if this was the main point of the post it was in, or just an aside, but I’m pretty confident he made a point like this at least once, and in particular commented on how the advantage we have is “unfair” or something like that, so that we shouldn’t feel at all sheepish about declaring old thinkers wrong.
Max Gluckman once said: “A science is any discipline in which the fool of this generation can go beyond the point reached by the genius of the last generation.” Science moves forward by slaying its heroes, as Newton fell to Einstein. Every young physicist dreams of being the new champion that future physicists will dream of dethroning.
Ayn Rand’s philosophical idol was Aristotle. Now maybe Aristotle was a hot young math talent 2350 years ago, but math has made noticeable progress since his day. Bayesian probability theory is the quantitative logic of which Aristotle’s qualitative logic is a special case; but there’s no sign that Ayn Rand knew about Bayesian probability theory when she wrote her magnum opus, Atlas Shrugged. Rand wrote about “rationality”, yet failed to familiarize herself with the modern research in heuristics and biases. How can anyone claim to be a master rationalist, yet know nothing of such elementary subjects?
(Not sure if the material is accurate, but I think it’s the post you’re looking for. There could have been more than one on that though.)
Weird thing I wish existed: I wish there were more videos of what I think of as ‘math/programming speedruns’. For those familiar with speedrunning video games, this would be similar except the idea would be to do the same thing for a math proof or programming problem. While it might seem like this would be quite boring since the solution to the problem/proof is known, I still think there’s an element of skill to and would enjoy watching someone do everything they can to get to a solution, proof, etc. as quickly as possible (in an editor, on paper, LaTex, etc.).
This is kind of similar to streaming ACM/math olympiad competition solving except I’m equally more in people doing this for known problems/proofs than I am for tricky but obscure problems. E.g., speed-running the SVD derivation.
While I’m posting this in the hope that others are also really interested, my sense is that this would be incredibly niche even amongst people who like math so I’m not surprised it doesn’t exist...
Last Thursday on the Discord we had people any% speedrunning and racing the Lean tutorial project . This fits very well into my general worldview: I think that doing mathematics in Lean is like solving levels in a computer puzzle game, the exciting thing being that mathematics is so rich that there are many many kinds of puzzles which you can solve.
This is awesome! I’ve been thinking I should try out the natural number game for a while because I feel like formal theorem proving will scratch my coding / video game itch in a way normal math doesn’t.
Serious in that I mean he might, I’d say, 0.1 that he’d be interested, but if that’s not negligible, I think if he took it up he’d be very good at it. I’ll ask him.
I had a similar idea which was also based on an analogy with video games (where the analogy came from let’s play videos rather than speedruns), and called it a live math video.
Cool, I hadn’t seen your page previously but our ideas do in fact seem very similar. I think you were right to not focus on the speed element and instead analogize to ‘let’s play’ videos.
I’m not super familiar with the competitive math circuit, but my understanding is that this is part of it? People are given a hard problem and either individually or as a team solve it as quickly as possible.
Yep, I touched on this above. Personally, I’m less interested in this type of problem solving than I am in seeing someone build to a well-known but potentially easier to prove theorem, but I suspect people solving IMO problems would appeal to a wider audience.
3Blue1Brown has a video where he sort of does this for a hard Putnam problem. I say “sort of” because he’s not solving the problem in real time so much as retrospectively describing how one might solve it.
The problem with this is that it is very difficult to figure out what counts as a legitimate proof. What level of rigor is required, exactly? Are they allowed to memorize a proof beforehand? If not, how much are they allowed to know?
Solutions might be better to go with than proofs—if the answer is wrong, that’s more straightforward to show that whether or not a proof is wrong.
Yeah what would be ideal is if theorem provers were more usable and then this wouldn’t be an issue (although of course there’s still the issue of library code vs. from scratch code but this seems easier to deal with).
Memorizing a proof seems fine (in the same way that I assume you end up basically memorizing the game map if you do a speedrun).
When I was in high school, I once had a conversation with a classmate that went something like this (except that it was longer and I was less eloquent):
Him: “German is a Scandinavian language.”
Me: “No, it’s not. German and the Scandinavian languages both fall under the umbrella of Germanic languages, but ‘Scandinavian languages’ refers to a narrower category that doesn’t include German.”
Him: “Well that’s your opinion.”
Me: “No??? That’s not what an opinion is???”
Him: “Look, it’s your opinion that German isn’t a Scandinavian language, and it’s my opinion that it is. We can agree to disagree.”
Me: ??????????????????!!!!!!!!!????!?!?!?!?! *punches self in face*
----
When I was taking a required intro biology course in college, I had already read a bunch of LW and SSC, notably including That Chocolate Study. So when the professor put Bohannon’s results and methodology up on the projector, I was ready as heck to talk about all of the atrocities therein. The professor asked us to pair up with the person next to us to discuss whether we believed Bohannon’s results, and I decided to give the freshman next to me the chance to speak first before I absolutelydemolished everything. The girl turned to me with wide eyes and a confident, creaky-voice drawl, and said, verbatim: “I think it’s true, because chocolate is known to be a superfood.”
I was floored. How could this be happening in real life? I was at an elite college with a sub-10% acceptance rate, and this person next to me had just said “known to be” and “superfood” like they explained anything – like they meant anything. I will never forget those words. Looking back, that may have been the day I decided to move to the Bay after graduating. No regrets.
I’ve been thinking about people’s mindset as it relates to spending their free time. Specifically, when you go to do something ‘productive’ like learn about a new topic, work through exercises in a textbook, go through an online course, etc...do you feel that you have to intentionally decide not to play video games, watch Netflix, etc and forego short-term happiness? Or do you feel that this decision is straightforward because that’s what you would prefer to be doing and you don’t feel like you sacrifice anything?
There is a theory of “the path of least resistance” that implies the one should go with the flow. With that in mind, how do you continue to nurture the growth resulting from challenges? Does the rationale of the path of least resistance conflict with the challenges of life that are required for change?
Does the rationale of the path of least resistance conflict with the challenges of life that are required for change?
No:
“Life provides enough problems without us making more.”
Yes:
If you’re up to your neck in water, (maybe) it’s time to stop walking, and start swimming.
If you don’t have enough challenges you can make what you’re doing more difficult—or go find something better (harder) to do.
Imagine you are at the beach. If you swim out far enough, you can surf back in. If you get caught in a rip tide/rip current, the current may be too strong to fight.*
Metaphorically, just because you’re going in the direction of the current doesn’t mean you have to just stay afloat** - you can swim.
* - swim perpendicular to the beach, then go back in.
** This behavior also seems characteristic of something having gone wrong. (If this is the case, identifying and addressing the problem may be as important as trying to change tack—which is not a 1d move in the literal world.)
Yes. Therefore you should not do that. “Least resistance” and “going with the flow” are for those who want to remain asleep, to do nothing, to be nothing.
Would you say the same of “the path” in emotions and relationships?
Hilbert’s Motel improvement
This hotel is 2 star at best, imagine having to pack up your stuff every time the hotel receives a new guest? I’ve decided to fix that. The hotel still has infinite rooms and guests but this time every other room is unoccupied which prepares the hotel for an infinite amount of new visitors without inconveniencing the current residence.
But isn’t the whole point that the hotel is full initially, and yet can accept more guests?
Yeah, the hotel being always half full no matter how many guests it has doesn’t seem as cool.
As soon as one more guest shows up it’s more than half full.
But if an infinite number of guests show up, then they might have to travel an infinite distance to their hotel room.
If the number of guests is countable (which is the usual assumption in Hilbert’s setup), then every guest will only have to travel a finite (albeit unboundedly long) distance before they reach their room.
With climate change getting worse by the day we need to switch to sustainable energy sources sooner rather than later. The new Molten salt reactors are small, clean and safe, but still carry the stigma of nuclear energy. Since these reactors (like others) can use old nuclear waste as a fuel source, I suggest we rebrand them to “Nuclear Waste Eaters” and give them (or a company that makes them) a logo in the vein of this quick sketch I made: https://postimg.cc/jWy3PtjJ
Hopefully a rebranding to “thing getting rid of the thing you hate, also did you know it’s clean and safe” will get people more motivated for these kinds of energy sources.
Anything sufficiently far enough away from you is causally isolated from you. Because of the fundamental constraints of physics, information from there can never reach here, and vice versa. you may as well be in separate universes.
The performance of AlphaGo got me thinking about algorithms we can’t access. In the case of AlphaGo, we implemented the algorithm (AlphaGo) which discovered some strategies we could never have created. (Go Master Ke Jie famously said “I would go as far as to say not a single human has touched the edge of the truth of Go.”)
Perhaps we can imagine a sort of “logical causal isolation.” An algorithm is logically causally isolated from us if we cannot discover it (e.g. in the case of the Go strategies that AlphaGo used) and we cannot specify an algorithm to discover it (except by random accident) given finite computation over a finite time horizon (i.e. in the lifetime of the observable universe).
Importantly, we can devise algorithms which search the entire space of algorithms (e.g.
generate all permutations all possible strings of bits less than length n as n approaches infinity), but there’s little reason to expect that such a strategy will result in any useful outputs of some finite length (there appear to be enough atoms in the universe (1080) to represent all possible algorithms of length log2(1080)≈265.There’s one important weakness in LCI (that doesn’t exist in Physical Causal Isolation). We can randomly jump to algorithms of arbitrary lengths. This stipulation gives us the weird ability to pull stuff from outside our LCI-cone into it. Unfortunately, we cannot do so with the expectation of arriving at a useful algorithm. (There’s an interesting question about which I haven’t yet thought about the distribution of useful algorithms of a given length.) Hence we must add the caveat to our definition of LCI “except by random accident.”
We aren’t LCI’d from the strategies AlphaGo used, because we created AlphaGo and AlphaGo discovered those strategies (even if human Go masters may never have discovered them independently). I wonder what algorithms exist beyond not just our horizons, but the horizons of all the algorithms which descend from everything we are able to compute.
2 things necessary for an algorithm to be useful:
If it’s not fast enough, it doesn’t matter how good it is
If we don’t know what it’s good for, it doesn’t matter how good it is (until we figure that out)
Part of the issue with this might be programs that don’t work or do anything (Beyond the trivial, it’s not clear how to select for this, outside of something like AlphaGo.)
Sure! My brute-force bitwise algorithm generator won’t be fast enough to generate any algorithm of length 300 bits, and our universe probably can’t support any representation of any algorithm of length greater than (the number of atoms in the observable universe) ~ 10^82 bits. (I don’t know much about physics, so this could be very wrong, but think of it as a useful bound. If there’s a better one (e.g. number of Planck volumes in the observable universe), substitute that and carry on, and also please let me know!)
Another class of algorithms that cause problems are those that don’t do anything useful for some number of computations, after which they begin to output something useful. We don’t really get to know if they will halt, so if the useful structure emerges after some number of steps, we may not be committed to or able to run it that long.
I’m not a physicist either, but quantum mechanics might change the limits. (If it scales, though this might leave input and output limits; if the quantum computer can’t store the output in classical mode, then it’s ability to run the program probably doesn’t matter. This might make less efficient crypto systems more secure, by virtue of size.*)
*Want your messages to be more secure? Padding.
Want your key more secure? Length.
You would hope that people actually saw steelmanning as an ideal to follow. If that was ever true, the corona pandemic and the policy response seem to have killed the demand for this. It seems to become acceptable to attribute just any kind of seemingly-wrong behavior to either incredible stupidity or incredible malice, both proving that all institutions are completely broken.
I like the word “institurions”. Some mix of institutions, intuitions, and centurions, and I agree that they’re completely broken.
:-) Thanks. But I corrected it.
For the foreseeable future, it seems that anything I might try to say to my UK friends about anything to do with LW-style thinking is going to be met with “but Dominic Cummings”. Three separate instances of this in just the last few days.
Can you give some examples of “LW-style thinking” that they now associate with Cummings?
On Twitter I linked to this saying
Response:
Huh. Wow.
Seems like a good discussion could be had about long-term predictions and how much evidence there is to be had in short-term political fluctuations. The Cummings silliness vs unprecedented immigration restrictions—which is likely to have impact 5 years from now?
[Book Review] Surfing Uncertainty
Surfing Uncertainty is about predictive coding, the theory in neuroscience that each part of your brain attempts to predict its own inputs. Predictive coding has lots of potential consequences. It could resolve the problem of top-down vs bottom-up processing. It cleanly unifies lots of ideas in psychology. It even has implications for the continuum with autism on one end and schizophrenia on the other.
The most promising thing about predictive coding is how it could provide a mathematical formulation for how the human brain works. Mathematical formulations are great because once they let you do things like falsify experiments and simulate things on computers. But while Surfing Uncertainty goes into many of the potential implications of predictive codings, the author never hammers out exactly what “prediction error” means in quantifiable material terms on the neuronal level.
This book is a reiteration of the scientific consensus[1]. Judging by the total absense of mathematical equations on the Wikipedia page for predictive coding, I suspect the book never defines “prediction error” in mathematically precise terms because no such definition exists. There is no scientific consensus.
Perhaps I was disappointed with this book because my expectations were too high. If we could write equations for how the human brain performs predictive processing then we would be significantly closer to building an AGI than where we are right now.
The book contains 47 pages of scientific citations. ↩︎
Has anyone tried to work with neural networks predicting the weights of other neural networks? I’m thinking about that in the context of something like subsystem alignment, e.g. in an RL setting where an agent first learns about the environment, and then creates the subagent (by outputting the weights or some embedding of its policy) who actually obtains some reward
Observation: It should generally be safe to forbid non-termination when searching for programs/algorithms.
In practice, all useful algorithms terminate: If you know that you’re dealing with a semi-decidable thing and doing serious work, you’ll either (a) add a hard cutoff, or (b) structure the algorithm into a bounded step function and a controller that decides whether or not to run for another step. That transformation is not adding significant overhead size-wise, so you’re bound to find a terminating algorithm “near” a non-terminating one!
Sure, that slightly changes the interface – it’s now allowed to abort with “don’t know”, but that’s a transformation that you likely would have applied anyway. Even if you consider that a drawback, not having to deal with potentially non-terminating programs / being able to use a description format that cannot represent non-terminating forms should more than make up for that.
(I just noticed this while thinking about how to best write something in Coq (and deciding on termination by “fuel limit”), after AABoyles’ shortform on logical causal isolation with its tragically simple bit-flip search had recently made me think about program enumeration again…)
I had a very useful conversation with someone about how and why I am rambly. (I rambled a lot in the conversation!).
Disclaimer: I am not making much effort to not ramble in this post.
A couple takeaways:
1. Working Memory Limits
One key problem is that I introduce so many points, subpoints, and subthreads, that I overwhelm people’s working memory (where human working memory limits is roughly “4-7 chunks”).
It’s sort of embarrassing that I didn’t concretely think about this before, because I’ve spent the past year SPECIFICALLY thinking about working memory limits, and how they are the key bottleneck on intellectual progress.
So, one new habit I have is “whenever I’ve introduced more than 6 points to keep track of, stop and and figure out how to condense the working tree of points down to <4.
(Ideally, I also keep track of this in advance and word things more simply, or give better signposting for what overall point I’m going to make, or why I’m talking about the things I’m talking about)
...
2. I just don’t finish sente
I frequently don’t finish sentences, whether in person voice or in text (like emails). I’ve known this for awhile, although I kinda forgot recently. I switch abruptly to a new sentence when I realize the current sentence isn’t going to accomplish the thing I want, and I have a Much Shinier Sentence Over Here that seems much more promising.
But, people don’t understand why I’m making the leap from one half-finished thought to another.
So, another simple habit is “make sure to finish my god damn sentences, even if I become disappointed in them halfway through”
...
3. Use Mindful Cognition Tuning to train on *what is easy for people to follow*, as well as to improve the creativity/usefulness of my thoughts.
I’ve always been rambly. But a thing that I think has made me EVEN MORE rambly in the past 2 years is a mindful-thinking-technique, where you notice all of your thoughts on the less-than-a-second level, so that you can notice which thought patterns are useful or anti-useful.
This has been really powerful for improving my thought-quality. I’m fairly confident that I’ve become a better programmer and better thinker because of it.
But, it introduces even more meta-thoughts for me to notice while I’m articulating a sentence, which distract me from the sentence itself.
What I realized last weekend was: I can use Mindful Cognition to notice what types of thoughts/sentences are useful for *other people’s comprehension of me*, not just how useful m original thought processes are.
The whole point of the technique is to improve your feedback loop (both speed and awareness), which makes it easier to deliberate practice. I think if I just apply that towards Being More Comprehensible, it’ll change from being a liability in rambliness to an asset.
re working memory: never thought of it during conversations, interesting. it seems that we sometime hold the nodes of the conversation tree to go back to them afterward. and maybe if you’re introducing new concepts while you’re talking people need to hold those definitions in working memory as well.
Thoughts on Ryan Carey’s Incorrigibility in the CIRL Framework (I am going to try to post these semi-regularly).
This specific situation looks unrealistic. But it’s not really trying to be too realistic, it’s trying to be a counterexample. In that spirit, you could also just use R2(a,sd)=1000, which is a reward function parametrized by θ that gives the same behavior but stops me from saying “Why Not Just set θ=−1”, which isn’t the point.
How something like this might actually happen: you try to have your R1 be a complicated neural network that can approximate any function. But you butcher the implementation and get something basically random instead, and this R2 cannot approximate the real human reward.
An important insight this highlights well: An off-switch is something that you press only when you’ve programmed the AI badly enough that you need to press the off-switch. But if you’ve programmed it wrong, you don’t know what it’s going to do, including, possibly, its off-switch behavior. Make sure you know under which assumptions your off-switch will still work!
Assigning high value to shutting down is incorrigible, because the AI shuts itself down. What about assigning high value to being in a button state?
The paper considers a situation where the shutdown button is hardcoded, which isn’t enough by itself. What’s really happening is that the human either wants or doesn’t want the AI to shut down, which sounds like a term in the human reward that the AI can learn.
One way to do this is for the AI to do maximum likelihood with a prior that assigns 0 probability to the human erroneously giving the shutdown command. I suspect there’s something less hacky related to setting an appropriate prior over the reward assigned to shutting down.
The footnote on page 7 confuses me a bit—don’t you want the AI to always defer to the human in button states? The answer feels like it will be clearer to me if I look into how “expected reward if the button state isn’t avoided” is calculated.
Also I did just jump into this paper. There are probably lots of interesting things that people have said about MDPs and CIRLs and Q-values that would be useful.
Thoughts on Dylan Hadfield-Menell et al.’s The Off-Switch Game.
I don’t think it’s quite right to call this an off-switch—the model is fully general to the situation where the AI is choosing between two alternatives A and B (normalized in the paper so that U(B) = 0), and to me an off-switch is a hardware override that the AI need not want you to press.
The wisdom to take away from the paper: An AI will voluntarily defer to a human—in the sense that the AI thinks that it can get a better outcome by its own standards if it does what the human says—if it’s uncertain about the utilities, or if the human is rational.
This whole setup seems to be somewhat superseded by CIRL, which has the AI, uh, causally find UA by learning its value from the human actions, instead of evidentially(?) doing it by taking decisions that happen to land it on action A when UA is high because it’s acting in a weird environment where a human is present as a side-constraint.
Could some wisdom to gain be that the high-variance high-human-rationality is something of an explanation as to why CIRL works? I should read more about CIRL to see if this is needed or helpful and to compare and contrast etc.
Why does the reward gained drop when uncertainty is too high? Because the prior that the AI gets from estimating the human reward is more accurate than the human decisions, so in too-high-uncertainty situations it keeps mistakenly deferring to the flawed human who tells it to take the worse action more often?
The verbal description, that the human just types in a noisily sampled value of UA, is somewhat strange—if the human has explicit access to their own utility function, they can just take the best actions directly! In practice, though, the AI would learn this by looking at many past human actions (there’s some CIRL!) which does seem like it plausibly gives a more accurate policy than the human’s (ht Should Robots Be Obedient).
The human is Boltzmann-rational in the two-action situation (hence the sigmoid). I assume that it’s the same for the multi-action situation, though this isn’t stated. How much does the exact way in which the human is irrational matter for their results?
Thoughts on Abram Demski’s Partial Agency:
When I read Partial Agency, I was struck with a desire to try formalizing this partial agency thing. Defining Myopia seems like it might have a definition of myopia; one day I might look at it. Anyway,
Formalization of Partial Agency: Try One
A myopic agent is optimizing a reward function R(x0,y(x0)) where x is the vector of parameters it’s thinking about and y is the vector of parameters it isn’t thinking about. The gradient descent step picks the δx in the direction that maximizes R(x0+δx,y(x0)) (it is myopic so it can’t consider the effects on y), and then moves the agent to the point (x0+δx,y(x0+δx)).
This is dual to a stop-gradient agent, which picks the δx in the direction that maximizes f(x0+δx,y(x0+δx)) but then moves the agent to the point (x0+δx,y(x0)) (the gradient through y is stopped).
For example,
Nash equilibria - x are the parameters defining the agent’s behavior.y(x0) are the parameters of the other agents if they go up against the agent parametrized by x0. R is the reward given for an agent x going up against a set of agents y.
Image recognition with a neural network - x is the parameters defining the network, y(x0) are the image classifications for every image in the dataset for the network with parameters x0, and R is the loss function plus the loss of the network described by x on classifying the current training example.
Episodic agent - x are parameters describing the agents behavior.y(x0) are the performances of the agent x0 in future episodes.R is the sum of y, plus the reward obtained in the current episode.
Partial Agency due to Uncertainty?
Is it possible to cast partial agency in terms of uncertainty over reward functions? One reason I’d be myopic is if I didn’t believe that I could, in expectation, improve some part of the reward, perhaps because it’s intractable to calculate (behavior of other agents) or something I’m not programmed to care about (reward in other episodes).
Let R1 be drawn from a probability distribution over reward functions. Then one could decompose the true, uncertain, reward into R′=R0(x0)+R1(x0) defined in such a way that E(R1(x0+δx)−R1(x0))≈0 for any δx? Then this is would be myopia where the agent either doesn’t know or doesn’t care about R1, or at least doesn’t know or care what its output does to R1. This seems sufficient, but not necessary.
Now I have two things that might describe myopia, so let’s use both of them at once! Since you only end up doing gradient descent on R0, it would make sense to say R′(x)=R(x,y(x)) , R0(x)=R(x,y(x0)) , and hence that R1(x)=R(x,y(x))−R(x,y(x0)).
Since R1(x0+δx)=R1(x0)+δx∂R1∂x for small δx, this means that E(∂R1∂x)=0 , so substituting in my expression for R1 gives E(∂R∂x+∂R∂y∂y∂x−∂R∂x)=0 , so E(∂R∂y∂y∂x)=0 . Uncertainly is only over R, so this is just the claim that the agent will be myopic with respect to y if E(∂R∂y)=0. So it won’t want to include y in its gradient calculation if it thinks the gradients with respect to y are, on average, 0. Well, at least I didn’t derive something obviously false!
But Wait There’s More
When writing the examples for the gradient descenty formalisation, something struck me: it seems there’s a R(x)=r(x)+∑iyi(x) structure to a lot of them, where r is the reward on the current episode, and yi are rewards obtained on future episodes.
You could maybe even use this to have soft episode boundaries, like say the agent receives a reward rt on each timestep so R(x)=r0(x)+r1(x)α+r2(x)α2+∑i=3ri(x)αi , and saying that α3≪1 so that ∂R∂ri≪1 for i≥3, which is basically the criterion for myopia up above.
Unrelated Note
On a completely unrelated note, I read the Parable of Predict-O-Matic in the past, but foolishly neglected to read Partial Agency beforehand. The only thing that I took away from PoPOM the first time around was the bit about inner optimisers, coincidentally the only concept introduced that I had been thinking about beforehand. I should have read the manga before I watched the anime.
So the definition of myopia given in Defining Myopia was quite similar to my expansion in the But Wait There’s More section; you can roughly match them up by saying r(x)=∑ifiri(x) and yi(x)=(1−fi)ri(x) , where fi is a real number corresponding to the amount that the agent cares about rewards obtained in episode i and ri is the reward obtained in episode i. Putting both of these into the sum gives R(x)=∑iri(x), the undiscounted, non-myopic reward that the agent eventually obtains.
In terms of the R=R0+R1 definition that I give in the uncertainty framing, this is R0=R(x,y0)=∑ifiri(x)+∑i(1−fi)ri(x0), and R1=R(x,y)−R(x,y0)=∑i(1−fi)(ri(x)−ri(x0)).
So if you let r be a vector of the reward obtained on each step and f be a vector of how much the agent cares about each step then x→x+ϵ∑ifi∂ri∂x , and thus the change to the overall reward is R→R+ϵ∑i∂ri∂x∑jfj∂rj∂x , which can be negative if the two sums have different signs.
I was hoping that a point would reveal itself to me about now but I’ll have to get back to you on that one.
Some other people who play to win
It’s a crowd I’d come into contact with as a manager of an online bookshop (and most of the reason I quitted). Usually, I can pretend they don’t exist, but… we all know how it goes… and now that they don’t make my blood boil every weekend, I can afford to speak about them.
“Some other people” will play to win—say, a facebook lottery with a book for a prize, and they will mean it. If they don’t win, they will say the lottery was rigged. Public righteous indignation on every player’s behalf is a weapon (and for the manager, a potent vaccine against righteously indignant polemics of many other kinds). Private appeals to the manager’s pity; commenting the rules’ exploitable/exploited loopholes—after the winner is announced; repeating actions which have already been answered elsewhere in the thread. I don’t include ‘filing a complaint’ here, because it’s frankly too straightforward for most of them, most of the time; the bookshop would likely send them a book with an eloquent blessing/apology, just to get them to shut up and earn good PR points for “owning up to mistakes”. But in practice, it still matters too much to be the actual winner, and the brain of the trophy-gatherer works like other brains don’t. At least not for a while.
I’m not unusually out-of-touch with customers; I was recommended for the job after two years in an offline shop. And this was… entirely different. I’d never encountered people with whole profiles dedicated to reposting online lotteries—living people I had to call on the phone. It is another world.
When I read about (simple) “pure” game theoretical problems, in which the players “care only about winning”, I cannot reconcile the image of Worthy Rivals the author has in mind with the actual Really-Want-This-Whatever Whiners who seek out such contests. Get it, not the passively allowing themselves to be drawn into a strategic game kind of players, but the self-sorting to exploit as many offers as possible kind. They will be few, yes. Nobody of them might force their way through every single time.
But they will define the meaning of the rules you think you write.
What’s the book?
Not any particular book, but rather some frequent conditions of game theory problems I have seen here and elsewhere (my fb friend keeps posting such pieces). “The players care only about winning” etc. Well, some people actually do.
There are two kinds of pleasurable feelings. The first one is a self-reinforcing loop, where the in-the-moment pleasure leads to craving for more pleasure, such as mindlessly scrolling through social media, or eating highly-processed, highly-palatable food. The second is pleasure gained through either thoughtfully consuming good content, like listening to good music or reading good books, or the fulfillment of a task that’s meaningful, such as getting good grades or getting a promotion for sustained conscientious effort.
The first is pleasure for its own sake, without any “real world rewards” that come with it, ie, distractions. The second isn’t as “addictive” as the first, nor does it cause the same spikes in pleasure, but it comes with real world tangible rewards.
There is no way to completely eliminate the human need for the first pleasure. But the need can be reduced. The ratio of second:first pleasure, is the degree to which a person is able to achieve his goals, the degree to which a person is successful.
There’s a large piece missing from your model—why seek pleasure at all? Wouldn’t it be better to measure success directly based on meeting long-term goals (clippy interjects from the balcony: like number of paperclips created!).
As an add-on, I found this LW article today that captures the essence of “first pleasure”
I’m not quite sure I got this part, could you please elaborate on it?
Here, I would argue that the feeling of the second pleasure is essential to meeting long-term goals. Feeling good about accomplishing sub-tasks will keep someone working towards an important goal, especially if it requires a long period of sustained effort.
Thus, meeting short-term goal → success → second pleasure → working on next short-term goal; with enough iterations, that will lead to the meeting of long-term goals and thus success.
https://wiki.lesswrong.com/wiki/Paperclip_maximizer is the canonical example of over-simplified goal optimization. I bring it up mostly as a reminder that getting your motivational model wrong can lead to undesirable actions and results.
Which leads to my main point. You’re recommending one type of pleasure over another, based on it being more aligned with your non-pleasure-measured goals. I’m wondering why you are arguing for this, as opposed to just pursuing the goals directly, without consideration of pleasure.
Ah, now I’ve got what you mean. Thanks for referring me to that thought experiment, I don’t have much prior knowledge on the field of AI so that was definitely a new insight for me.
I see now that my original shortform did not explicitly state that my terminal value was indeed the fulfillment of important goals. I was reflecting more on the distinction between pleasurable feelings that led to distraction & bad habits, vs ones that led to the actual fulfillment of goals. It was a personal reminder to experience the latter in place of the former, as much as I can.
Now, I hold the view that pleasure can be a useful tool in the pursuit of my goals. It is a means to that end. An important caveat that your response reminded me of, though, is that sometimes pursuing goals might not be immediately pleasurable, so it might be wise not to naïvely expect pleasure from every part of that process.
Sentences spoken aloud are a latent space embedding of our thoughts; when trying to move a thought from our mind to another’s, our thoughts are encoded with the aim of minimizing the other person’s decoder error.
There’s a problem at parties where there’ll be a good, high-context conversation happening, and then one-too-many-people join, and then the conversation suddenly dies.
Sometimes this is fine, but other times it’s quite sad.
Things I think might help:
If you’re an existing conversation participant:
Actively try to keep the conversation small. The upper limit is 5, 3-4 is better. If someone looks like they want to join, smile warmly and say “hey, sorry we’re kinda in a high context conversation right now. Listening is fine but probably don’t join.”
If you do want to let a newcomer join in, don’t try to get them up to speed (I don’t know if I’ve ever seen that actually work). Instead, say “this is high context so we’re not gonna repeat the earlier bits, maybe wait to join in until you’ve listened enough to understand the overall context”, and then quickly get back to the conversation before you lose the Flow.
If you want to join a conversation:
If there are already 5 people, sorry, it’s probably too late. Listen if you find it interesting, but if you actively join you’ll probably just kill the conversation.
Give them the opportunity to gracefully keep the conversation small if they choose. (say something like “hey can I join? It sounds like maybe a high context conversation, no worries if you wanna keep it small.”)
Listen for longer before joining. Don’t just wait till you understand the current topic – try to understand the overall vibe, and what previous topics might be informing the current one. Try to get a sense of what each current participant is getting out the conversation. When you do join, do so in a small way that gives them affordance to shift back to an earlier topic if your new contribution turned out to be not-actually-on-topic.
I hosted an online-party using zoom breakout rooms a few weeks ago and ran into similar problems.
Half-way through the party I noticed people were clustering in suboptimal size conversations and bringing high-context conversations to a stop, so I actually brought everybody backed to the lobby then randomly assigned them to groups of 2 or 3 - and when I checked 10 minutes later everyone was in the same two rooms again with groups of 8 − 10 people.
AFAICT this was status/feelings driven—there were a few people at the party who were either existing high-status to the participants, or who were very charismatic, and everyone wanted to be in the same conversation as them.
I think norm-setting around this is very hard, because it’s natural to want to be around high-status and charismatic people, and it’s also natural to want to participate in a conversation you’re listening to.
I’m going to try to add your suggestions to the top of the shared google doc next time I host one of these and see how it goes.
Agreed with the status/feelings cause. And I’m not 100% sure the solution is “prevent people from doing the thing they instinctively want to do” (especially “all the time.”)
My current guess is “let people crowd around the charismatic/and/or/interesting people, but treat it more like a panel discussion or fireside chat, like you might have at a conference, where mostly 2-3 people are talking and everyone else is more formally ‘audience.’”
But doing that all the time would also be kinda bad in different ways.
In this case… you might actually be able to fix this with technology? Can you literally put room-caps on the rooms, so if someone wants to be the 4th or 6th person in a room they… just… can’t?
+lots. Some techniques:
physically separate the group. Go into another room or at least corner. Signal that you’re not seeking additional participants.
When you notice this, make it explicit—“I’m really enjoying the depth of this conversation, should we move into the lounge for a brandy and a little more quiet?”
Admit (to yourself) that others may feel excluded, because they are. At many gatherings, such discussions/situations are time-bound and really can’t last more than 10-45 minutes. The only solution is to have more frequent, smaller gatherings.
Get good at involved listening—it’s different than 1:1 active listening, but has similar goals: don’t inject any ideas, but do give signals that you’re following and supporting. This is at least 80% as enjoyable as active participation, and doesn’t break the flow when you join a clique in progress.
I wonder what analogs there are to online conversations. I suspect there’s a lot of similarity for synchronous chats—too many people make it impossible to follow. For threaded, async discussions, the limits are probably much larger.
FYI, the actual motivating example here was at a party in gather.town, (formerly online.town, formerly town.siempre), which has much more typical “party” dynamics. (i.e people can wander around an online world and video chat with people nearby).
In this case there were actually some additional complexities – I had joined a conversation relatively late, I did lurk for quite awhile, and wait for the current set of topics to die down completely before introducing a new one. And then the conversation took a turn that I was really excited by, and at least 1-2 other people were interested in, but it wasn’t obvious to me that it was interesting to everyone else (I think ~5 people involved total?)
And then a new person came in, and asked what we were talking about and someone filled them in… …and then immediately the conversation ended. And in this case I don’t know if the issue was more like “the newcomer killed the conversation” or “the convo actually had roughly reached it’s natural end, and/or other people weren’t that interested in the first place.”
But, from my own perspective, the conversation had just finished covering all the obvious background concepts that would be required for the “real” conversation to begin, and I was hoping to actually Make Real Progress on a complex concept.
So, I dunno if this counted as “an interesting conversation” yet, and unfortunately the act of asking the question “hey, do we want to continue diving deep into this, or wrap up and transition into some other convo?” also kinda kills the conversation. Conversations are so god damn fragile.
What I really wished was that everyone already had common knowledge of the meta-concept, wherein:
Party conversations are particularly fragile
Bringing a newcomer up to speed is usually costly if the conversation is doing anything deep
We might or might not want to continue delving into the current convo (but we don’t currently have common knowledge of this in either direction)
And if everyone (newcomer included) had those concepts, and new everyone had those concepts, then I feel like I could have asked more gracefully “hey, I’m kinda interested in continuing to hash out some ideas here. Are people up for taking this high context”, and had people give their honest answer.
[EDIT, was intended as a response to Raemon, not Dagon.]
Maybe it’s the way you phrase the responses. But as described, I get the impression that this norm would mainly work for relatively extroverted persons with low rejection sensitivity.
I’d be much less likely to ever try to join a discussion (and would tend to not attend events with such a norm). But maybe there’s a way to avoid this, both from “my side” and “yours”.
Hmm, seems like important feedback. I had specifically been trying to phrase the responses in a way that addressed this specific problem. Sounds like it didn’t work.
There is some intrinsic rejection going on here, which probably no amount of kind wording can alleviate for a rejection-sensitive person.
For my “sorry, we’re keeping the convo small” bit, I suggested:
The Smile Warmly part was meant to be a pretty active ingredient, helping to reassure them it isn’t personal.
Another thing that seems pretty important, is that this applies to all newcomers, even your friends and High Status People. (i.e. hopefully if Anxious Alex gets turned away, but later sees High Status Bob also get turned away, they get reassured a bit that this wasn’t about them)
Virtue ethics seems like model-free consequentialism to me.
I’ve was thinking along similar lines!
From my notes from 2019-11-24: “Deontology is like the learned policy of bounded rationality of consequentialism”
I’m looking for an old post where Eliezer makes the basic point that we should be able to do better than intellectual figures of the past, because we have the “unfair” advantage of knowing all the scientific results that have been discovered since then.
I think he cites in particular the heuristics and biases literature as something that thinkers wouldn’t have known about 100 years ago.
I don’t remember if this was the main point of the post it was in, or just an aside, but I’m pretty confident he made a point like this at least once, and in particular commented on how the advantage we have is “unfair” or something like that, so that we shouldn’t feel at all sheepish about declaring old thinkers wrong.
Anybody know what post I’m thinking of?
1. https://www.lesswrong.com/posts/96TBXaHwLbFyeAxrg/guardians-of-ayn-rand
(Not sure if the material is accurate, but I think it’s the post you’re looking for. There could have been more than one on that though.)
2. https://www.lesswrong.com/posts/7s5gYi7EagfkzvLp8/in-defense-of-ayn-rand
References 1.
Thanks!
Weird thing I wish existed: I wish there were more videos of what I think of as ‘math/programming speedruns’. For those familiar with speedrunning video games, this would be similar except the idea would be to do the same thing for a math proof or programming problem. While it might seem like this would be quite boring since the solution to the problem/proof is known, I still think there’s an element of skill to and would enjoy watching someone do everything they can to get to a solution, proof, etc. as quickly as possible (in an editor, on paper, LaTex, etc.).
This is kind of similar to streaming ACM/math olympiad competition solving except I’m equally more in people doing this for known problems/proofs than I am for tricky but obscure problems. E.g., speed-running the SVD derivation.
While I’m posting this in the hope that others are also really interested, my sense is that this would be incredibly niche even amongst people who like math so I’m not surprised it doesn’t exist...
Somewhat related:
https://xenaproject.wordpress.com/2020/05/23/the-complex-number-game/
This is awesome! I’ve been thinking I should try out the natural number game for a while because I feel like formal theorem proving will scratch my coding / video game itch in a way normal math doesn’t.
I have a friend who might be into programming speedrunning https://merveilles.town/@cancel/104005117320841920
Seems like the post you linked is a joke. Were you serious about the friend?
Serious in that I mean he might, I’d say, 0.1 that he’d be interested, but if that’s not negligible, I think if he took it up he’d be very good at it. I’ll ask him.
Cool!
I had a similar idea which was also based on an analogy with video games (where the analogy came from let’s play videos rather than speedruns), and called it a live math video.
Cool, I hadn’t seen your page previously but our ideas do in fact seem very similar. I think you were right to not focus on the speed element and instead analogize to ‘let’s play’ videos.
Related: here, DJB lays out the primary results of a single-variable calculus course in 11 LaTex-ed pages.
I’m not super familiar with the competitive math circuit, but my understanding is that this is part of it? People are given a hard problem and either individually or as a team solve it as quickly as possible.
Yep, I touched on this above. Personally, I’m less interested in this type of problem solving than I am in seeing someone build to a well-known but potentially easier to prove theorem, but I suspect people solving IMO problems would appeal to a wider audience.
Do you know of any videos on this? Ideally while the person is narrating their thoughts out loud.
3Blue1Brown has a video where he sort of does this for a hard Putnam problem. I say “sort of” because he’s not solving the problem in real time so much as retrospectively describing how one might solve it.
Yeah, that is one of my favorite videos by 3Blue1Brown and more like it would be pretty good.
The problem with this is that it is very difficult to figure out what counts as a legitimate proof. What level of rigor is required, exactly? Are they allowed to memorize a proof beforehand? If not, how much are they allowed to know?
Solutions might be better to go with than proofs—if the answer is wrong, that’s more straightforward to show that whether or not a proof is wrong.
Yeah what would be ideal is if theorem provers were more usable and then this wouldn’t be an issue (although of course there’s still the issue of library code vs. from scratch code but this seems easier to deal with).
Memorizing a proof seems fine (in the same way that I assume you end up basically memorizing the game map if you do a speedrun).
When I was in high school, I once had a conversation with a classmate that went something like this (except that it was longer and I was less eloquent):
Him: “German is a Scandinavian language.”
Me: “No, it’s not. German and the Scandinavian languages both fall under the umbrella of Germanic languages, but ‘Scandinavian languages’ refers to a narrower category that doesn’t include German.”
Him: “Well that’s your opinion.”
Me: “No??? That’s not what an opinion is???”
Him: “Look, it’s your opinion that German isn’t a Scandinavian language, and it’s my opinion that it is. We can agree to disagree.”
Me: ??????????????????!!!!!!!!!????!?!?!?!?! *punches self in face*
----
When I was taking a required intro biology course in college, I had already read a bunch of LW and SSC, notably including That Chocolate Study. So when the professor put Bohannon’s results and methodology up on the projector, I was ready as heck to talk about all of the atrocities therein. The professor asked us to pair up with the person next to us to discuss whether we believed Bohannon’s results, and I decided to give the freshman next to me the chance to speak first before I absolutely demolished everything. The girl turned to me with wide eyes and a confident, creaky-voice drawl, and said, verbatim: “I think it’s true, because chocolate is known to be a superfood.”
I was floored. How could this be happening in real life? I was at an elite college with a sub-10% acceptance rate, and this person next to me had just said “known to be” and “superfood” like they explained anything – like they meant anything. I will never forget those words. Looking back, that may have been the day I decided to move to the Bay after graduating. No regrets.
I’ve been thinking about people’s mindset as it relates to spending their free time. Specifically, when you go to do something ‘productive’ like learn about a new topic, work through exercises in a textbook, go through an online course, etc...do you feel that you have to intentionally decide not to play video games, watch Netflix, etc and forego short-term happiness? Or do you feel that this decision is straightforward because that’s what you would prefer to be doing and you don’t feel like you sacrifice anything?