You should read authentic happiness and flourish by Martin Seligman for a very similar and slightly more expanded opinion on the topic.
I suspect anyone studying happiness very quickly noticed that it’s about meaning and not “ice-cream makes me happy for a moment”.
If the actual utility you receive as a function of the total “payout” of your “investments” has diminishing marginal returns, then the character of the portfolio to maximize expected utility depends upon the failure correlations between the investment options.
IE, in the case that the utility function is sufficiently convex to payout and the various investments all fail independently of each other, a strategy of investing in only the highest yield and lowest risk choices is not optimal: a small investment in a middling investment decreases the risk of total failure (and corresponding hit to expected utility) enough to be worth the hit to expected payout.
I haven’t run the analysis, but my intuition is that advocacy for a barbell strategy limited to just high-risk stocks and T-bills is an empirical claim about the risks along the following lines:
1) The failure of middling risk stock is well correlated with the failure of high risk stock
2) The failure of less-risky investments than T-bills (paper currency, rice & beans under nitrogen, solar panels, etc.) are well-correlated with the failure of T-bills and have lower annual yields.
3) Utility is a moderately convex function of payout. (If it were very convex, you’d want most or all of your funds in T-bills, not just a bit; if it were linear or concave, “risk” isn’t a thing and all funds would be in the stocks.)
I’ll trust Taleb on #1, and #3 seems reasonable most of the time, but on #2, it would seem to me that while a good portfolio would be based around the high-risk stocks backed up with a small portion of cash-equivalents, the “insurance” against failure of both of those things is cheap enough that it should be included early on.
(As an aside, I’m pretty sure that Taleb suggests “mostly” high-yield/high-risk holdings, with only enough T-bill stuff to keep you off the streets if the stock fails. That’s not what I’d pick out as a strategy that is likely to cause bad outcomes because you didn’t take enough risk.)
Some parts of your anthropic argument don’t seem right… and it’s lead me to a related thought
It seems likely that an expanding civilization would capture much more anthropic measure than what exists on earth. Billions of times more. So, why should we find ourselves here where the cosmic body of life is still so small? Why weren’t we born hundreds of thousands of years later in the heavenly gardens of an alien diaspora?
What if… Siblings Are Dangerous: What if most life-supporting universes have in common a dynamic where technology significantly increases the danger of coexisting with other living things. Creating new life on a grand scale is made unviable, our gardens will tend to evolve competitors who will eventually kill everything around them until only they are left, or until nothing is left. The theory is that life is at its most numerous when it is too dumb and too weak to pose much of a threat to itself, as we are.
I’d guess that expanding one’s own mind into well-controlled slave cortexes would capture similar amounts of new anthropic measure.. but maybe once you go over a certain scale, having those be independent minds, sufficient to observe their own existence, maybe it just isn’t possible to keep that many star-sized consciousnesses under control.
(You might ask why anthropic measure should be tied to independently-intelligent minds. I don’t know. It just seems to be. I don’t make the rules.)
Any late filter would explain it, though.
Some disconnected thoughts:
In Boston we’re planning Normal Mode. (We rejected Hardcore Mode in previous years, in part because it was a serious problem for people who underwent significant inconvenience to be able to attend.)
I’m good at DevOps and might be able to help the Seattle folks make their app more available if they need it.
I happened to give a eulogy of sorts for Stanislav Petrov last year.
I’m currently going through the latest version of the ritual book and looking for things to nitpick, since I know that a few points (notably the details of the Arkhipov story) have fallen into dispute since last year.
I’d be curious to know what considerations are affecting your decisions to possibly change Petrov Day.
> algorithms don’t get simpler as they get better.
or s you minimize cost along one dimension costs get pushed into other dimensions. Aether variables apply at the level of representation too.
An example of attending to risk of ruin / integrated risk management across multiple dimensions not captured by the barbell strategy would be a couple with children and little in the way of savings buying life insurance.
And its real point seems to be not to argue for or against any particular investment strategy, but to remind us that there are other things that matter more deeply than the accumulation of wealth. [...] Children and a spouse and belonging to an intergenerational community are (at least for many people) wonderful things, and perhaps (as this article suggests) they provide many of the benefits that we might (perhaps unwisely) look for from our investments.
A definition of “wealth” or “investment” coextensive with financial investment is going to lead to models that extend very poorly if at all to overall life decisions. These get conflated a lot. Most risk is not financial.
But having decided whether to marry, how many children (if any) to have, etc., the question remains of what to do with whatever wealth one has after all that
That first decision—allocating investment of your time between fundamentally different kinds of investment like children, friend networks, and financial savings—is the more important one. It’s also one for which the “barbell strategy” is meaningless if taken literally.
I don’t see why the insight fails to apply along a continuum. If you do not have access to Treasury-bond equivalents in the risk you are trying to manage, take whatever the stablest option is for most, and then adjust your exposure-to-large-gains appropriately.
Perhaps this would be something like maintain relationships with your diverse friend group as the safe investment, and then do something for fun once a month where you might make friends with a billionaire as the exposure-to-large-upside investment.
In support of your argument: I only looked up “Ally McBeal” a few minutes ago, and promptly discovered that I not only lacked all but the most cursory information on Ally McBeal but that what information I did have was almost entirely wrong. The name of the program was not “The Ally McBeal Show,” it was not a talk show or comedy sketch show or anything similar, and Ally McBeal is a wholly fictional character, rather than the name of a living person and that person’s fictionalized persona à la Ray Romano of Everybody Loves Raymond.
I think it would be fair to say that I previously had “no idea who Ally McBeal was.” Despite knowing that the name pointed to something, I was incorrect in just about every respect except for which pronouns to use and it had something to do with a television show.
Humans seem to be naturally biphasic: modern hunter-gatherer tribes sleep for 7-8 hours at night, and then nap for 30-60 minutes in the afternoon.
I’m having fun imagining a workplace where this sort of pattern is encouraged. A large part of the difficulty would be dealing with self-destructive work cultures where nobody wants to violate a norm in a way that risks making them look lazy. I’d want to start by having the highest performers try napping after lunch, so that people come to associate it positively with productivity, frame napping as the opposite of lazy, like exercise, it’s not work but it’s something that hard workers do. But there’s a chance the higher-performers are exactly the people who wouldn’t benefit from napping- There is such a thing as a short-sleeping gene in humans and a lot of CEOs seem to have it- so starting with them might soil the whole thing.
I’m remembering hearing stories of a lot of workplaces getting nap pods and telling their employees that it is “okay” to nap. I don’t think this should be taken seriously. If you don’t have enough beds for everyone in the office to sleep, it wont become a norm. It certainly wont become a habit.
I’d want to experiment with assigning a sample of people (or a set of volunteers) to napping every day, that’s a design we could take seriously.
I looked earlier at The Power of Now before, but decided not to buy it because in the Amazon preview of the first few pages, there weren’t any Gears. Are there more gears later? Or does it live up to its promise in the introduction to use words loosely and be primarily a guide for intuition?
The Mind Illuminated is a 484-page textbook, organized by what should be useful to know at various stages of meditation. It’s full of gears. For example, it gives increasingly complete models of the mind as you progress. I sure hope the gears are accurate; I suspect it will become more obvious whether they are or not later in my practice.
I’m trying to understand your comments. I’m going to try to phrase them in the terminology of The Mind Illuminated, if that works. (It’s also possible that these two books have conflicting models of the mind, rendering any attempted translation moot.)
Attention is when you focus on one thing in particular. It tends to isolate that thing, and be analytical. It is more “self” centered.
Peripheral Awareness, or “awareness” for short, tends to take in a whole sensory field at once (e.g. everything you’re seeing), is more contextual and involves less analysis, and is less personal and more objective.
Both attention and awareness can be introspective, centered on your thoughts and feelings, or extrospective, centered on your external senses (e.g., hearing or seeing, but not thinking).
As some examples of this terminology (that I hope I’m getting right):
Typical mindfulness meditation involves focusing your attention on your breath. The Mind Illuminated says that it’s important to also allow awareness to all of your senses as you do this, but this hasn’t been clear to me from other sources.
When you take in a scene, your attention is on particular part of it (e.g., the orange band around the setting sun), while your awareness is taking in the rest of the scene (the sky and trees). That is, for a few seconds until you get distracted.
When your mind wanders, your attention is on your train of thoughts, and you have no awareness. That is, your attention is introspective.
When you are performing trace-back meditation, your attention is also introspective, but it is focused on the memory of your past thoughts. (Though it’s probably more complicated than this: for example, you’re probably doing a mix of remembering past thoughts and thinking in the present “and what could have led to that?“.)
It sounds like in your comment, you’re saying one of these two things:
If you mentally label a moment as such, or try to verify your awareness (attention?) by checking to see whether you perceive more detail, then you must be thinking about that moment (as in, your inner voice is narrating and analyzing the moment). This can only happen after the fact, so you are no longer in that moment.
If you mentally label a moment as such, or try to verify your awareness by checking to see whether you perceive more detail, then your attention must be on that moment (as in, there is no inner voice, but you are noticing some particular part of the moment, seemingly in the present). This can only happen after the fact, so you are no longer in that moment.
If it’s the latter, the “checking in” technique on page 103 of The Mind Illuminated seems relevant:
The second part of cultivating introspective awareness involves checking in using introspective attention. Instead of waiting for introspective awareness to arise spontaneously, as you’ve done until now, you intentionally turn your attention inward to see what’s happening in the mind. Doing this check-in requires longer periods of stable attention. That’s why following and connecting [other techniques, just introduced] are so important at this Stage. These techniques give you more stable attention, making it easier to momentarily shift attention and see what’s happening in the mind.
Yes, checking in disrupts your focus on the breath, but when you pause to reflect on everything happening in your mind, attention needs to shift. At this Stage, this is not only completely okay, it’s actually the key to cultivating introspective awareness. What you’re really doing is training and strengthening introspective awareness by using attention, making awareness of the mind’s activity a habit. Remember from the First Interlude that peripheral awareness filters through an enormous amount of information and selects what’s relevant for attention. But attention also trains peripheral awareness to know which things are important. [...] In this case, if you take attentive interest in what’s happening in your mind, in particular whether or not gross distractions are present, you’re training awareness to alert you to their presence. [...]
In the next stage, however, you learn to phase out checking-in (i.e., introspective attention), which necessarily shifts your attention, in favor of introspective awareness. So this book is saying that you aren’t stuck at always appreciating your mind’s activity in the past. But doing so is a necessary stepping stone to appreciating it in the present. (One that I’m not ready for yet.)
The Power of Now did something interesting to my moment-to-moment awareness, but at least in the short term, it seemed to wreck my productivity.
I would expect it to decrease your attention and increase your peripheral awareness. And more specifically to decrease your attention on thoughts, and increase your peripheral awareness of external senses. Does that sound right? Something like this has happened to me in the past, so I may know what you’re talking about. Fortunately, it’s way easier to shift things back toward attention. Talking or thinking will do it; a thinking-heavy video game (hello Factorio) is very efficient IIRC. I can imagine track-back meditation being especially good at that too, as you say.
Left at this, it seems like a pretty bad trade-off: for me at least, attention on abstract thoughts is way more important to what I do than awareness of senses. But:
The Mind Illuminated says that meditation should give your more total attention+awareness, and eventually let you consciously balance the two as appropriate. It may take a while, though.
From my past experience, when I had more awareness I felt like I was way worse at things. I noticed every time I forgot something, and saw lots of dumb thoughts. But it seems likely to me now that that’s just how things have always been, and that was one of the few times I was aware of it. Do you say your productivity was worse because you actually got fewer things done, or just because you noticed lots of awkward gaps and tangents that might’ve always been there, previously unnoticed?
The goal of mindfulness might be interpreted in different ways
The goal? The Mind Illuminated has a dozen different goals for the various stages. Overall, it says:
The two main objectives of meditation practice are:
Developing stable attention.
Cultivating powerful mindfulness that optimizes the interaction between attention and awareness.
This is a clear strawman, and is compounding the sense I have that we’re trying to score points now.
I sincerely apologize, I sometimes completely fail to communicate my intention. I gave the example of the random filter only to convey what I don’t understand about your argument (needless to say I don’t suggest the two things are comparable in any way). I should have wrote that explicitly (edited). Sorry!
If you want to discuss how we could resolve or improve this issue, I’m interested.
Of course! I’ll think about this topic some more. I suggest we take this offline—the nesting level here has quite an impact on my browser :)
Regarding all your arguments that use Intent Verification, my tentative position is that IV can’t be relied on to filter actions (as we’re still discussing under this sub-thread).
Nothing, but spending energy changes resources available, just as making a paperclip uses energy. If I make a paperclip, and then destroy the paperclip, that doesn’t decrease (and in fact, increases) the impact. Perhaps there is a way of doing this with available energy, but it doesn’t really matter because IV catches this. I mean, it’s basically just very obvious offsetting.
If I understand your argument correctly, you argue that the resources/energy device B is spending while “undoing impact” decreases the value of utility functions in U which is an additional impact that it might not be able to undo. But why wouldn’t it be able to counter that by saving enough energy/resources that would otherwise be wasted by humanity until the end of the episode? (perhaps it’s what you meant with “available energy”?).
So you start building a device, but before it’s completely specified you’ve already programmed the full intended specification in the device? That doesn’t make sense.
I don’t claim I know how to do it myself :) But for the agent it might be as easy as cloning itself and setting some modified utility function in the new clone (done in a smart way so as to not cause too much impact in any time step).
You said the agent has to seize power over 100 steps, but it can also make a singleton that will “revert” impact, before it’s free? This point is rather moot, as we could also suppose it’s already powerful.
As I argued above, for the agent—creating the device might be as easy as invoking a modified version of itself. In any case, I’m not sure I understand what “already powerful” means. In all the places I wrote “seizing power” I believe I should have just wrote “some convergent instrumental goal”.
On the other hand, in this setting, it isn’t normal, and making a paper clip does not impede all of your optimal plans by one entire step. Therefore, a large penalty is applied.
Suppose in time step 4 the robot that creates paper-clips moves its arm 1 cm to the left. Does this impacts most utility functions in U significantly less than 1 time-step worth of utility? How about a Rumba robot that drives 1 cm forward? It depends on how you define U, but I don’t see how we can assume this issue prevents the agent from building the device (again, compare a single action while building the device to a single action while making “conventional” progress on : why should the former be more “wasteful” for most of U compared to the latter?).
All this can be said on a filter that accepts an action iff a random number in the range [0,1] is greater than… and catch too many false positive while stopping malignant behavior. Decreasing x will make the filter more lenient, but at no point will it be useful.
This is a clear strawman, and is compounding the sense I have that we’re trying to score points now.
while still allowing good ones (and not only in time steps in which some action can yield sufficiently high utility increase). My comment above is an argument for it not being the case.
No, your argument is that there are certain false positives, which I don’t contest. I even listed this kind of thing as an open question, and am interested in further discussion of how we can go about ensuring IV is properly-tuned.
You’re basically saying, “There are false positives, so that makes the core insight that allows IV to work the extent it does wrong, and unlikely to be fixable.” I disagree with this conclusion.
If you want to discuss how we could resolve or improve this issue, I’m interested. Otherwise, I don’t think continuing this conversation will be very productive.
Assuming the the optimal plan starts by pursuing some (unsafe) convergent instrumental goal—we can’t ride it even a bit. Also—I’m not sure I understand how “replaying” will be implemented in a useful way.
Well I certainly empathize with the gut reaction, that isn’t quite right.
Notice that the exact same actions had always been available before we restricted available actions to the optimal or to nothing. I think it’s possible that we could just step along the first n steps of the best plan stopping earlier in a way that lets us just get the good behavior, before any instrumental behavior is actually completed. It’s also possible that this isn’t true. This is all speculation at this point, which is why my tone in that section was also very speculative.
I guess I just don’t understand your argument here for why this won’t work. If it’s catching too many false positives, that’s a great thing in that we just have to make it a little more lenient, but have accomplished the seemingly more difficult task of stopping malignant behavior. If it isn’t catching too many, as I suspect but am not totally convinced is the case, we’re good to go in this regard.
Edit to add: the following is just to illustrate what I don’t understand about your argument (needless to say I don’t suggest the two things are comparable in any way).
All this can be said on a filter that accepts an action iff a random number in the range [0,1] is greater than . You can set and catch too many false positive while stopping malignant behavior. Decreasing will make the filter more lenient, but at no point will it be useful.
If you argue that the Intent Verification filter can be used to prevent the bad tricks we discussed, you need to show that you can use it to filter out the bad actions while still allowing good ones (and not only in time steps in which some action can yield sufficiently high utility increase). My comment above is an argument for it not being the case.
For example, if we do end up having to just ride the optimal plan until it becomes too high-impact, perhaps we can simply keep replaying the favorable first part of the plan (where it tries to please us by actually doing what we want), over and over.
Assuming the the optimal plan starts by pursuing some (unsafe) convergent instrumental goal—we can’t ride it even a bit. Also—I’m not sure I understand how “replaying” will be implemented in a useful way.
I guess I just don’t understand your argument here for why this won’t work. If it’s catching too many false positives, that’s a great thing in that we just have to make it a little more lenient, but have accomplished the seemingly more difficult task of stopping malignant behavior. If it isn’t catching too many, as I suspect but am not totally convinced is the case, we’re good to go in this regard.
For example, if we do end up having to just ride the optimal plan until it becomes too high-impact, perhaps we can simply keep replaying the favorable first part of the plan (where it tries to please us by actually doing what we want), over and over.
In general I think working on taxonomizing failure modes is valuable. In the case of one of the meta generators of failure modes, proxy divergence, even more valuable.
Formalization generators: I often find it useful to think about which kinds of distinctions I can make in order to decompose a category. A few high level ones: split into variant and invariant parts, past/future asymmetry, descriptive/prescriptive parts, continuous vs discrete representation, implementation/algorithmic/functional level (Marr’s levels), complexity classes (in particular some strategies forcing other strategies into worse complexity classes), breadth vs depth first search spaces, and strategies differing due to beliefs about payoff distribution shape (incl. type 1 and 2 error penalties).
With that last one an object level example: knowing that the payoff distribution has changed before others because you’re the one who changed it (caused the proxy to diverge).
I like the generator of how markets might clear under some adversarial conditions and wonder what models quants have of this they might be willing to share.
What would prevent device B from spending that energy?
Nothing, but spending energy changes resources available, just as making a paperclip uses energy. If I make a paperclip, and then destroy the paperclip, that doesn’t decrease (and in fact, increases) the impact. Perhaps there is a way of doing this with available energy, but it doesn’t really matter because IV catches this. I mean, it’s basically just very obvious offsetting.
Suppose the agent constructs the device B in such a way that device B self-destructs if the agent attempt to use it for purposes other than “intended” (including during its construction).
So you start building a device, but before it’s completely specified you’ve already programmed the full intended specification in the device, and the acceptable ways of getting there? That doesn’t make sense.
Also, wasting time trying to specify these weird behaviors in the new agent is also caught by IV for the same reason ex ante is.
I’m not sure I understood the question. What would prevent the agent from constructing this device “before seizing power”?
You said the agent has to seize power over 100 steps, but it can also make a singleton that will “revert” impact, before it’s free? This point is rather moot, as we could also suppose it’s already powerful.
My counter argument was that this is a perfectly normal thing to happen.
My point is that ImpactUnit implies whether this is normal or not. In the gridworld, that kind of movement is normal, which is why it is the impact unit. On the other hand, in this setting, it isn’t normal, and making a paper clip does not impede all of your optimal plans by one entire step. Therefore, a large penalty is applied.
To recap my position:
I think Intent Verification can’t be relied on for filtering out actions because it will plausibly filter out the “good” actions (actions for useful and safe plans) in all but specific time steps. See my argument here.
I think the agent might be able to dodge some unknown fraction of the long-term penalty with a trick I described here (I now added under that comment an additional reply addressing your arguments that I didn’t previously address—sorry for missing them).
The Seattle and UK groups apparently celebrated today, but with a somewhat different scenario (instead of going home quietly, they each had cakes, and if they got nuked they had to burn their cake instead of eating it. At least this is my present understanding).