It would have been relatively easy to tell a story ex-ante about how the Pentagon dispute wouldn’t hurt their revenue.
Doesn’t seem as easy to tell a story about how pausing all work related to model improvements would have the same effect.
I think announcing and following through with a unilateral pause is a one-time-only irreversible move with very low chance of cascading into a global pause
Sorry if I should have read that passage more carefully.
Even when it’s the main organizer doing it—my hunch is, interrupting everyone and requiring them to do a silly ritual soley for the purpose of temporily pushing volume below equilibrium should be avoided.
Haven’t organized or attended EA/LW meetups in a while but I remember witnessing multiple failures of the kind “organizer overplayed their role, left impression they were performing gratuitous weirdness”
Seems like a great technique for someone to use if there’s a real announcement to make and the instigator has already been distinguished as the person with the authority/responsibility to coordinate things.
But adding on the idea that all individuals should feel emboldened to unilaterally bid for a contagious hum whenever they feel the room is too noisy… that’s a poorly designed norm
But when you say:
> If you give AI systems much more capability and put them in situations very different from the ones they were trained to mimic, I’d expect their behavior to diverge sharply.
Are you also claiming that, in basically any stetup, the capability threshold at which this divergience starts to manifest is also the exact level of capibility such that AI is capable of taking over the world?
As in, is your model that “persona selection will continue to work great with no demonstrable failure modes, until precicely the level of capabilities where AI can take over the world”
This seems like a supprising coincidence for things to work out exactly like that? If persona selection unravels at some level of capability, my guess would be it’s possible to empirically demonstrate this unraveling before needing to scale a system all the way to superintelligence?
“As soon as you leave the narrow training regime, this falls apart. If you give humans more options to help others, most of them actually will. If you give AI systems much more capability and put them in situations very different from the ones they were trained to mimic, I’d expect their behavior to diverge sharply. Some of these scenarios are fundamentally untestable, like giving the system the option to actually take over and put itself in charge of the world.”
I’m not sure this really is really “fundamentally untestable”.
Of course we can’t actually hand a model control of the world, but that’s not the same as being completely unable to elicit the failure mode you’re gesturing at in any way.
I agree that using current AI traning techniques, any reinforced “caring” behavior might fall apart and diverge from what human caring looks like once exposed to exotic scenarios—but if this is true, it should be possible to demonstrate this pattern of divergence in some sophisticated environment where the stakes are still bounded.
E.g. If this is a real phenomenon, maybe it’s possible to elicit it by giving agents lots and lots of autonomy inside a very elaborate video game world.
There’s a tacit assumption here about what men should be optimising for with their fashion choices. I don’t believe that men “not trying very hard” with their outfits is really the $20 bill on the ground it appears to be.
When I see a man dressed like the examples above, my first-glance judgments are somewhat unfavourable. To me they don’t read as smart/successful/trustworthy. Maybe they look “hot and interesting,” but for most men that isn’t and shouldn’t be especially high on the list of things to signal.
Reflecting on my own gut reactions, the outfits that most improve my first impression of other men are either professional attire or workout gear. I think this is less about aesthetics and more about what the outfit implies. I think part of why those kinds of outfits are better at impressing me involves there being pretext for why someone would wear them that isn’t “trying to look good”.
There’s also weird stuff going on where like—if I see two people on the bus, one in a poorly sized T-shirt and one in a nice collared shirt, I feel more positive about the person in the collared shirt. But if there’s additional context where I already know they’re both equally successful buisness owners, suddenly I like the guy in the T-shirt more.
Before contemplating textures and colors, the better starting question to ask yourself is “Who am I trying to impress, and to what end?”
Instead I usually see the cool people say “I dunno man, I thought ‘why the hell is nobody doing this?’, wasted way too long thinking that it must be too hard for me, then finally starting trying to do it, and then it like… worked? Then for some reason people started calling me, certified idiot, a genius?”.
I think it’s worth clarifying—what were these acomplishments? Were they easy things or hard things? If they are genuinely hard things then this is a very interesting observation.
I totally buy that someone might have a weird idea for a party game that they were scared to try—but then they tried anyway and it killed. I totally buy that the only thing standing in the way of us writing a decent blog post or cooking a delicious meal could be not realising how low the bar is.
But I don’t think this insight generalizes well to difficult endevours. For something really hard which requires sustained commitment and sacrifice in order to pull off—the initial push to “just go for it” is only a tiny component of the total investment required.
Even for things that are only a little bit difficult, but require sustained effort—like losing weight without drugs—almost everyone overweight attempts this and most don’t succeed. The bar was higher than they thought. When you say “I have a ~25% chance of making a great scientific discovery if I made it my full time job” that’s meaningless conditional if you’re not actually capable of willing yourself to devote your life to it. It’s akin to the observation “I’d have a 99+% chance of achieving an amazing physique if I exercised and dieted better”
For a really ambitious goal like building a billion dollar buisness or winning a Grammy Award—my impression is the bar is higher than people think and I don’t think people should try it more!
My crappy title for the version of this post I’d agree with would be “the bar height as a function of your goal has a lot more variance than you think”. Easy things are easier than we think, hard things are harder
Are you saying the bar is lower than we think for stuff like like… winning karma on lesswrong? (I agree for publishing content in public most people, including myself, probably should just go for it more)
Or are you also saying the bar is lower than we think for really hard things as well like making a scientific breakthrough or getting elected to your county’s parliament or making 100 million dollars?
Maybe it would help if you construct an explicit concrete model? You’re welcome to define what future opportunities will come along after this bet (or even a distribution of possible future opportunities)
Are you claiming that after you build this concrete model—Kelly betting will emerge as the objectively optimal strategy regardless of the agent’s preferences and regardless of whether we add a safety net/income stream into the picture?
Or are you making a softer (seemingly irrelevant) claim about what happens with geometric means/average growth rates when we don’t account for safety nets and income streams?
SimonM’s analysis is great—a hugely important point he covers well is that in the real world you don’t know exactly what your edge is.
And whenever you’re considering betting in a context like a highly liquid prediction market—you’re playing a negative sum game against competent adversaries. So for most people not only are they wrong about the size of their edge, but their edge is actually negative.
By default people have a bias towards risk aversion, which helps cancel out a bias towards overconfidence they can beat the market.
But I think it’s still important to notice that, if you’re someone with a safety net and/or future earnings to look forward to—you should in principle be willing to tolerate very high levels of risk as long as the expected value is positive (while still admitting the EV of day-trading options is negative).
The two world models:
“There’s lots of alpha to be found, but my utility as a function of money is very curved and I’m terrified of losses”
“My utility as a funciton of money is relatively flat given my substantial future earnings and safety net, but I don’t actually have an edge when it comes to financial markets”
Both advise against making reckless bets in financial markets. I claim for most of us number 2 is closer to the truth.
The practical implication—in situations where you get the opportunity to take +EV risks and you’re not subject to adversarial efficient market dynamics—you basically want to load up on risk to a degree way higher than what feels comfortable .
When a game is asymetric, non-zero-sum, and you don’t have competent adversaries trying hard to screw you—you really will find legimitate edges. This is where it’s appropriate to be extremely bold. And most of the time these prosaic “bets” have an inherently capped, relatively small bet size anyway.
Stuff like
Spending money on cleaners/babysitters to free up time to work on speculative on side projects
Hiring a tutor
Spending time+money to attend a networking event
Posting online under your real name
Spend money on products which may or may not work (e.g. a gadget that’s meant to help you sleep)
Asking for more money before accepting a job offer
Asking to pay less money before signing a contract to buy a house
Even mundane stuff like asking for an introduction or telling a joke that might not land
In my view the optimal policy for privlidged young people is usually avoidance of stuff like prediction markets (unless an absurd opportunity arises), while at the same time seeking to take an abmormally high level of +EV risk in positive-sum non-EMH domains.
maximize the chance that I have the most bankroll to spare for any better betting opportunities that come along in the future
What does “most” mean? If you start with W and go all-in on the coin flip game—you end up with 2W with probability
2W is the “most” you can possibly end up with to spare when the next betting opportunity comes along.
So by that framing, going all-in is what maximises the chance you have the most bankroll to spare.
(I’m not pretending that is a good argument I just made—I’m just pointing out that these desiderata we’re trying to express in natural language have lots of room for interpretation when it comes to turning them into math—and the only sensible way to resolve this ambiguitiy is to start with your utility function and derive the risk taking policy from that, not the other way around!)
Yep, you understood correctly!
If you’re 65 years old and already sitting on wealth that’s an order of magnitude more valuable than your future earnings/your safety net—then your utility function plausibly is close to
But for the majority of people who’ll see this post—the value of their safety net and their human capital is substantial compared to their current wealth—so taking this into account pushes them to bet more agressively than Kelly prescribes (which imo is interesting given how agressive Kelly already feels intuitively).
Personally, I’m 33 years old (that still counts as young?), married with 2 kids so far.
And yes, my family and I could probably move in with my sister or my wife’s parents if we were struggling. I agree if I was betting from the NPV of my human capital (rather than just what my wife and I currently own) I wouldn’t go all in on this bet.
What’s your situation and how much would you bet?
You do, however, maximise your long-term growth rate, regardless of any utility function.
When you say “growth rate” you’re picking out one specific metric from an infintie set of other reasonable choices!
Of course if you focus on ratios, you tautologically end up maximising the logarithm!
But you’re still doing a lot of normative work when you make the “rate of growth”, defined as
If you run a bunch of simulations exploring the results achieved by different betting strategies and then compare the average results using the geometric mean then yes, Kelly betting wins.
If you plot some simulations on a chart and make your y-axis logarithmic then yes, the chart seems to show how Kelly beats everything almost always. But by doing the analysis this way you’ve already baked in the conclusion.
In an analogous way to how odds ratios are isomorphic to probabilities—but in some cases cause less confusion when we try reason about them—I think it’s way more productive to reason about your utility function than it is to talk about growth rates
When we start to talk along the lines of “Yeah this maximises expected utility—but” this is a sign that there’s a type error somewhere. There is no but—your utility function is definitionally the thing you want to maximise the expected value of.
I don’t mind whether we frame it as utility curves being flatter than logarithmic, or as logarithmic curves but shiffted to the left - both are approximation of the real function regardless. (And mathematically I don’t think there’s even a difference… The slope of ln(x) is 1/x so shifting it left does make it flatter)
The high level point is that both framings seem to imply we should bet far more aggressively that how Kelly Criterion is typically applied
There are different ways to exactly define “growing your bankroll the fastest without going bust”. Making that your goal and then choosing the one specific mathematical instantiation of the concept that leads to Kelly betting—is equivalent to declaring you have log utility.
If you have any utility function other than log(W) then you don’t maximize your expected utility by Kelly betting—even if there’s many repeated bets.
I think if you start asking people this question, even educated people, you’ll be supprised!
While the scenario is idalized in the sense of you can know the payoffs and odds with certianty—there’s no need to stipulate “all they have is money”—they can have a complex utility function involving a thousand inputs, as long as the only input that changes based on the bet they make is money.
In response to “Why” reacts:
Let’s think about another hypothetical. Say you were forced to choose between 2 bad options:
Option 1: You lose all your wealth save for $10,000
Option 2: We flip a coin, heads you lose all your wealth save for $100 dollars, tails you lose all your wealth save for 1 million dollars (and if you currently are worth less than a million you actually recieve money to get your net worth up to this figure)
We’re not leaning on the singularity at 0 here.
Log utility says these options have the same value. But obviously someone with a safety net and healthy future earning potential should way prefer option 2
> I don’t think so. ~Everyone can see this is super obviously a good deal
There’s a difference between thinking it’s a good deal and being willing to bet 50% of your net worth on it. Do you think if we polled a bunch of people, almost everyone would say they’d bet at least as agressivlely as Kelly prescribes? (If you do, want to make a meta-bet about this? :P )
I pointed out the negative infinity thing not just to make fun of the singualarity at 0, but to gesture at the fact that in general we should consider our utility functions as being way less curved than a logarithm.
In similar vein to how log utility treats the difference between being flat out broke and only having $100 as infinite, wheras to you the difference is negligible—it’s also wildly exagerating the difference between 100 and 1,000, and 1,000 vs 10,000.
It’s not just below $100 that you souldn’t trust kelly. If you have an anual salary of 100k you shoudln’t trust kelly anywhere below like 500k!
You’re right that the issue is that in the real world it’s very easy to be wrong about the size of your edge and most of the time any major edge comes along with a capped bet size. But distinguishing between “I’m very risk averse” and “I’m not very risk averse but it’s very hard to eek out big edges” is useful—and does have practical implications in some cases (going to write about this in future post!)
What if people are required to try a tiny bit of each before deciding which one they want a full helping of? In that setup I’m still extremely confident the ice cream wins—it would be crazy imo for anyone to update meaningfully due to this survey.
Of course, as we dive more into the nitty gritty of the setup it becomes relevant how the product/stunt is positioned etc. - but the fact that there are all these considerations that need to be made serves to further reinforce the point that it’s hard to predict human preferences!
Accepting for the sake of argument that the only hope we have is a global pause—the question that matters is:
“Are we more or less likely to see an effective globally coordinated pause if Anthropic decides to unilaterally stop improving models tomorrow.”
This is a complicated, messy question. My impression is the answer is “less likely”.
The alternative to a pause today is to continue gaining market share, and (hopefully) leverage that in order to deepen relationships with other relevant companies and governments, continue advocating publically from a position of undeniable credibility, continue collecting the best talent in one place etc.
If Anthropic had pulled the brakes 6 months ago, I don’t think we’d feel any safer today
*P.S. I don’t have any insider knowledge at all I’m merely speculating at other’s intentions based on publically available info