Hypothesis I am not confident about: maybe Silicon Valley usage is a big slice of the cake, but it’s stable because efficient power users keep agents running at night or do shifts to manage them? And so the variation in demand is driven by non-power-users like employees at non-tech companies.
I keep hearing that Dario’s style comes off as extremely stubborn, arrogant and condescending
In all video appearances that I have seen, Dario clearly looks like a weirdo. I would probably love the guy in person, but I also know that many people just don’t love that kind of thing, and I’d guess Dario just blurts out things and doesn’t control his face as much as normies, nevermind people adept to negotiation and politics.
The field of AI safety is more than playing with LLMs, though recently it may look like just that, so I think your opening is a bit of an hyperbole. Agree with the general thrust.
Dakka Does Not Disproportionally Motivate The Poor
You would think that it would do this. We all assume they would do this. One of the objections to fixed payments for parents is that this could result mainly in poor people having more kids, since you would assume richer parents would respond a lot less.
I don’t think this would hold if you scaled the amounts, but also I’m surprised by the result in the first place?
Lyman Stone: So I went back through all those studies to check and see if they reported heterogeneity tests. Spoiler: many did. In total, from 60 initial intervention estimates, I was able to find 23 estimates of effects specifically on low-income people, and 24 for middle/high.
…
So we use that standard cost estimate from my study in May, and we apply it symmetrically to lower- and higher-income treated groups alike? This is basically saying “For a fixed budgetary spend, which group boosts fertility more?”
There’s basically no difference: if you give a middle/high earner $3,000 to have a kid, their odds of having a kid rise by essentially the identical amount as if you give a low earner $3,000 to have a kid.
A lot of people have this mental model where effects should always scale with income, but this is obviously wrong for a lot of effect types! If I give somebody a $10 hamburger voucher, poor people will not by 4x as many hamburgers even though it’s 4x the share of their income.
The reason is that it is a voucher for hamburgers and the price of hamburgers is relatively fixed with respect to income. Sure rich people may go to a bougier place and poor people a cheaper one, but by and large most people are going to purchase pretty similar hamburgers.
Despite the cultural narratives purporting to tell you there are absolutely massive class differences in parenting, the actual fact of the matter is the economic costs associated with how people would prefer to parent are actually not that variable across most of the income spectrum.
There are obvious exceptions. Private schools, extra space and hired help are obvious ways in which richer people will spend a lot more money. Despite this, the differences are not that vast, and we get this very interesting result.
It looks to me like the plot does not match the text. Maybe I’m not reading it right?
On the x axis, the plot has “benefit as % of income”. On y, it has “increase in probability of having a child”. The plot shows that the trend line for rich people is the same as the trend line for poor people. Since x is in % of income, this means that if you make 2x, you need 2x benefit to increase birth probability by the same amount. So it would be proportional to income, instead of independent like the text says.
Sorry for being cursory, by “No” I meant to reply to the question in yourAorou’s comment “Isn’t this post an elaborate way of saying that today’s posteriors are tomorrow’s priors?” ⇒ “No, I don’t think that’s the point of the post”. I separately think that yes, today’s posteriors are tomorrow’s priors.
Pragmatically, in data analysis tasks, what you do is a separate preliminary data collection that you only use to decide the priors (the whole data analysis structure, really) and then collect data again on which you run the actual analysis. This applies to non-Bayesian data analysis as well. This duplicated data collection helps you stay objective and not sneak into the prior any information which would not be Bayes-kosher to glean from the data. Of course it’s less efficient because you are not using all the data in the final analysis.
I don’t know anything about the case beyond what you say in your comment; from that my first though is that it would take time for them to get used to do everything with less emotions. If they had been born with that low level of emotions, they would have learned how to function that way already.
AI safety training applications are my intellectual clock.
In this period of my life I spend some time applying to various AI safety short courses and programs like MATS. Apply to all the good acronyms, as they say. The applications often declare themselves to take 1-2 hours to fill in. I invariably put in something more like 10 hours. A hypothetical application could start asking some euphemistic form of
> <what do you want to do with your life?>
Oh man, I don’t even know what I ate for dinner yesterday. I pace back and forth through my study, then decide to postpone completing the application to the next day. I fall asleep, mulling over the meaning of life.
The next day:
> <is AI going to kill us all?>
...having a p(doom) always stroke me as a bad idea, turns out Yudkowsky agrees but also has a p(doom) apparently, which I think is decision-theoretically consistent with him believing everyone else to be an idiot, amongst other possible explanations; should I make up some bafflegab about decision theory? Maybe I should just say “maybe”. This is the easy question.
> <do you know how to improve the state of the art on this thing below?>
Oh yes, I was chatting about this yesterday with von Neumann and my sister, she said to just use quantum computronium, and check the error bars.
I pace back and forth the study, sit down, sit up again, pace, sit down, re-open the laptop, sit up, close the laptop, pace, open the laptop, sit down, up, feint pacing but then close the laptop before it can realize. I go to bed and think. After one hour, I conclude I’m going to have milk next breakfast as I do every day since I was 6, and fall asleep.
Next day. Answer the question. I improved the state of the art, yes, yes. Next question.
> <do you like more your mom or your dad?>
Fear leads to anger, anger leads to hate, hate leads to suffering, suffering leads to the dark side. So I guess mom?Perhaps I should get some sleep again, as night is the mother of counsel, and write the answer from scratch tomorrow.
Tomorrow:
> <please predict the future, explaining your reasoning (100 words, no cheating)>
Uhm so I guess I was supposed to cheat on the other questions. Better go back and rewrite all answers with cheating. Prompting is Turing-complete, so there should be some way to prompt gpt5 to output my answers; would that count as cheating?
...aaand 10 hours have passed!
I already know about all these important issues before getting through the applications. But being asked about them in a bland anonymous form I fill in while bored bears on me the psychological effect of a stranger stopping me on the road, grabbing my arms and shouting “what is the meaning of life, QUICK” as they jolt me. In no other context I am compelled to answer such questions (in 100–200 words). So I take the occasion of each application deadline to sit down and think about my life.
I was referring to the fact that you set LessWrong posts with karma thresholds as target metrics. This kind of thing has in general the negative side effect of incentivizing exploitation of loopholes in the LessWrong moderation protocol, karma system, and community norms, to increase the karma of one own’s posts. See Goodhart’s law.
I do not think this is currently a problem. My superficial impression of your experiment is that it is good. However, this kind of thing could become a problem down the line if it becomes more common. This will be born out as a mix of lowering the quality of the forum and increased moderation work.
Now that the value of OpenAI minus the nonprofit’s share has tripled to $500 billion, that is even more true. We are far closer to the end of the waterfall. The nonprofit’s net present value expected share of future profits has risen quite a lot. They must be compensated accordingly, as well as for the reduction in their control rights, and the attorneys general must ensure this.
I think this reasoning is flawed, but my understanding of economics is pretty limited so take my opinion with a grain of salt.
I think it’s flawed in that investors may have priced in the fact that the fancy nonprofit, the AGI dream, & whatnot, where mostly a dumb show. So 500 G$ is closer to the full value of OpenAI, rather than close to the value left out of the nonprofit according to the current setup interpreted to the letter.
To make this the case, the new paradigm should, when properly studied and optimized, lead to more efficient AI systems than DL, below the threshold where it stops scaling. The alternative paradigm should allow to reach the same level of performance with less compute spent. For example, imagine the new paradigm used statistical models with a training procedure close to kosher Bayesian inference, thus having a near-guarantee of squeezing all the information out of the training data (within the capped intelligence of the model).
I now think that LLM pre-training is probably already pretty statistically efficient, so nope, can’t do substantially better through the route in this specific example.
A general way my mental model of how statistics works disagrees with what you write here is on whether the specific properties that are in different contexts required of estimators (calibration, unbiasedness, minimum variance, etc.) are the things we want. I think of them as proxies, and I think Goodhart’s law applies: when you try to get the best estimator in one of these senses, you “pull the cover” and break some other property that you would actually care about on reflection but are not aware of.
(Not answering many points in your comment to cut it short, I prioritized this one.)
Bayesian here. I’ll lay down my thoughts after reading this post in no particular order, I’m not trying to construct a coherent argument pro/against the argument of your post, not due to lack of interest but due to lack of time, though in general it’ll be evident I’m generally pro-Bayes:
I have the impression that Yudkowsky has lowered his level of dogmaticism since then, as is common with aging. I’ve never read this explicitly discussed by him, but I’ll cite one holistic piece of my evidence: that I’ve learned more about the limitations of Bayes from LessWrong and MIRI than from discussions with frequentists (I mean, people preferring or mostly using frequentist stuff or at least feeling about Bayes like a weird thing, not that they would be ideologues and not use it). Going beyond Bayes seems like a central theme in Yudkowsky’s work, though it’s always framed as extending Bayes. So I’ll take a stab at guessing what Yudkowsky thinks right now about this, and it would be that AIXI is the simplest complete idealized model of Bayesian agent, it’s of course a model and not reality, and general out-of-formal-math evidence aggregation points to Bayes being substantially an important property of intelligence, agency, knowledge that’s going to stick around like Newton is still used after Einsten, and this amounts to saying that Bayes is a law, though he wouldn’t today describe it with the biblical vibes he had when writing the sequences.
I empirically observed Bayes is a good guide to finding good statistical models, and I venture that if you think otherwise you are not good enough at the craft. It took me years of usage and study to use it myself in that sense, rather than just using Bayesian stuff as pre-packaged tools and basically equivalent to other frequentist stuff if not for idiosyncratic differences in convenience in each given problem.
I generally have the impression that the mathematical arguments you mention focus a lot on the details and miss the big picture. I don’t mean that they are wrong; I trust that they are right. But the overall picture I’d draw out of them is that Bayes is the right intuition, it’s substantially correct, though it’s a simplified model of course and you can refine it in multiple directions.
Formally, frequentist estimators are just any function. Though of course you’ll require sensible properties out of your estimators, there’s no real rule about what a good frequentist estimator should be. You can ask it to be unbiased, or to minimize MSE, under i.i.d. repetition. What’s if i.i.d. repetition is not a reasonable assumption? What if unbiased is in contradiction with minimizing MSE? Bayes gives you a much much smaller subset of stuff to pick from in a given problem, though still large in absolute terms. That’s the strength of the method; that you should not in practice need anything else out of that much smaller subset of potential solutions for your practical inference problems. They are also valid as frequentist solutions, but this does not mean that Bayes is equivalent to frequentist, because the latter does not select so specifically those solutions.
OLS is basically Bayesian. If you don’t like the improper prior, pick a proper very diffuse one. This should not matter in practice. If it happens to matter in some case, I bet the setup is artificial and contrived. OLS is not a general model of agency and intelligence, it’s amongst the simplest regression models, and it need not work under extreme hypothetical scenarios, it needs to work for simple stuff. If I ran OLS and got beta_1 = 1′000′000′000′000, I would immediately think “I fucked up”, unless I was already well aware of putting wackily unscaled values into the procedure, so a wide proper prior matches practical reasoning at an intuitive level. Which does not mean that Bayes is a good overall model of my practical reasoning at that point, which should point to “re-check the data and code”, but I take it as a good sign for Bayes that it points in the right direction within the allowance of such a simplified model.
Thank you for the many pointers to the literature on this, this is the kind of post one gets back to in the future (even if you consider it a rush job).
Hypothesis I am not confident about: maybe Silicon Valley usage is a big slice of the cake, but it’s stable because efficient power users keep agents running at night or do shifts to manage them? And so the variation in demand is driven by non-power-users like employees at non-tech companies.