If you have any questions for me or just want to talk, feel free to reach out by sending a private message on this site or by sending an e-mail to egeerdil96@gmail.com.
You can also find me on Metaculus at https://www.metaculus.com/accounts/profile/116023/, or on Discord using the tag Starfall#7651.
This meta-analysis says that paracetamol is effective at reducing fever in a short time, but doesn’t seem to accelerate fever clearance in a statistically significant way. So it depends on what you mean by “reducing fever”; as short-term impact on fever is usually considered a secondary outcome in these studies.
It’s true that the mechanism of action is not well understood. However, depending on exactly how you interpret the claim above, it might be true that there is “scientific evidence” for it, or it might not be. Ditto for a claim such as “racial group differences in IQ tests are genetic”. What exactly are we talking about when we make this claim?
What’s “scientific evidence”? Is there scientific evidence that paracetamol helps with fever, for example?
This looks correct to me—this is indeed how the model is able to disentangle algorithmic progress from scaling of training compute budgets.
The problems you mention are even more extreme with dataset size because plenty of the models in our analysis were only trained on ImageNet-1k, which has around 1M images. So more than half of the models in our dataset actually just use the exact same training set, which makes our model highly uncertain about the dataset side of things.
In addition, the way people typically incorporate extra data is by pretraining on bigger, more diverse datasets and then fine-tuning on ImageNet-1k. This is obviously different from sampling more images from the training distribution of ImageNet-1k, though bigger datasets such as ImageNet-21k are constructed on purpose to be similar in distribution to ImageNet-1k. We actually tried to take this into account explicitly by introducing some kind of transfer exponent between different datasets, but this didn’t really work better than our existing model.
One final wrinkle is the irreducible loss of ImageNet. I tried to get some handle on this by reading the literature, and I think I would estimate a lower bound of maybe 1-2% for top 1 accuracy, as this seems to be the fraction of images that have incorrect labels. There’s a bigger fraction of images that could plausibly fit multiple categories at once, but models seem to be able to do substantially better than chance on these examples, and it’s not clear when we can expect this progress to cap out.
Our model specification assumes that in the infinite compute and infinite data limit you reach 100% accuracy. This is probably not exactly right because of irreducible loss, but because models are currently around over 90% top-1 accuracy I think it’s probably not too big of a problem for within-distribution inference, e.g. answering questions such as “how much software progress did we see over the past decade”. Out-of-distribution inference is a totally different game and I would not trust our model with this for a variety of reasons—the biggest reason is really the lack of diversity and the limited size of the dataset and doesn’t have much to do with our choice of model.
To be honest, I think ImageNet-1k is just a bad benchmark for evaluating computer vision models. The reason we have to use it here is that all the better benchmarks that correlate better with real-world use cases have been developed recently and we have no data on how past models perform on these benchmarks. When we were starting this investigation we had to make a tradeoff between benchmark quality and the size & diversity of our dataset, and we ended up going for ImageNet-1k top 1 accuracy for this reason. With better data on superior benchmarks, we would not have made this choice.
I think this question is interesting but difficult to answer based on the data we have, because the dataset is so poor when it comes to unusual examples that would really allow us to answer this question with confidence. Our model assumes that they are substitutes, but that’s not based on anything we infer from the data.
Our model is certainly not exactly correct, in the sense that there should be some complementarity between compute and algorithms, but the complementarity probably only becomes noticeable for extreme ratios between the two contributions. One way to think about this is that we can approximate a CES production function
in training compute and algorithmic efficiency when by writing it as
which means the first-order behavior of the function around doesn’t depend on , which is the parameter that controls complementarity versus substitutability. Since people empirically seem to train models in the regime where is close to this makes it difficult to identify from the data we have, and approximating by a Cobb-Douglas (which is what we do) does about as well as anything else. For this reason, I would caution against using our model to predict the performance of models that have an unusual combination of dataset size, training compute, and algorithmic efficiency.
In general, a more diverse dataset containing models trained with unusual values of compute and data for the year that they were trained in would help our analysis substantially. There are some problems with doing this experiment ourselves: for instance, techniques used to train larger models often perform worse than older methods if we try to scale them down. So there isn’t much drive to make algorithms run really well with small compute and data budgets, and that’s going to bias us towards thinking we’re more bottlenecked by compute and data than we actually are.
I would guess that making progress on AGI would be slower. Here are two reasons I think are particularly important:
ImageNet accuracy is a metric that can in many ways be gamed; so you can make progress on ImageNet that is not transferable to more general image classification tasks. As an example of this, in this paper the authors conduct experiments which confirm that adversarially robust training on ImageNet degrades ImageNet test or validation accuracy, but robustly trained models generalize better to classification tasks on more diverse datasets when fine-tuned on them.
This indicates that a lot of the progress on ImageNet is actually “overlearning”: it doesn’t generalize in a useful way to tasks we actually care about in the real world. There’s good reason to believe that part of overlearning would show up as algorithmic progress in our framework, as people can adapt their models better to ImageNet even without extra compute or data.
Researchers have stronger feedback loops on ImageNet: they can try something directly on the benchmark they care about, see the results and immediately update on their findings. This allows them to iterate much faster and iteration is a crucial component of progress in any engineering problem. In contrast, our iteration loops towards AGI operate at considerably lower frequencies. This point is also made by Ajeya Cotra in her biological anchors report, and it’s why she chooses to cut the software progress speed estimates from Hernandez and Brown (2020) in half when computing her AGI timelines.
Such an adjustment seems warranted here, but I think the way Cotra does it is not very principled and certainly doesn’t do justice to the importance of the question of software progress.
Overall I agree with your point that training AGI is a different kind of task. I would be more optimistic about progress in a very broad domain such as computer vision or natural language processing translating to progress towards AGI, but I suspect the conversion will still be significantly less favorable than any explicit performance metric would suggest. I would not recommend using point estimates of software progress on the order of a doubling of compute efficiency per year for forecasting timelines.
You would do it in log space (or geometrically). For your example, the answer would be .
Should society eliminate schools?
The question is too vague as it’s stated, but I think society should eliminate schools in their present form. This is a rather worthless statement though, at least unless it’s fleshed out by a reasonably detailed description of what that alternative world would look like.
I think it would be a substantial win to at least cut down the years of schooling on the margin and replace them with work and/or apprenticeships whenever possible. An uncontroversial example: the fact that physicians and lawyers in the US have to complete a whole separate undergraduate degree before going to medical school or law school seems like a colossal waste of time and resources, and many civilized places in the world get by just fine without this extension.
So on the margin, I think it’s good to move in the direction of “eliminating schools”. Whether you want to go all the way and what happens if you do is more complicated, though I think there are definitely more promising alternative systems that would qualify. These are more speculative and only of theoretical interest given where we currently are as a society, though.
Should we have more compulsory schooling?
On the margin, I don’t see how more compulsory schooling would help with anything useful, and the costs are significant, even aside from the moral concerns with forcing children to go to school et cetera. So the answer here looks fairly overdetermined to be “no” unless marginal years of schooling are shown to have substantial benefits.
Should you send your kids to school?
Depends on the situation. Do the kids want to go to school? Do you think careers that would be the best fit for them require one to go through some formal accreditation process that involves schooling? How feasible it is for you to arrange an alternative to going to school for purposes that are relevant, and what are the costs of not participating in the existing system?
I would put significant weight on the preference of the kids in question here, and I can easily imagine that some of them want to go to school and others don’t. A “one size fits all” policy seems inappropriate here.
Should you prefer to hire job candidates who have received more schooling, beyond school’s correlation with the g factor?
There are other reasons to prefer such candidates, but it depends on exactly which job you’re hiring for. People who are “competent” despite not going to school right now are all highly unusual people in various ways, and they might generally be unusual in a way that makes them poor fits for the specific job you have in mind. So in that case going to school would be a valuable signal above and beyond the correlation with g.
Should we consider the spread of education requirements to be a form of class war by the better-educated against the worse-educated which must be opposed for the sake of the worse-educated and the future of society?
Probably not. I don’t see what reason there is to invent such an explanation for the phenomenon of schooling, or what predictive power or utility it would have.
I find it more productive to view schooling and its shortcomings (as many other things) as coordination failures and problems imposed by scarcity than any kind of “class war” by some group against another. Useful thinking about these questions should contend with the coordination issues surrounding signaling etc. and the substantial opportunity cost of having high-quality teachers in too many classrooms.
I wanted to thank you for writing this comment—while I have also been reasonably active on social media about this topic, and playing level 3+ games is sometimes necessary in the real world, I don’t think this post actually offers any substantive content that goes beyond “fraud is bad and FTX was involved in fraudulent activities”.
I agree that it’s not a good fit for LW, though I think the post does fit in the EA Forum given recent events.
In my opinion, the fact that the original tweet, along with Bankman-Fried’s other tweets in the same thread, has been deleted in an attempt to shift the narrative is noteworthy in and of itself.
I think the tweet is more likely to be deleted in this way in worlds where the claim in it was made in bad faith rather than not, though perhaps unknown details about FTX’s strategy for getting out of this quagmire necessitated such a move even in worlds where the claim had originally been made in good faith. Still, the optics are not good, and I have appropriately updated towards FTX having engaged in dishonest conduct.
I agree with evhub that we don’t yet know if the claim was made in bad faith, but we can still make some guesses in the absence of such definitive knowledge.
Most of the people on freenode migrated over to libera.chat, from what I know. Is this true of the LessWrong channel?
It’s likely not an uncommon attitude among Eastern Europeans, given the relevant history. I’ve heard similar sentiments expressed by Czech and Polish people before.
I agree that in this context it seems nonsensical, though. Ukrainian national socialists wanting to take down communist monuments is quite overdetermined; as the monuments are not only symbols of an ideology that they strongly oppose, but also of a foreign state that was an occupying force in Ukraine in living memory (and is an occupying force right now, depending on your interpretation of the continuity between the USSR and the Russian Federation).
I think we’re not communicating clearly with each other. If you have the time, I would be enthusiastic about discussing these potential disagreements in a higher frequency format (such as IRC or Discord or whatever), but if you don’t have the time I don’t think the format of LW comments is suitable for hashing out the kind of disagreement we’re having here. I’ll make an effort to respond regardless of your preferences about this, but it’s rather exhausting for me and I don’t expect it to be successful if done in this format.
That’s in part why we had this conversation in the format that we did instead of going back-and-forth on Twitter or LW, as I expected those modes of communication would be poorly suited to discussing this particular subject. I think the comments on Katja’s original post have proven me right about that.
Yeah, that matches my experience with chess engines. Thanks for the comment.
It’s probably worthwhile to mention that people have trained models that are more “human-like” than AlphaGo by various parts of the training process. One improvement they made on this front is that they changed the reward function so that while almost all of the variance in reward is from whether you win or lose, how many points you win by still has a small influence on the reward you get, like so:
This is obviously tricky because it could push the model to take unreasonable risks to attempt to win by more and therefore risk a greater probability of losing the game, but the authors of the paper found that this particular utility function works well in practice. On top of this, they also took measures to widen the training distribution by forcing the model to play handicap games where one side gets a few extra moves, or forcing the model to play against weaker or stronger versions of itself, et cetera.
All of these serve to mitigate the problems that would be caused by distributional shift, and in this case I think they were moderately successful. I can confirm from having used their model myself that it indeed makes much more “human-like” recommendations, and is very useful for humans wanting to analyze their games, unlike pure replications of AlphaGo Zero such as Leela Zero.
I think you’re misunderstanding the reason I’m making this argument.
The reason you in fact don’t fix the discriminator and find the “maximally human like face” by optimizing the input is that this gives you unrealistic images. The technique is not used precisely because people know it would fail. That such a technique would fail is my whole point! It’s meant to be an example that illustrates the danger of hooking up powerful optimizers to models that are not robust to distributional shift.
Yes, in fact GANs work well if you avoid doing this and stick to using them in distribution, but then all you have is a model that learns the distribution of inputs that you’ve given the model. The same is true of diffusion models: they just give you a Langevin diffusion-like way of generating samples from a probability distribution. In the end, you can simplify all of these models as taking many pairs of (image, description) and then learning distributions such as P(image), P(image | description), etc.
This method, however, has several limitations if you try to generalize it to training a superhuman intelligence:
The stakes for training a superhuman intelligence are much higher, so your tolerance of making mistakes is much lower. How much do you trust Stable Diffusion to give you a good image if the description you give it is sufficiently out of distribution? It would make a “good attempt” by some definition, but it’s quite unlikely that it would give you what you actually wanted. That’s the whole reason there’s a whole field of prompt engineering to get these models to do what you want!
If we end up in such a situation with respect to a distribution of P(action | world state), then I think we would be in a bad position. This is exacerbated by the second point in the list.
If all you’re doing is learning the distribution P(action humans would take | world state), then you’re fundamentally limited in the actions you can take by actions humans would actually think of. You might still be superhuman in the sense that you run much faster and with much less energy etc. than humans do, but you won’t come up with plans that humans wouldn’t have come up with on their own, even if they would be very good plans; and you won’t be able to avoid mistakes that humans would actually make in practice. Therefore any model that is trained strictly in this way has capability limitations that it won’t be able to get past.
One way of trying to ameliorate this problem is to have a standard reinforcement learning agent which is trained with an objective that is not intended to be robust out of distribution, but then combine it with the distribution P such that P regularizes the actions taken by the model. Quantilizers are based on this idea, and unfortunately they also have problems: for instance, “build an expected utility maximizer” is an action that probably would get a relatively high score from P, because it’s indeed an action that a human could try.
The distributional shift that a general intelligence has to be robust to is likely much greater than the one that image generators need to be robust to. So on top of the high stakes we have in this scenario per point (1), we also have a situation where we need to get the model to behave well even with relatively big distributional shifts. I think no model that exists right now is actually as robust to distributional shift as much as we would want an AGI to be, or in fact even as much as we would want a self-driving car to be.
Unlike any other problem we’ve tried to tackle in ML, learning the human action distribution is so complicated that it’s plausible an AI trained to do this ends up with weird strategies that involve deceiving humans. In fact, any model that learns the human action distribution should also be capable of learning the distribution of actions that would look human to human evaluators instead of actually being human. If you already have the human action distribution P, you can actually take your model itself as a free parameter Q and maximize the expected score of the model Q under the distribution P: in other words, you explicitly optimize to deceive humans by learning the distribution humans would think is best rather than the distribution that is actually best.
No such problems arise in a setting such as learning human faces, because P is much more complicated than actually solving the problem you’re being asked to solve, so there’s no reason for gradient descent to find the deception strategy instead of just doing what you’re being asked to do.
Overall, I don’t see how the fact that GANs or diffusion models can generate realistic human faces is an argument that we should be less worried about alignment failures. The only way in which this concern would show up is if your argument was “ML can’t learn complicated functions”, but I don’t think this was ever the reason people have given for being worried about alignment failures: indeed, if ML couldn’t learn complicated functions, there would be no reason for worrying about ML at all.
I agree with this, but for the reason you specified I think that precision would be of greater utility elsewhere.
I don’t think this is the case, because actually the claim “paracetamol helps with fever” is fairly underdetermined.