I write software for a living and sometimes write on substack: https://taylorgordonlunt.substack.com/
Taylor G. Lunt
Karma: 544
Turning Grey
Minimizing Loss ≠ Maximizing Intelligence
Halloween Tombstone Simulacra
It was subtle. I could just sometimes sense the prickly, digital nature of my sense of touch if I moved it smoothly along e.g. a blanket.
I managed to find one of the Sam Harris clips where he talks about this
...even in the immediate aftermath of an atrocity like this. This will come as no surprise. They will tell you that this has nothing to do with Islam. It has nothing to do with heartfelt religious convictions. No, it has everything to do with capitalism and the oppression of minorities, and the racism of white people in Europe, and the racism of cartoonists at a magazine like Charlie Hebdo. That is the cause of this behaviour. That’s what causes someone to grab an AK-47 and murder 12 cartoonists and then scream Allahu Akbar in the streets. It is a completely insane analysis.
I get the sense that people still don’t understand what we’re dealing with here. Have you seen any of these interviews with captured Isis fighters? Religion is the whole story. They are totally fixated on getting into paradise. In fact, the Kurds have put female soldiers into the field, and this terrifies members of Isis because they believe that they won’t go to paradise if they get killed by a woman. They literally run away from these female soldiers.
That’s a fair point. I know who Paul Fussell is, so it was obvious to me he’s dead, but I understand how it’s clickbait from your point of view.
I edited the title.
Ah, but there’s another more powerful god who, annoyed with gods dishing out infinite rewards willy-nilly, will revoke them unless they were the reward for joining a religion specifically.
Sam Harris used to talk about how nobody listened to what Islamic terrorists said about their own motivations. People would make up stories about how the terrorist acts were being committed for geopolitical reasons, or because “they hate us for our freedoms”, but the actual terrorists were pretty explicit about their religious motivations. According to Sam Harris; I never looked into it any further.
Asking AI What Writing Advice Paul Fussell Would Give
Halfhaven Digest #3
Unsureism: The Rational Approach to Religious Uncertainty
By training GPU software I mean improving it in an indirect way, like how you train a neural net.
I think verification is part of training. You feed your intuition data, and use logic to detect when this process is going astray.
The GPU/CPU analogy is just to show that the intuition is a powerful parallel computing process, whereas the logical mind is not parallel and is computationally limited.
I think trying to create intelligence via gradient descent is a dead end, so we’ll have to switch to an entirely different and more expensive architecture. I’ll probably write a post about that soon.
Oh, yeah, I got confused. I originally wrote the post taking into account a growing population, but removed that later to make it a bit simpler. Taking into account a growing population with an extra 1 or 2 billion people, everyone dying later is worse because it’s more people dying. (Unless it’s much later, in which case my mild preference for humanity continuing kicks in.) With equal populations, if everyone dies in 100 or 200 years it doesn’t really matter to me, besides a mild preference for humanity continuing. But it’s the same amount of suffering and number of lives cut short because of the AI apocalypse.
My p(doom) is around 10% probably in part because I imagine a pretty slow takeoff. I don’t think 60% is necessarily unreasonable, nor 1% nor 99%. It’s hard to estimate from essentially zero actual information.
Honestly, the more I engage with this thread, the less certain I become that any of this conversation is productive. Yeah, that’s one way the future could go. It feels less like discussing whether a potential drug will be safe or not, and more like discussing how many different types of angels there will turn out to be in heaven. There’s just such little information going into this discussion that maybe the conclusion from all this is that I am just unsure.
Honestly, I’m not even sure we can call any of this a calculation, given the uncertainty. It just seems like a bunch of random guesswork. The main thing I’m learning from all this is how uncertain I am, and how skeptical of anyone who claims to be more certain.
I think it shouldn’t be hard to believe how a superintelligent AI could cure mortality. For example, it could quickly cure all diseases and biological aging, and dramatically reduce the incidence of accidents. Then we have lifespans of like 10,000 years, and that’s 10,000 years for the superintelligent AI to become even more superintelligent and figure something out.
I agree that everyone dies at some point, but if that happens in a trillion years, presumably we’ll at least have figured out how to minimize the tragedy and suffering of death, aside from the nonexistence itself.
I agree that accounting for suffering could possibly make a difference, but that sounds harder than just estimating deaths and I’m not sure how to do it. I’m pretty sure it will shift me further against a pause though. A pause will create more business-as-usual suffering by delaying AGI, but will reduce the chances of doom (possibly). I don’t expect doom will involve all that much suffering compared to a few decades of business-as-usual suffering, unless we end up in a bad-but-alive state, which I really doubt.
I agree that the numbers are so approximate as to be relatively useless. I feel like the useful part of this exercise for me was really in seeing how uncertain I am about whether or not we should have an AI pause. Relatively small differences in my initial assumptions could sway the issue either way. It’s not as if the cautious answer is obviously to pause, which I assumed before. Right now I’m extremely weakly against.
Yes, I am assuming, mostly for the sake of simplicity, that superintelligent AGI cures mortality immediately. I don’t think it would be likely to take more than 10 years though, which is why I’m comfortable with that simplification. I’m also comfortable using deaths as a proxy for suffering because I don’t expect a situation where the two diverge, e.g. an infinite torture torment nexus scenario.
I’m happy to bite this bullet.
Setting aside the fact that I personally fear death. So let’s imagine we’re talking about the universe ending either in 100 or 200 years (and population keeps growing during that time). I guess I would prefer the former, yes.
More people experiencing some horrible apocalypse and having their lives cut short sounds bad to me. Pausing would mean more years for the people who exist at the 100 year mark, but then you’re also creating more people later who will have their lives tragically cut short. How many young people would it be acceptable to create and destroy in a fiery apocalypse later, to give people who exist now more years?
If this factor is important to you, I encourage you to do your own napkin math that takes it into account! I don’t want anyone to think I’m trying to publish an objectively correct AI pause calculator. I’m just trying to express my own values on paper and nudge others to do the same.
Thanks for your detailed response. I agree that if we have enough data/compute, we could overcome the data/compute inefficiency of AI models. I suspect the AI models are so intensely data/compute inefficient that this will be very difficult though, and that’s what I tried to gesture at in my post. If I could prove it, I’d have written a white paper or something instead of a blog post, but I hoped to at least share some of my thoughts on the subject.
Some specific responses:
Just increasing compute. I agree this is why we measure loss, but that doesn’t imply that measuring loss will get us to superintelligence long-term. Also, for this: “benchmark performance, which is nonzero only for models large enough”, I think you could have benchmarks that scale with the model, like novel games that start simple, and grow more complex as the model gains capability. Either manually, or implicitly as with AlphaGo Zero.
Higher quality data. Thanks for bringing my attention to CALM, I’ll have to look into that. I don’t think using a not-so-intelligent LLM to check whether the student’s idea and the real idea are the same will work in the limit, for the same reason it would be hard to get a kindergartner to grade a high school math test, even if they had access to a correct version written by the teacher. (Assuming the test wasn’t multiple choice, or single numerical answers or something easy to verify.)
Using another smaller LLM as an evaluator. I’m definitely not against all approaches that use a smaller LLM to evaluate a larger LLM, and you’re right to push back here. In fact, I almost suggested one such approach in my “what might work” section. Narrow models like AlphaGo Zero do something like this to great effect. What I’m against specifically is asking smaller models to evaluate the “goodness” of an output, and trusting the smaller LLM to have good judgement about what is good. If it had to judge something specific and objective, that would possibly work. You want to trust the small model only for what it’s good at (parsing sentence structure/basic meaning of outputs, for example) and not what it’s bad at.
RLHF. RLHF works for what it does, but no amount of RLHF can overcome the problems with self-supervised learning I discussed in the post. It’s still a general “reality-stuffing” model. That’s all I meant.
Transformers and “attention”. I do not take the benchmarks like solving the IMO seriously. These same AI models fail to solve kindergarten math worksheets, and fail to solve very basic problems in practice all the time. In particular, it does not seem smart to test how well a model can think by giving it problems that may require a whole lot of thinking, or may require not much, depending on what similar things happened to be in the training data, which we have no idea about. You mentioned P=NP. Solving problems is much easier if you already know how to solve similar-enough problems. We don’t know what similar problems a given model does or does not know how to solve. Rendering the benchmark useless. Unless you construct a benchmark such that we know there can’t have been anything meaningfully similar in the training data (e.g. novel games). (I am unsure whether to take FrontierMath Tier 4 a bit more seriously because the problems seem really hard and unlikely to be similar to anything in the training data, but ideally you’d have a benchmark that works even for less difficult problems anyway.) As for your comment about online learning, I don’t think solving any particular task should require a model to totally reorganize its weights across the entire model. Updating only a little part of weights should be fine. An analogy to humans should show that much. I agree though that having to hold onto fine-tuned partial models for users, even briefly, is more expensive than what we’re doing now, but the capabilities gains may eventually be worth it if non-online-learning models do plateau.