Dumping out a lot of thoughts on LW in hopes that something sticks. Eternally upskilling and accelerating.
I write the ML Safety Newsletter
DMs open, especially for promising opportunities in AI Safety and potential collaborators.
Alice Blair
Karma: 378
This does seem to be getting closer, yes. I still think the models are overall too stupid to do meaningful deception yet, although I haven’t yet gotten to play around with Opus 4. My use cases have also shifted in this time to less hackable things.
Shrug, it does work for long spans (usually months before I need another) for me. This was a recent patch to a recent problem, but I’ve had this technique for years and it does in fact get me out of positive feedback loops of hedonic set point raising, no ketamine required. If I had to guess why it lasts, I’d say that it serves as a good reminder and willpower booster that allows me to resist further really-useless superstimuli.
Notes from Dopamine Detoxing
Oops, I wrote that without fully thinking about diffusion models. I meant to contrast diffusion LMs to more traditional autoregressive language transformers, yes. Thanks for the correction, I’ll clarify my original comment.
This seems like it’s only a big deal if we expect diffusion language models to scale at a pace comparable or better to more traditional autoregressive language transformers, which seems non-obvious to me.
Right now my distribution over possible scaling behaviors is pretty wide, I’m interested to hear more from people.
LLMs also just have their own quirks, and I think Qwen might just really like hell and cats?. For example, Claude Sonnet seems to really like bioluminescence as a topic, reliably enough in different instances to where Janus gets some impressive predictive accuracy.
I think that these are all pretty relevant ways to think about being an EA, but are mostly of a different fundamental type than the thing I’m pointing at. Let me get a bit more into the aforementioned math to show why this is approximately a binary categorization along the axis I was pointing at in this post.
Say that there are three possible world states:
I live a normal life and sit on the couch a lot. I take care of my plants.
I’m a highly engaged EA for many years and do a lot of verifiably altruistic things.
I’m a serial killer.
As a culture, on average, our values look something like:
EA > gardener > murderer
This is very reasonable and I don’t think we can or should change this, as a community.
I have some mental approximation of a utility function. One of the main differences between my internal representation and an actual utility function is that the point I choose as “zero utility” doesn’t matter in the formalism, but very much matters to my emotions. If we set EA=0 utility, then the myopic point-maximizing part of my brain feels okay if I do EA things, but awful if I’m getting negative points by being either of the other options. This is the moral obligation frame, where things are only barely emotionally okay if you push as far up the preference ordering as possible.
If we set gardener=0, then things feel emotionally okay if I just take the normal path. I’m not gaining or losing points. It’s then positively great if I do EA things and still positively bad if I kill people. This is the moral opportunity frame, and I find it emotionally much better for me. I predict that this frame is better for community health as well, although I have only vibes and anecdata to back me up on this claim.
There are several other points I have left unnamed:
murderer = 0: I really really don’t want to be this culture for reasons that are hopefully obvious.
gardener<0<EA: this is just a less extreme moral obligation framework, where you don’t need to do as much for things to be okay.
murderer<0<gardener: this is a more lenient moral opportunity frame, where doing some less-than-default-amount-of-good things occasionally is still okay. I think there are healthy and unhealthy versions of this
everything is >0: same as before, please don’t make this a culture
everything is <0: I also really don’t want this culture, because then everything anyone does is bad and this is not how you build a community.
Now that’s a lot of possibilities. I promised that I had “approximately a binary categorization”, so where’s the binary come in? Well, the dividing line I’m drawing is ultimately is “is being a normal member of society ‘okay’ or ‘not okay’?” Alternatively, we ask the question of how our community responds to Carol, the gardener. Are we friends with her? I say yes. Do we give her the same positive reinforcement for her new daffodils that we give to someone when they donate a large sum to EA charities? I say no.
(I am intentionally neglecting the various “evil” cultures from this consideration. Technically it’s not a binary if you want to include them, but I really don’t see why we would ever consider doing that in real life.)
These other frames you mention are important shards of a healthy EA community, I think. They’re just not quite the concept boundary I was trying to draw.
Edit: this comment seems to be incorrect; see comments.
They did not say that, on my reading? The footnote about bulk preorders says
Bulk preorders don’t count. The people who compile bestseller lists distinguish and discount bulk preorders.
Which I read as semantically equivalent to “bulk preorders neither help nor hurt” with an implicit “so don’t do a bulk preorder solely for the sake of making the numbers better, but other reasons are acceptable.”
Retracted my previous response due to faulty reasoning.
Yes, positive reinforcement gives a more precise signal than positive punishment. Maybe I didn’t elaborate on this fact enough, but that is the point. I like it when a community precisely signals what its values are. I like it when a community isn’t adversarial to bystanders. These are, among other reasons, precisely why I want a positive reinforcement culture. The point of the post is that these two frames are different. You pointed out a difference in the post that I discussed.
Positive reinforcement, however, is not more restrictive. There is no restriction going on, that’s the point. It is just a more precise signal. If I give you a cookie every time you do the dishes and nothing when you don’t, that is not restricting the counterfactual-yous who don’t do dishes from being in my community. Somebody else just does the dishes and gets the cookie.
No, that’s not quite what I’m saying in my post. I’m pointing at the times when a culture could either positively reinforce X or positively punish not-X, which is exactly the same amount of restriction.
Maybe, one could argue, in practice positive reinforcement creates a more precise signal, which on my model is a good thing; I like communities that clearly signal their values and aren’t adversarial to bystanders. I really don’t think this is a trick, I think the post is in fact about the differences between these two framings, including about the reinforcement/punishment distinction.
Moral Obligation and Moral Opportunity
For people who are on the fence about donating and want an outside opinion:
I think CAIP is one of the best orgs at what they do in DC, and I think that what they do is important for many of the reasons Jason laid out above. I continue to think efforts for good AI governance is one of the highest-leverage things we can do for AI safety right now. My experience is that the CAIP team is highly competent and very well networked across AI safety and the policy world.
My favorite review from a congressional staffer of one of the AI risk demos at a CAIP event is “that’s terrifying.” They’re concretely shifting opinion in DC and keeping things impressively nonpartisan while doing so.
I don’t work for CAIP, but I’ve collaborated with them in the past. Whether or not people donate doesn’t affect my finances or any of the work I do right now—I just think that they’re doing good work and should have the funding to keep doing it.
Did I entirely miss these? I can’t find the post anywhere. I am still interested in having more music.
I work mostly as a distiller (of xrisk-relevant topics). I try to understand some big complex thing, package it up all nice, and distribute it. The “distribute it” step is something society has already found a lot of good tech for. The other two steps, not so much.
Loom is lovely in the times I’ve used it. I would love to see more work done on things like this, things that enhance my intelligence while keeping me very tightly in the loop. Other things in this vein include:
memory augmentation beyond just taking notes (+Anki where appropriate). I’m both interested in working memory and long-term memory, but moreso in the former.
For digesting complex pieces, I’d like something better than a chat window for interactively getting up to my desired level of familiarity with some existing work. When I’m digesting the paper, do I want the one-sentence summary, the abstract, or to have a better understanding than the researchers behind it? NotebookLM is sort of doing this but I’ve found it lacking for this task for UI reasons (I also wish that it would automatically bring in other relevant sources).
Ai Labs is slightly better but still bad.
Could you give a link to this or a more searchable name? “Ai Labs” is very generic and turns up every possible result. Even if it’s bad, I’d be interested in investigating something “slightly better” and hearing a bit about why.
Clicking on the link on mobile Chrome sends me to the correct website. How do you replicate this?
In the meantime I’ve passed this along and it should make it to the right people in CAIS by sometime today.
I have not been able to independently verify this observation, but am open to further evidence if and only if it updates my p(doom) higher.
After reviewing the evidence, both of the EA acquisition and the cessation of Lightcone collaboration with the Fooming Shoggoths, I’m updating my p(doom) upwards 10 percentage points, from 0.99 to 1.09.
This seems very related to what the Benchmarks and Gaps investigation is trying to answer, and it goes into quite a bit more detail and nuance than I’m able to get into here. I don’t think there’s a publicly accessible full version yet (but I think there will be at some later point).
It much more targets the question “when will we have AIs that can automate work at AGI companies?” which I realize is not really your pointed question. I don’t have a good answer to your specific question because I don’t know how hard alignment is or if humans realistically solve it on any time horizon without intelligence enhancement.
However, I tentatively expect safety research speedups to look mostly similar to capabilities research speedups, barring AIs being strategically deceptive and harming safety research.
I median-expect time horizons somewhere on the scale of a month (e.g. seeing an involved research project through from start to finish) to lead to very substantial research automation at AGI companies (maybe 90% research automation?), and we could see nonetheless startling macro-scale speedup effects at the scale of 1-day researchers. At 1-year researchers, things are very likely moving quite fast. I think this translates somewhat faithfully to safety orgs doing any kind of work that can be accelerated by AI agents.
(I don’t think you included it) The blurb on corrigibility is also really good. Yet another thing that I’m not sure if Eliezer has actually written up anywhere else.