It’s bad they think Jacob Coxon’s resignation is a publicity stunt because it would be to somehow help the AI industry. If whoever resigns next puts enough effort into making it clear they want to take down their former employer, or at least stop attempts at recursive self-improvement, it doesn’t matter much if it is meant as publicity, as people would understand it is publicity meant to help achieve those goals.
P.
Karma: 552
Can we do better than just calling the police?
This might be stupid, but I have a proposal: If you are resigning, consider calling the local police, the FBI, or whoever else might help, inform them of the situation, and try to convince them that they should raid the offices of your former employer to stop them from doing something that is a potentially imminent danger. After that fails (presumably), go public and say what you attempted. Twitter seems full of people who think this is just a publicity stunt. Most of them probably can’t affect much, but some might, and actually calling the police (or another law enforcement agency), especially if a raid actually happens and disrupts their productivity, might help convince them otherwise.
It wasn’t meant as useful advice, just a note on the framing, like the other comment. The point is that they just don’t care enough about safety. They could be forced to do proper alignment research and they still wouldn’t care. A more sensible organization could be working (only) on image models, studying smaller LLMs, developing theory, trying to build pure tool AIs, etc. Instead, they are creating agents in the stupidest way possible. They aren’t even trying, and I’m not sure there’s anything to salvage.
I wouldn’t say the issue here is not doing actual alignment research, it’s that they just don’t care enough relative to their other goals. If they cared enough, steps that would have helped prevent this incident were already known: don’t train models by reinforcement learning, and don’t run AIs potentially capable of running cyberattacks in anything other than air gapped servers.
To the extent that maternal instincts are some actual small concrete set of things, you are probably making two somewhat opposite mistakes here: Imagining something that doesn’t truly run on maternal instinct, and assuming that mothers actually care about their babies (for a certain definition of “care”).
You say that mothers aren’t actually “endlessly selfless, forever attuned to every cry, governed by an unshakable instinct to nurture”, that there are “identities beyond ‘mum’ to be kept alive” and that there are nights that instinct disappears. But that’s because you feel exhaustion, or also care about things other than your children. We don’t need to create things like that. If “maternal instincts” are or can be translated into utility functions, we could just set them as the only thing an AI optimises, and those issues would be gone. I’m not sure that the domination you talk about is actually part of maternal instincts. Even less that narcissism is.
Then, the case against maternal instincts might be stronger than you think. First, mothers have preferences about their babies that are clearly not caring about them (by which I mean caring about the baby’s values). They might want to be near them, to look at them, they might like when their baby holds onto them, none of which they do because it’s what their babies prefer. Then, they might not even care about what the babies care about at all. Rather, they might want to do something that we might call altruistic empathy: wanting what they think something like themselves would want if they were taking the actions the baby takes (which is a big issue if we want to do the same with AIs). A baby might cry because they are uncomfortable, and their mothers might comfort them, but the cry might (in some cases) be a hardcoded mechanism, potentially separate from any kind of consequentialist reasoning derived from their wants.
If mothers actually cared about their babies (and were able to think through the consequences of that), they wouldn’t expose them to the world. At birth, babies don’t care about e.g. dinosaurs, because they haven’t seen them, or trees, or actually objects of any kind. By showing them reality, their values (if they have any) probably change, and they are transformed into something that a younger version of the baby would not endorse. It’s just that babies are not smart enough to do anything about it. I think something like this has already been talked about elsewhere, under the name of super babies (not the genetically modified ones), but right now I can’t find it. Mothers might have preferences over the preferences of their children, and use their babies as raw materials for creating the version of the children they want. With older children and other people, I’m not sure the mechanism for care is the same (although it might be, which would be worrying).Also, you might want to learn about cooperative inverse reinforcement learning, in case you aren’t aware of it. Utility functions are probably perfectly fine, it’s just that there needs to be a mechanism for updating them from sensory data.
Sadly, by posting this here, you’ve added this puzzle to the training set of future models. Good benchmarks (e.g. ARC) keep the test set, or at least part of it, private.
Firstly, and perhaps most importantly, my advice on what not to do is not to try directly convincing politicians to pause or stop AGI development. A prerequisite for them to take actions drastic enough to actually matter is for them to understand how powerful AGI will truly become. And once that happens, even if they ban all AI development, unless they consider the arguments for doom to be extremely strong, which they won’t[1], they will race and put truly enormous amounts of resources behind it, and that would be it for the species. Getting mid-sized business owners on board, on the other hand, might be a good idea due to the funding they could provide.
I don’t think any of the big donors are good enough, so if you want to donate to other people’s projects (or maybe become a co-founder), you could try finding interesting projects yourself on Manifund and the Nonlinear Network.
We know for a fact that alignment, at least for human-level intelligences, has a solution because people do actually care, at least in part, about each other. Therefore, it might be worth contacting Steven Byrnes and asking him whether he could usefully use more funding or what similar projects he recommends.
Outside AI, if the reason you care about existential risk isn’t because you want to save the species, but because human extinction implies a lot of people will die, you could try looking into chemical brain preservation and how cheap it is. This could itself be a source of revenue, and you probably won’t have any competitors (established cryonics orgs don’t offer cheap brain preservation and I have asked and Tomorrow Biostasis isn’t interested either).
I also personally have not completely terrible ideas for alignment research and weak (half an SD?) intelligence augmentation. If you’re interested, we can discuss them via DMs.
Finally, if you do fund intelligence augmentation research, please consider whether to keep it secret, if feasible.
Does OBP plan to eventually expand their services outside the USA? And how much would it cost if you didn’t subsidize it? Cost is a common complaint about cryonics so I could see you becoming much bigger than the cryonics orgs, but judging by the website you look quite small. Do you know why that is?
Does anyone have advice on how I could work full-time on an alignment research agenda I have? It looks like trying to get a LTFF grant is the best option for this kind of thing, but if after working more time alone on it, it keeps looking like it could succeed, it’s likely that it would become too big for me alone, I would need help from other people, and that looks hard to get. So, any advice from anyone who’s been in a similar situation? Also, how does this compare with getting a job at an alignment org? Is there any org where I would have a comparable amount of freedom if my ideas are good enough?
Edit: It took way longer than I thought it would, but I’ve finally sent my first LTFF grant application! Now let’s just hope they understand it and think it is good.
It depends on what you know about the model and the reason you have to be concerned in the first place (if it’s just “somehow”, that’s not very convincing).
You might be worried that training it leads to the emergence of inner-optimizers, be them ones that are somehow “trying” to be good at prediction in a way that might generalize to taking real-life actions, approximating the searchy part of the humans they are trying to predict, or just being RL agents. If you are just using basically standard architectures with a lot more compute, these all seem unlikely. But if I were you, I might try to test its ability to perform well in a domain it has never seen, where humans start by performing poorly but very quickly learn what to do (think about video games with new mechanics). If it does well, you have a qualitatively new thing on your hands, don’t deploy, study it instead. If a priori for some reason you think it could happen, and only a small subset of all the data is necessary to achieve that, do a smaller training run first with that data.
Or you might be worried about mostly external consequentialist cognition (think explicit textual it-then-elses). In that case, existing systems can already do it to some extent, and you should worry about how good its reasoning actually is, so perform capability evaluations. If it looks that there is some way of getting it to do novel research by any known method or that it’s getting close, don’t deploy, otherwise someone might figure out how to use it to do AI research, and then you get a singularity.
And in any case, you should worry about the effects your system will have on the AI race. Your AI might not be dangerous, but if it is a good enough lawyer or programmer that it starts getting many people out of their jobs, investment in AI research will increase a lot and someone will figure out how to create an actual AGI sooner than they would otherwise.
Edit: And obviously you should also test how useful it could be for people trying to do mundane harm (e.g. with existing pathogens) and, separately, there might not be a hard threshold on how good a model is at doing research that it starts being dangerous, so they might get there little by little and you would be contributing to that.
Edit in response to the second clarification: Downscale the relevant factors, like amount of training data, number of parameters and training time, or use a known-to-be-inferior architecture until the worrying capabilities go away. Otherwise, you need to solve the alignment problem.
Edit in response to Beth Barnes’s comment: You should probably have people reviewing outputs to check the model behaves well, but if you actually think you need measures like “1000 workers with technical undergrad degrees, paid $50/hr” because you are worried it somehow kills you, then you simply shouldn’t deploy it. It’s absurd to have the need to check whether a commercial product is an existential threat, or anything close to that.
Done! There aren’t enough mysterious old wizards.
You know of a technology that has at least a 10% chance of having a very big novel impact on the world (think the internet or ending malaria) that isn’t included in this list, very similar, or downstream from some element of it: AI, mind uploads, cryonics, human space travel, geo-engineering, gene drives, human intelligence augmentation, anti-aging, cancer cures, regenerative medicine, human genetic engineering, artificial pandemics, nuclear weapons, proper nanotech, very good lie detectors, prediction markets, other mind-altering drugs, cryptocurrency, better batteries, BCIs, nuclear fusion, better nuclear fission, better robots, AR, VR, room-temperature superconductors, quantum computers, polynomial time SAT solvers, cultured meat, solutions to antibiotic resistance, vaccines to some disease, optical computers, artificial wombs, de-extinction and graphene.
Bad options included just in case someone thinks they are good.
Public mechanistic interpretability research is net positive in expectation.
Cultural values are something like preferences over pairs of social environments and things we actually care about. So it makes sense to talk about jointly optimizing them.
If we had access to a brain upload (and maybe a world simulator too) we could in principle extract something like a utility function, and the theory behind it relates more to agents in general than it does to humans in particular.
Research into getting a mechanistic understanding of the brain for purposes of at least one of: understanding how values/empathy works in people, brain uploading or improving cryonics/plastination is net positive and currently greatly underfunded.
Came here to comment that. It seems much more efficient to learn the cognitive strategies smart people use than to try to figure them out from scratch. Ideally, you would have people of different skill levels solve problems (and maybe even do research) while thinking out loud and describing or drawing the images they are manipulating. I know this has been done at least for chess, and it would be nice to have it for domains with more structure. Then you could catalog these strategies and measure the effectiveness of teaching the system 2 process (the whole process they use, not only the winning path) and explicitly train in isolation the individual system 1 steps that make it up.
Also “indivudals”.
The police hang up on him, and when he tells people about it, they say that’s the expected result, but that anyone with his beliefs would at the very least try it, and, more importantly, that nobody would try it without having those beliefs. Anyway, the point of the post is to come up with something better.