P(doom) = 60%, 4 years to AGI.
I give full permission for anyone to post part or all of any of my comments/posts to other platforms, with attribution.
Currently upskilling for future work on technical AI alignment research. Want to do part time work translating AI developments in China for a Western audiance in the meantime. Would really appreciate funding on either.
Will analyze papers and do bioinformatics research for money. DM me if interested.
I never understood why smallpox resurrection is so feared. It was heavily suppressed with 19th century organization and technology in developed countries and eradicated even in Somalia with 20th century ones. If it reappeared in NYC somehow, it would be very easy to track given its visible symptoms and quickly eliminated.
If your moral theory gives humanity less moral worth than carp, so much the worse for your moral theory.
If morality as a concept irrefutably proves it, then so much the worse for morality.
I noticed a similar trend of loose argumentation and a devaluing of truth-seeking in the AI Safety space as public advocacy became more prominent.
I see your point. Maybe something like “resource domination” or just “instrumental resource acquisition” is a better term for what he is looking for, I think.
What is the difference between that and instrumental convergence?
I think he just wants an example of an agent being rewarded for something simple (like being rewarded for resource collection) exhibiting power-seeking behavior to the degree that it takes over the game environment. It’s an intuitive difference to a lot of people to an agent specifically maximizing an objective. I actually can’t name an example after looking for an hour, but I would bet money something like that already exists.
My guess is that if you plop two Starcraft AIs on a board and reward them every time they gather resources, with enough training, they would start fighting each other for control of the map. I would also guess that someone has already done this exact scenario. Is there an AI search engine for Reddit anyone would recommend?
ChatGPT is basically RLHF’d to approximate the responses of a human roleplaying as an AI assistant. So this proves that… it can roleplay a human roleplaying as an AI assistant roleplaying as a different AI in such a way that said different AI exhibits credit hacking behaviors.
I think we already had an example in gridworlds where an AI refuses to go on a hex that would change its reward function, even though it gives it a higher reward, but that might have just been a thought experiment.
Nice thoughts, but we live in a world where nuclear reactors get shut down because people plug in USBs they found in a parking lot into their work laptops.
Good luck, I guess.
Questions:
Would something like an agent trained to maximize minerals mined in Starcraft learning to attack other players to monopolize their resources count?
I assume it would count if that same agent was just rewarded every time it mined minerals, or the mineral count went up, without an explicit objective to maximize the amount of minerals it has?
Would a gridworld example work? How complex does the simulation have to be?
What do you think happens in a world where there is $100 billion in yearly alignment funding? How would they be making less progress? I want to note that even horrifically inefficient systems still produce more output than “uncorrupted” hobbyists—cancer research would produce much fewer results if it were done by 300 perfectly coordinated people, even if the 300 had zero ethical/legal restraints.
What about public funding? A lot of people are talking to politicians, but requesting more funding doesn’t seem to be a central concern—I’ve heard more calls for regulation than calls for public AI x-risk funding.
This seems like it’s probably a misunderstanding. With the exception of basically just MIRI, AI alignment didn’t exist as a field when DeepMind was founded, and I doubt Sam Altman ever actively sought employment at an existing alignment organization before founding OpenAI.
Yeah, in hindsight he probably meant that they got interested in AI because of AI safety ideas, then they decided to go into capabilities research after upskilling. Then again, how are you going to get funding otherwise, charity? It seems that a lot of alignment work, especially the conceptual kind we really need to make progress toward an alignment paradigm, is just a cost for an AI company with no immediate upside. So any AI alignment org would need to pivot to capabilities research if they wanted to scale their alignment efforts.
Keep in mind that “will go on to do capabilities work” isn’t the only -EV outcome; each time you add a person to the field you increase the size of the network, which always has costs and doesn’t always have benefits.
I strongly disagree. The field has a deficit of ideas and needs way more people. Of course inefficiencies will increase, but I can’t think of any other field that progressed faster explicitly because members made an effort to limit recruitment. Note that even very inefficient fields like medicine make faster progress when more people are added to the network—it would be very hard to argue, for example, that a counterfactual world where no one in China did medical research would have made more progress. My personal hope is 1 million people working on technical alignment which implies $100 billion+ annual funding. 10x that would be better, but I don’t think it’s realistic.
I’ve heard that it is very difficult to get funding unless you have a paradigmatic idea, and you can’t get a job without good technical AI skills. But many people who skill up to get a job in technical alignment end up doing capabilities work because they can’t find employment in AI Safety, or the existing jobs don’t pay enough. Apparently, this was true for both Sam Altman and Demis Hassabis? I’ve also experienced someone discouraging me from acquiring technical AI skills for the purpose of pursuing a career in technical alignment because they don’t want me to contribute to capabilities down the line. They noted that most people who skill up to work on alignment end up working in capabilities instead, which is kinda crazy.
My thinking is that I am just built different and will come up with a fundable paradigmatic idea where most fail. But yeah, the lack of jobs heavily implies that the field is funding-constrained because talent wants to work on alignment.
Creating a super-genius is almost trivial with germ-line engineering.
Not really true—known SNP mutations associated with high intelligence have relatively low effect in total. The best way to make a really smart baby with current techniques is with donor egg and sperm, or cloning.
It is also possible that variance in intelligence among humans is due to something analogous to starting values in neural networks—lucky/crafted values can result in higher final performance, but getting those values into an already established network just adds noise. You can’t really change macrostructures in the brain with gene therapy in adults, after all.
I have a suspicion about why DNA is base 4 vs base 2.
It has to do with the binding strengths of tRNA to mRNA during translation. Base 2 increases the length of codons needed to code for 23 amino acids from 3 to 5, which may lead to less fidelity in the resulting amino acid sequence. That is, partial matches have higher binding strength compared to base 4, which increases the chances the wrong tRNA is loaded. This could also make longer amino acid sequences harder to create—an accidentally loaded stop codon terminates translation!
A couple of things:
The 15 ug/L figure was just the first one I found—the study actually noted that most others have found lower (5-10 ug/L) levels in children.
Instead of working backwards, why not just compare blood AL levels in matched vaccinated and unvaccinated children? Different countries have different vaccination schedules and so on. I’m not digging further because I should be doing other stuff, but it’s a good lead.
I couldn’t find any reference Al levels for childrens’ brains.
1.67mg/g of Aluminum in a one-year-old’s brain doesn’t pass the sanity check. I think the above chart has a typo—that should be ug/g, not mg/g. Otherwise the mice would be very dead! Also, normal Al levels in the brain are around 1 ug/g, a mouse has a brain mass massing 0.4g (dry mass ~0.3g), and 1mg/g would imply ~0.3 mg of Al, far more than injected in the first place.
https://sci-hub.ru/10.1186/1741-7015-11-99
They did this and it was fine. Reference levels are around 1 mg/kg for humans. Yes, immune cells gobbled up the aluminum and spread it around the body. Some of it went to the brain, raising the local AL levels. No control group is a bit sus. And this is with an 18 ug injection into ~35g mice. That’s like a 500 ug injection into a 1kg baby. C57 are the wild-type mice, btw.
https://sci-hub.ru/10.1007/s12640-016-9656-y
The study they were citing was a typically underpowered mouse study with p-hacked groups. Why did they choose to run 2 groups, each with their own controls, on the same tests? The only significant behavioral difference between the groups was that controls performed better on the memory test.
But 1.5 mg/kg and 8.3 mg/kg are normal human-level AL intakes! And look at those error bars! Also, they ran their analysis completely wrong. This null hypothesis you’re supposed to be testing against is “there is no difference in exploration time (c-a) between each groups”. Instead, they were testing against “there is no difference in exploration time between a and c”. That is, their null hypothesis was that their mice had no memory, they found that their control group definitely did have memory, and concluded that AL had an effect… But then they used the wrong control group? Everything in this study is wrong and the authors and reviewers should feel bad.
Let me reiterate that. THEIR ONLY STATISTICALLY SIGNIFICANT RESULT WAS THAT CONTROL MICE HAD FUNCTIONAL MEMORY. THE ERROR BARS ON TEST GROUP MICE WERE SO HIGH YOU CAN’T TELL EITHER WAY. THEY DID THEIR ANALYSIS WRONG AND THEIR CONCLUSIONS SHOULD BE DISCARDED. So yeah, it’s completely bunk.
Surely running a dollar auction with decision-theory nerds won’t result in hurt feelings.
I think the Game Theory thing to do here is to send you $600 ASAP and post evidence of that everywhere I can, but the fact that I came up with a plan to give a random stranger on the internet $600 probably means I am in a scam and therefore I refuse.
I agree that most of the risk from smallpox comes from a weaponized strain. Given what we know about the Soviet bioweapons program, I think any form of weaponized smallpox released would be engineered to bypass existing vaccines. This would make getting the smallpox vaccine in anticipation negative EV.
Also, wasn’t there a theory that the smallpox vaccine gave partial protection against HIV?