Googling for “scurvy low mood”, I find plenty of sources that indicate that scurvy is accompanied by “mood swings — often irritability and depression”. IIRC, this has remarked upon for at least two hundred years.
Carl Feynman
Karma: 118
The recent announcement that OpenAI had GPT-4 since last August, but spent the intervening time evaluating it, instead of releasing it, constitutes a “deliberate slowdown” by a “major lab”. Do you require that multiple labs slow down before the bet is voided?
They tried to detect and prevent questions appearing in the training set being asked as part of the tests. It didn’t seem to make much difference. See table 10, “contamination data for exams”. It’s a pretty tiny fraction of the data, and removing it didn’t make much difference.
I assume when you say fine tuning, you mean RLHF. This is table 8 on page 27 of the paper. Some scores went up a few percent, some scores went down a few percent, overall no significant change.
The biggest changes were that it’s a much worse microeconomist and a much better sommelier. Pretty impressive for a machine with no sense of taste.
You say “It’s hard, for example, to think of an organization where the political appointees are at the bottom of the hierarchy and the technocrats on the top.” How about a political campaign? In the US, at least, a campaign tends to run by a few salaried operators in charge of a bunch of volunteers. The operators make a career of it, moving from campaign to campaign, while the volunteers self-select based on political conviction.
I don’t see any generalizations to draw here.
You’re preferentially remembering programs that came to a successful conclusion. Counterbalance the Apollo Project with Project Pluto, Project Orion, the X-33, and the National Aero-Space Plane, which consumed lots of effort and never resulted in flyable products. Camp Century and the larger Project Iceworm turned out not to be a good idea once they tried it. The Japanese Fifth Generation project was a total washout. Also consider the War on Cancer and the Space Shuttle, which produced results, but far less than the early proponents imagined.
None of these seemed like dumb ideas going in.
if an AI appreciates ecology more than we do, among its goals is to prevent human harm to ecosystems, and so among its early actions will be to kill most or all humans. You didn’t think of this, because it’s such an inhuman course of action.
Almost every goal that is easy to specify leads to human disempowerment or extinction, if a superhuman entity tries hard enough to accomplish it. This regrettable fact takes a while to convince yourself of, because it is so strange and terrible. In my case, it was roughly 1997-2003. Hopefully humanity learns a bit faster.
>If you have technical understanding of current AIs, do you truly believe there are any major obstacles left?
I‘ve been working in AI (on and off) since 1979. I don’t work on it any more, because of my worries about alignment. I think this essay is mostly correct about short timelines.
That said, I do think there is at least one obstacle between us and dangerous superhuman AI. I haven’t seen any good work towards solving it, and I don’t see any way to solve it myself in the short term. That said, I take these facts as pretty weak evidence. Surprising capabilities keep emerging from LLMs and RL, and perhaps we will solve the problem in the next generation without even trying. Also, the argument from personal incomprehension is weak, because there are lots of people working on AI, who are smarter, more creative, and younger.
I’m of mixed feelings about your request not to mention the exact nature of the obstacle. I respect the idea of not being explicit about the nature of the Torment Nexus. But I think we could get more clarity about alignment by discussing it explicitly. I bet there are people working on it already, and I don’t think discussing it here will cause more people to work on it.
Contract Research Organization. Basically an outfit you can hire to perform experiments for you.
Why do you care if people around you, who presumably have lives to live, care about AI risk? It’s not a problem like AIDS or groundwater pollution, where individual carefulness is needed to make a difference. In those cases, telling everybody about the problem is important, because it will prevent them having unprotected sex, or dumping used motor oil in their backyard. Unaligned AGI is a problem like nuclear war or viral gain-of-function research, where a few people work on the problem pretty much full time. If you want to increase the population of such people, that’s fine, but telling your mother-in-law that the world is doomed isn’t going to help.
You ask two interesting questions, with rather separate answers. I will discuss each in turn.
First, It’s plausible to think that “it’s possible to find an architecture for general intelligence a lot more efficiently than evolution”. Our process of engineering development is far faster than evolution. People get good (or bad) ideas, try stuff, copy what works, speak at conferences, publish, make theories, teach undergraduates… and the result is progress in decades instead of millions of years. We haven’t duplicated all the achievements of life yet, but we’ve made a start, and have exceeded it in many places. In particular, we’ve recently made huge progress in AI. GPT-3 has pretty much duplicated the human language faculty, which takes up roughly 1% of the brain. And we’ve duplicated visual object recognition, which takes another few percent. Those were done without needing evolution, so we probably don’t need evolution for the remaining 90% of the mind.
Second, “an intelligence that does the exact things we want” is the ideal that we’re aiming for. Unfortunately it does not seem possible to get to that, currently. With current technology, what we get is “an intelligence that does approximately what we rewarded it for, plus some other weird stuff we didn’t ask for.” It’s not obvious, but it is much harder than you think to specify a set of goals that produce acceptable behavior. And it is even harder (currently impossible) to provide any assurance that an AI will continue to follow those goals when set free to exert power in the world.
https://arxiv.org/abs/1806.02404 “Dissolving the Fermi paradox”.
The Drake equation gives an estimate for the number of technological civil actions to ever arise, by multiplying a number of parameters. Many of these parameters are unknown, and reasonable estimates range over many orders of magnitude. This paper takes defensible ranges for these parameters from the literature, and shows that if they are all small, but reasonable, we are the only technological civilization in the universe.
Earth was not eaten by aliens or an AGI in the past, nor do we see them in the sky, so we are probably alone. (Or else interstellar expansion is impossible, for some reason. But that seems unlikely.)
In 1982 or so, Eric Drexler had the idea of looking at photographs of the nearer galaxies and seeing if any had circular areas of darkness in them, suggesting a spreading civilization. It was in an atlas of galaxies, that had one galaxy per page, so a few hundred galaxies at most. At least that’s what I remember from talking to him about it at the time.
Since then, automated galaxy surveys have looked at millions of galaxies, with “funny looking” ones reviewed by humans. That’s how Julianne Dalcanton found Comet Dalcanton, for example: the program kicked it out and said “What’s with this funny looking galaxy?” And when she looked at it, she realized it was not a galaxy, but a comet. Perhaps this kind of survey would turn up a civilized galaxy, but I don’t know how to estimate the probability of it being detected.
Here‘s a 2015 study that looked for Dyson spheres in 1359 galaxies: https://iopscience.iop.org/article/10.1088/0004-637X/810/1/23
You might want to look at”The major transitions in life revisited.” That’s a book from 2011 that looks at what has been learned in the field since the publication of the original in 1995. I know that we know far more about eukaryogenesis than we did then. Also, there are better models of the RNA world than either stochastic correction or Eigen’s hypercycles. That is, models which can explain the ability of life to become complex in an environment of inaccurate replication and free riders. If that book isn’t enough to get you more up to date, email carlf at abinitio dot com and I’ll shoot you some references.
Let’s hope not.