Professors are selected to be good at research not good at teaching. They are also evaluated at being good at research, not at teaching. You are assuming universities primarily care about undergraduate teaching, but that is very wrong.
(I’m not sure why this is the case, but I’m confident that it is)
A more charitable interpretation is that this is a probability rounded to the nearest percent
Counterpoint: spending 40 hours a week on your job is a huge time commitment. It’s also a huge willpower drain (doing worthwhile things requires effort to grind out results, not just time). It’s hard for me to believe that the hours money allows you to “buy back” are worth the “wasted hours” on work. So it’s important that work not be a waste.
Also, you will probably make more money and do more worthwhile things at a job you enjoy.
Money is definitely a big factor, but I don’t think it totally dominates everything else.
My sense is that the existing arguments are not very strong (e.g. I do not find them convincing), and their pretty wide acceptance in EA discussions mostly reflects self-selection (people who are convinced that AI risk is a big problem are more interested in discussing AI risk). So in that sense better intro documents would be nice. But maybe there simply aren’t stronger arguments available? (I personally would like to see more arguments from an “engineering” perspective, starting from current computer systems rather than from humans or thought experiments). I’d be curious what fraction of e.g. random ML people find current intro resources persuasive.
That said, just trying to expose more people to the arguments makes a lot of sense; however convincing the case is, the number of convinced people should scale linearly with the number of exposed people. And social proof dynamics probably make it scale super-linearly.
Tim Cook could not do all the cognitive labor to design an iPhone (indeed, no individual human could). The CEO of Boeing could not fully design a modern plane. Elon Musk could not make a Tesla from scratch. All of these cases violate all of your three bullet points. Practically everything in the modern world is too complicated for any single person to fully understand, and yet it all works fairly well, because successful outsourcing of cognitive labor is routinely successful.
It is true that a random layperson would have a hard time verifying an AI’s (or anyone else’s) ideas about how to solve alignment. But the people who are going to need to incorporate alignment ideas into their work—AI researchers and engineers—will be in a good position to do that, just as they routinely incorporate many other ideas they did not come up with into their work. Trying to use ideas from an AI sounds similar to me to reading a paper from another lab—could be irrelevant or wrong or even malicious, but could also have valuable insights you’d have had a hard time coming up with yourself.
The key question here is how difficult the objective O is to achieve. If O is “drive a car from point A to point B”, then we agree that it is feasible to have AI systems that “strongly increase the chance of O occuring” (which is precisely what we mean by “goal-directedness”) without being dangerous. But if O is something that is very difficult to achieve (i.e. all of humanity is currently unable to achieve it), then it seems that any system that does reliably achieve O has to “find new and strange routes to O” almost tautologically.
Once we build AI systems that find such new routes for achieving an objective, we’re in dangerous territory, no matter whether they are explicit utility maximizers, self-modifying, etc. The dangerous part is coming up with new routes that achieve the objective, since most of these routes will contain steps that look like “acquire resources” or “manipulate humans”.”
This seems pretty wrong. Many humans are trying to achieve goals that no one currently knows how to achieve, and they are mostly doing that in “expected” ways, and I expect AIs would do the same. Like if O is “solve an unsolved math problem”, the expected way to do that is to think about math, not try to take over the world. If O is “cure a disease”, the expected way to do that is doing medical research, not “acquiring resources”. In fact, it seems hard to think of an objective where “do normal work in the existing paradigm” is not a promising approach.
I think you are failing to distinguish between “being able to pursue goals” and “having a goal”.
Optimization is a useful subroutine, but that doesn’t mean it is useful for it to be the top-level loop. I can decide to pursue arbitrary goals for arbitrary amounts of time, but that doesn’t mean that my entire life is in service of some single objective.
Similarly, it seems useful for an AI assistant to try and do the things I ask it to, but that doesn’t imply it has some kind of larger master plan.
Why should we expect AGIs to optimize much more strongly and “widely” than humans? As far as I know a lot of AI risk is thought to come from “extreme optimization”, but I’m not sure why extreme optimization is the default outcome.
To illustrate: if you hire a human to solve a math problem, the human will probably mostly think about the math problem. They might consult google, or talk to some other humans. They will probably not hire other humans without consulting you first. They definitely won’t try to get brain surgery to become smarter, or kill everyone nearby to make sure no one interferes with their work, or kill you to make sure they don’t get fired, or convert the lightcone into computronium to think more about the problem.
IMO the biggest hole here is “why should a superhuman AI be extremely consequentialist/optimizing”? This is a key assumption; without it concerns about instrumental convergence or inner alignment fall away. But there’s no explicit argument for it.
Current AIs don’t really seem to have goals; humans sort of have goals but very far from the level of “I want to make a cup of coffee so first I’ll kill everyone nearby so they don’t interfere with that”.
It turns out that it’s not clear this is actually true—some studies have found more money leads to greater happiness up through the highest income levels examined.
The “highest income levels examined”—based on the chart on that page—appear to be 128k/yr. Since the income satisficing level (for an unattached individual) is ~75k, this doesn’t seem like good evidence one way or another.
A bold claim, since no one understands “the algorithms used by the brain”. People have been trying to “understand how intelligence works” for decades with no appreciable progress; all of the algorithms that look “intelligent” (Deep Blue, Watson, industrial-strength machine learning) require massive computing power.
Isn’t “bomb all sufficiently advanced semiconductor fabs” an example of a pivotal act that the US government could do right now, without any AGI at all?
If current hardware is sufficient for AGI than maybe that doesn’t make us safe, but plausibly current hardware is not sufficient for AGI, and either way stopping hardware progress would slow AI timelines a lot.
Because I want to be alive. I don’t just want humanity to have the benefit of my skills and knowledge.
(This comment is on career stuff, which is tangential to your main points)
I recently had to pick a computer science job, and spent a long time agonizing over what would have the highest impact (among other criteria). I’m not convinced startups or academia have a higher expected value than large companies. I would like to be convinced otherwise.
(Software) Startups:
1) Most startups fail. It’s easy to underestimate this because you only hear the success stories.
2) Many startups are not solving “important” problems. They are solving relatively minor problems for relatively rich people, because that’s where the money is. Snapchat, Twitter, Facebook, Instagram are examples.
3) Serious problems are complicated, and usually require more resources than a startup can bring to bear.
4) Financially: If you aren’t a founder, your share of the company is negligible.
(Computer Science) Academia:
1) My understanding is that there are dozens of applications for each tenure-track opening. So your chance of success is low, and your marginal advantage over the next-best applicant is probably low.
2) I trust markets more than grant committees for distributing money.
3) It seems easier to get sidetracked into non-useful work in academia
Nate’s view here seems similar to “To do cutting-edge alignment research, you need to do enough self-reflection that you might go crazy”. This seems really wrong to me. (I’m not sure if he means all scientific breakthroughs require this kind of reflection, or if alignment research is special).
I don’t think many top scientists are crazy, especially not in a POUDA way. I don’t think top scientists have done a huge amount of self-reflection/philosophy.
On the other hand, my understanding is that some rationalists have driven themselves crazy via too much self-reflection in an effort to become more productive. Perhaps Nate is overfitting to this experience?
“Just do normal incremental science; don’t try to do something crazy” still seems like a good default strategy to me (especially for an AI).
Thanks for this write up; it was unusually clear/productive IMO.
(I’m worried this comment comes off as mean or reductive. I’m not trying to be. Sorry)
The link at the top is to the wrong previous scenario
“This is what it looks like in practice, by default, when someone tries to outsource some cognitive labor which they could not themselves perform.”
This proves way too much. People successfully outsource cognitive labor all the time (this describes most white-collar jobs). This is possible because very frequently, it is easier to be confident that work has been done correctly than to actually do the work. You shouldn’t just blindly trust an AI that claims to have solved alignment (just like you wouldn’t blindly trust a human), but that doesn’t mean AIs (or other humans) can’t do any useful work.