Jonathan Paulson

Karma: 232

Jonathan Paulson 20 Mar 2023 11:29 UTC
3 points
0
on: Will people be motivated to learn difficult disciplines and skills without economic incentive?
The idea that all cognitive labor will be automated in the near-future is a very controversial premise, not at all implied by the idea that AI will be useful for tutoring. I think that’s the disconnect here between Altman’s words and your interpretation.

Jonathan Paulson 14 Mar 2023 14:38 UTC
4 points
2
on: Discussion with Nate Soares on a key alignment difficulty
Nate’s view here seems similar to “To do cutting-edge alignment research, you need to do enough self-reflection that you might go crazy”. This seems really wrong to me. (I’m not sure if he means all scientific breakthroughs require this kind of reflection, or if alignment research is special).

I don’t think many top scientists are crazy, especially not in a POUDA way. I don’t think top scientists have done a huge amount of self-reflection/philosophy.

On the other hand, my understanding is that some rationalists have driven themselves crazy via too much self-reflection in an effort to become more productive. Perhaps Nate is overfitting to this experience?

“Just do normal incremental science; don’t try to do something crazy” still seems like a good default strategy to me (especially for an AI).

Thanks for this write up; it was unusually clear/productive IMO.

(I’m worried this comment comes off as mean or reductive. I’m not trying to be. Sorry)

Jonathan Paulson 10 Mar 2023 12:20 UTC
6 points
3
in reply to: johnswentworth’s comment on: Why Not Just Outsource Alignment Research To An AI?
Tim Cook could not do all the cognitive labor to design an iPhone (indeed, no individual human could). The CEO of Boeing could not fully design a modern plane. Elon Musk could not make a Tesla from scratch. All of these cases violate all of your three bullet points. Practically everything in the modern world is too complicated for any single person to fully understand, and yet it all works fairly well, because successful outsourcing of cognitive labor is routinely successful.
It is true that a random layperson would have a hard time verifying an AI’s (or anyone else’s) ideas about how to solve alignment. But the people who are going to need to incorporate alignment ideas into their work—AI researchers and engineers—will be in a good position to do that, just as they routinely incorporate many other ideas they did not come up with into their work. Trying to use ideas from an AI sounds similar to me to reading a paper from another lab—could be irrelevant or wrong or even malicious, but could also have valuable insights you’d have had a hard time coming up with yourself.

Jonathan Paulson 10 Mar 2023 0:23 UTC
44 points
41
on: Why Not Just Outsource Alignment Research To An AI?
“This is what it looks like in practice, by default, when someone tries to outsource some cognitive labor which they could not themselves perform.”
This proves way too much. People successfully outsource cognitive labor all the time (this describes most white-collar jobs). This is possible because very frequently, it is easier to be confident that work has been done correctly than to actually do the work. You shouldn’t just blindly trust an AI that claims to have solved alignment (just like you wouldn’t blindly trust a human), but that doesn’t mean AIs (or other humans) can’t do any useful work.

Jonathan Paulson 28 Feb 2023 0:02 UTC
3 points
0
on: [S] D&D.Sci: All the D8a. Allllllll of it. Evaluation and Ruleset
The link at the top is to the wrong previous scenario

Jonathan Paulson 17 Feb 2023 3:51 UTC
1 point
0
in reply to: Daniel Kokotajlo’s comment on: Why should we expect AIs to coordinate well?
I don’t think “they” would (collectively) decide anything, since I don’t think it’s trivial to cooperate even with a near-copy of yourself. I think they would mostly individually end up working with/for some group of humans, probably either whichever group created them or whichever group they work most closely with.

I agree humans could end up disempowered even if AIs aren’t particularly good at coordinating; I just wanted to put some scrutiny on the claim I’ve seen in a few places that AIs will be particularly good at coordinating.

[Question] Why should we expect AIs to coordinate well?

Jonathan Paulson14 Feb 2023 15:50 UTC

25 points

9 comments1 min readLW link

Jonathan Paulson 24 Oct 2022 14:44 UTC
10 points
0
on: Response to Katja Grace’s AI x-risk counterarguments

The key question here is how difficult the objective O is to achieve. If O is “drive a car from point A to point B”, then we agree that it is feasible to have AI systems that “strongly increase the chance of O occuring” (which is precisely what we mean by “goal-directedness”) without being dangerous. But if O is something that is very difficult to achieve (i.e. all of humanity is currently unable to achieve it), then it seems that any system that does reliably achieve O has to “find new and strange routes to O” almost tautologically.

Once we build AI systems that find such new routes for achieving an objective, we’re in dangerous territory, no matter whether they are explicit utility maximizers, self-modifying, etc. The dangerous part is coming up with new routes that achieve the objective, since most of these routes will contain steps that look like “acquire resources” or “manipulate humans”.”

This seems pretty wrong. Many humans are trying to achieve goals that no one currently knows how to achieve, and they are mostly doing that in “expected” ways, and I expect AIs would do the same. Like if O is “solve an unsolved math problem”, the expected way to do that is to think about math, not try to take over the world. If O is “cure a disease”, the expected way to do that is doing medical research, not “acquiring resources”. In fact, it seems hard to think of an objective where “do normal work in the existing paradigm” is not a promising approach.

Jonathan Paulson 26 Aug 2022 16:07 UTC
4 points
2
on: Is population collapse due to low birth rates a problem?
It’s true that more people means we each get a smaller share of the natural resources, but more people increases the benefits of innovation and specialization. In particular, the benefits of new technology scale linearly with the population (everyone can use the) but the costs of research do not. Since the world is getter richer over time (even as the population increases), the average human is clearly net positive.

Jonathan Paulson 4 Aug 2022 20:43 UTC
22 points
5
in reply to: jimrandomh’s comment on: What do ML researchers think about AI in 2022?
A more charitable interpretation is that this is a probability rounded to the nearest percent

Jonathan Paulson 18 Jul 2022 21:52 UTC
−1 points
0
in reply to: iamthouthouarti’s comment on: All AGI safety questions welcome (especially basic ones) [monthly thread]
I don’t think most people are trying to explicitly write down all human values and then tell them to an AI. Here are some more promising alternatives:
1. Tell an AI to “consult a human if you aren’t sure what to do”
2. Instead of explicitly trying to write down human values, learn them by example (by watching human actions, or reading books, or…)

Jonathan Paulson 18 Jul 2022 21:33 UTC
6 points
1
on: All AGI safety questions welcome (especially basic ones) [monthly thread]
Why should we expect AGIs to optimize much more strongly and “widely” than humans? As far as I know a lot of AI risk is thought to come from “extreme optimization”, but I’m not sure why extreme optimization is the default outcome.

To illustrate: if you hire a human to solve a math problem, the human will probably mostly think about the math problem. They might consult google, or talk to some other humans. They will probably not hire other humans without consulting you first. They definitely won’t try to get brain surgery to become smarter, or kill everyone nearby to make sure no one interferes with their work, or kill you to make sure they don’t get fired, or convert the lightcone into computronium to think more about the problem.

Jonathan Paulson 26 Jun 2022 7:30 UTC
2 points
0
in reply to: Rob Bensinger’s comment on: The inordinately slow spread of good AGI conversations in ML
I agree with it but I don’t think it’s making very strong claims.

I mostly agree with part 1; just giving advice seems too restrictive. But there’s a lot of ground between “only gives advice” and “fully autonomous” and “fully autonomous” and “globally optimizing a utility function”, and I basically expect a smooth increase in AI autonomy over time as they are proved capable and safe. I work in HFT; I think that industry has some of the most autonomous AIs deployed today (although not that sophisticated), but they’re very constrained over what actions they can take.

I don’t really have the expertise to have an opinion on “agentiness helps with training”. That sounds plausible to me. But again, “you can pick training examples” is very far from “fully autonomous”. I think there’s a lot of scope for introducing “taking actions” that doesn’t really pose a safety risk (and ~all of Gwern’s examples fall into that, I think; eg optimizing over NN hyper parameters doesn’t seem scary).

I guess overall I agree AIs will take a bunch of actions, but I’m optimistic about constraining the action space or the domain/world-model in a way that IMO gets you a lot of safety (in a way that is not well-captured by speculating about what the right utility function is).

Jonathan Paulson 25 Jun 2022 14:23 UTC
5 points
2
on: The inordinately slow spread of good AGI conversations in ML
My sense is that the existing arguments are not very strong (e.g. I do not find them convincing), and their pretty wide acceptance in EA discussions mostly reflects self-selection (people who are convinced that AI risk is a big problem are more interested in discussing AI risk). So in that sense better intro documents would be nice. But maybe there simply aren’t stronger arguments available? (I personally would like to see more arguments from an “engineering” perspective, starting from current computer systems rather than from humans or thought experiments). I’d be curious what fraction of e.g. random ML people find current intro resources persuasive.

That said, just trying to expose more people to the arguments makes a lot of sense; however convincing the case is, the number of convinced people should scale linearly with the number of exposed people. And social proof dynamics probably make it scale super-linearly.

Jonathan Paulson 9 Jun 2022 0:06 UTC
3 points
−1
in reply to: Steven Byrnes’s comment on: AGI Ruin: A List of Lethalities
I expect people to continue making better AI to pursue money/fame/etc., but I don’t see why “better” is the same as “extremely goal-directed”. There needs to be an argument that optimizer AIs will outcompete other AIs.

Eliezer says that as AI gets more capable, it will naturally switch from “doing more or less what we want” to things like “try and take over the world”, “make sure it can never be turned off”, “kill all humans” (instrumental goals), “single-mindedly pursue some goal that was haphazardly baked in by the training process” (inner optimization), etc. This is a pretty weird claim that is more assumed than argued for in the post. There’s some logic and mathematical elegance to the idea that AI will single-mindedly optimize something, but it’s not obvious and IMO is probably wrong (and all these weird bad consequences that would result are as much reasons to think this claim is wrong as they are reasons to be worried if its true).

Jonathan Paulson 7 Jun 2022 23:08 UTC
6 points
1
on: AGI Ruin: A List of Lethalities
IMO the biggest hole here is “why should a superhuman AI be extremely consequentialist/optimizing”? This is a key assumption; without it concerns about instrumental convergence or inner alignment fall away. But there’s no explicit argument for it.

Current AIs don’t really seem to have goals; humans sort of have goals but very far from the level of “I want to make a cup of coffee so first I’ll kill everyone nearby so they don’t interfere with that”.

Jonathan Paulson 7 Jun 2022 22:51 UTC
1 point
0
in reply to: Vaniver’s comment on: AGI Ruin: A List of Lethalities
I don’t think “burn all GPUs” fares better on any of these questions. I guess you could imagine it being more “accessible” if you think building aligned AGI is easier than convincing the US government AI risk is truly an existential threat (seems implausible).
“Accessibility” seems to illustrate the extent to which AI risk can be seen as a social rather than technical problem; if a small number of decision-makers in the US and Chinese governments (and perhaps some semiconductor companies and software companies) were really convinced AI risk was a concern, they could negotiate to slow hardware progress. But the arguments are not convincing (including to me), so they don’t.

In practice, negotiation and regulation (I guess somewhat similar to nuclear non-proliferation) would be a lot better than “literally bomb fabs”. I don’t think being driven underground is a realistic concern—cutting-edge fabs are very expensive.

Jonathan Paulson 7 Jun 2022 5:35 UTC
5 points
1
on: AGI Ruin: A List of Lethalities
Isn’t “bomb all sufficiently advanced semiconductor fabs” an example of a pivotal act that the US government could do right now, without any AGI at all?

If current hardware is sufficient for AGI than maybe that doesn’t make us safe, but plausibly current hardware is not sufficient for AGI, and either way stopping hardware progress would slow AI timelines a lot.

Jonathan Paulson 9 Apr 2022 14:27 UTC
1 point
in reply to: acylhalide’s comment on: Why Instrumental Goals are not a big AI Safety Problem
A > B > human. I expect B < human would also be quite useful.

B does not have a lot of opportunity for action—all it can do is prevent A from acting. It seems like its hard to “eliminate humans” with just that freedom. I agree B has an incentive to hamper A.

Even a B that is dumber than humans is valuable, because it is faster than humans. B can provide real-time feedback on every action that A takes, whereas consulting humans for every action would be impractical—it costs too much human time and it would prevent A from operating in realtime.

Jonathan Paulson 9 Apr 2022 14:22 UTC
1 point
in reply to: Evan R. Murphy’s comment on: Why Instrumental Goals are not a big AI Safety Problem
Why isn’t it competitive? A is being trained the same way as an agentic system, so it will be competitive.

Adding B is a 2x runtime/training-cost overhead, so there is a “constant factor” cost; is that enough to say something is “not competitive”? In practice I’d expect you could strike a good safety/overhead balance for much less.

Jonathan Paulson

[Question] Why should we ex­pect AIs to co­or­di­nate well?

[Question] Why should we expect AIs to coordinate well?