JakubK

Karma: 391

JakubK 7 Jun 2022 17:04 UTC
9 points
on: A Quick Guide to Confronting Doom
“My guess is that people who are concluding P(Doom) is high will each need to figure out how to live with it for themselves.”

The following perspective helps me feel better.

First, it’s not news that AGI poses a significant threat to humanity. I felt seriously worried when I first encountered this idea in 2018 listening to Eliezer on the Sam Harris podcast. The “Death With Dignity” post revived these old fears, but it didn’t reveal new dangers that were previously unknown to me.

Second, many humans have dealt with believing “P(impending Doom) = high” at many times throughout history. COVID, Ukraine, famine in Yemen, WWI, WWII, the Holocaust, 9/11, incarceration in the US, Mongol conquests, the Congo Free State’s atrocities, the Great Purge, the Cambodian genocide, the Great Depression, the Black Death, the Putumayo genocide, the Holodomor, the Trail of Tears, the Syrian civil war, the Irish Potato Famine, the Vietnam War, slavery, colonialism, more wars, more terrorist attacks, more plagues, more famines, more genocides, etc.

It’s easy to read these events without simulating how the people involved felt. In most of these cases, “Doom” didn’t mean “everyone on the planet will die” but rather “I will lose everything” or “everything that matters will be destroyed” or “everyone I know will die” or “my culture will die” or “my family will die” or simply “I will die.” The thoughts these people had probably felt considerably more devastating than the thoughts I’m having these days. Heck, some people get terrified just reading the news. I’m not alone in worrying about the future.

Again: “I’m not alone in worrying about the future.” I find this immensely comforting. People are not at all oblivious to the world having problems, even if they disagree with me on which problems are the most important. Everyone has fears.
What links here?
- JakubK's comment on All AGI Safety questions welcome (especially basic ones) [~monthly thread] by Robert Miles (4 Nov 2022 10:10 UTC; 2 points)

JakubK 8 Aug 2022 16:16 UTC
1 point
0
in reply to: Multicore’s comment on: Can we get full audio for Eliezer’s conversation with Sam Harris?
Yes. Here’s how I imagine some people will respond to getting a link to this video. “Oh, it’s some weird YouTube video with capitalized words in the title and 265 views.” And the channel is called “Thinking Atheist” and has very few subscribers. It’s way less likely to be taken seriously than the full audio on the official podcast.

Also, it’s hard to listen to YouTube videos when moving around because people can’t (easily) download them. And people have to keep their screen on the whole time (and not use their phone for any other purpose) unless they have some premium YouTube subscription.

JakubK 9 Aug 2022 15:21 UTC
1 point
0
in reply to: gjm’s comment on: Can we get full audio for Eliezer’s conversation with Sam Harris?
I mean that a YouTube video (especially this YouTube video) is not good enough for sharing with people (at least in my case), so I want an audio version from a source that looks somewhat official. (I want something “more than this.”)

When I do a short “yudkowsky harris” search on YouTube and Google, this is the only full version of the interview that I can find.

JakubK 19 Aug 2022 20:07 UTC
2 points
0
in reply to: Jonas Hallgren’s comment on: Announcing the Distillation for Alignment Practicum (DAP)
(I want to learn infra-bayesianism can someone please bushwack this).
You might be interested in this post.

JakubK 7 Sep 2022 23:56 UTC
3 points
0
in reply to: Vladimir_Nesov’s comment on: The AGI has to actually ‘care’ about humans
With enough wealth to trivially save all monkeys, no monkeys would go extinct, provided we care even a little bit more than precisely not at all.
I’m confused about this point. Perhaps you mean wealth in a broad sense that includes “we don’t need to worry about getting more wood.” But as long as wood is a useful resource that humans could use more of to acquire more wealth and do other things that we value more than saving monkeys, then we will continue to take wood from the monkeys. Likewise, even if an AGI values human welfare somewhat, it will still take our resources as long as it values other things more than human welfare.

I found the monkey example much more compelling than “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.” Taking resources from humans seems more likely than using humans as resources.

JakubK 8 Sep 2022 23:04 UTC
1 point
0
in reply to: Thomas Larsen’s comment on: (My understanding of) What Everyone in Technical Alignment is Doing and Why
Maybe the “AI Watch” page could incorporate ideas from this post and serve as an equivalent to “a living document on GitHub.”

JakubK 18 Sep 2022 1:22 UTC
1 point
0
on: AI Safety field-building projects I’d like to see
Reposting part of my comment from the EA Forum since I’m confused why you’re excited about this particular idea.
Something like a big version of SERI-Mats … (My impression is that SERI-Mats could become this one day, but I’d also be excited to see more programs “compete” with SERI-Mats).
At EAG-SF I asked a MATS organizer if we could get other versions of MATS, e.g. a MATS competitor at MIT. Their response was that only one of the two could survive because there are currently only ~15 people capable of doing this kind of mentorship. Mentors are the bottleneck for scaling up programs like MATS, not field builders.

JakubK 20 Sep 2022 4:59 UTC
1 point
0
in reply to: Evie Cottrell’s comment on: Seven ways to become unstoppably agentic
Wow, I have exactly the opposite experience. I check Twitter once per week for 15-60 minutes, and the main goal is to get new ideas from smart people.

JakubK 25 Oct 2022 3:17 UTC
1 point
in reply to: Bezzi’s comment on: Why so little AI risk on rationalist-adjacent blogs?
Scott Aaronson started working on AI safety.

JakubK 4 Nov 2022 9:48 UTC
1 point
0
in reply to: chanamessinger’s comment on: All AGI Safety questions welcome (especially basic ones) [~monthly thread]
Paul’s comment here is relevant, but I’m also confused.

JakubK 4 Nov 2022 10:10 UTC
2 points
0
in reply to: Cookiecarver’s comment on: All AGI Safety questions welcome (especially basic ones) [~monthly thread]
I wrote a relevant comment elsewhere. Basically I think that finding joy in the face of AGI catastrophe is roughly the same task as finding joy in the face of any catastrophe. The same human brain circuitry is at work in both cases.
In turn, I think that finding joy in the face of a catastrophe is not much different than finding joy in “normal” circumstances, so my advice would be standard stuff like aerobic exercise, healthy food, sufficient sleep, supportive relationships, gratitude journaling, meditation, etc (of course, this advice is not at all tailored to you and could be bad advice for you).

Also, AGI catastrophe is very much impending. It’s not here yet. In the present moment, it doesn’t exist. It might be worth reflecting on the utility of worry and how much you’d like to worry each day.

JakubK 4 Nov 2022 10:16 UTC
1 point
0
in reply to: trevor’s comment on: All AGI Safety questions welcome (especially basic ones) [~monthly thread]
So the same question? :P

I’d imagine the AGI Safety Fundamentals introductory governance curriculum (Google doc) is somewhat optimized for getting people up to speed quickly.

JakubK 6 Nov 2022 6:00 UTC
1 point
0
in reply to: owencb’s comment on: PSA about differential technological development
Don’t we already understand the basic shape of things that will get to AGI? Seems plausible that AGI will consist of massive transformer models within some RLHF approach. But maybe you’re looking for something more specific than that.

JakubK 6 Nov 2022 6:24 UTC
1 point
0
on: PSA about differential technological development
Here’s my naive summary; please critique if I’m misunderstanding. EDIT: I think I’m framing things differently from the post. Here’s a separate comment with my confusions about the post’s framing.
- Say there are 3 parts/chunks/segments of solving alignment that have to happen sequentially (each one can only start once the previous part is complete), and each part requires at least 5 years no matter how many people we throw at it. In this world, you need at least 15 years to solve alignment.
  - Why does each part take at least 5 years? Because of diminishing returns. There are constraints on how much you can parallelize the work, so eventually new people can’t do anything but work on projects that other people are already working on (in a way that doesn’t speed up progress on these projects).
    I’m not sure how rapidly returns will diminish. Researcher retreats, operations work, personal assistants, AI tools, etc could continue to shorten the time to complete each chunk even after technical researchers saturate the field. Regardless, there is some minimum length of time required across all realistic worlds.
- If your research project helps with parallelizing the current chunk so that we can do it in e.g. 7 years instead of 8 years, that seems great—except what if it also decreases timelines from 16 years to 14 years? Then it’s a death sentence.
  - Shortening timelines burns/reduces our serial time—the time we have left to complete all 3 chunks (basically a reframing of “timelines”?). Trading serial time for more labor (parallelized to solve the current chunk) is extremely dangerous, even though it helps us complete the current chunk faster.
- How might progress be serial? Maybe we need to deeply understand minds/optimization/agency before we can start coming up with alignment proposals that actually work. Within each of these steps there are more serial substeps. Also, progressing from one major step to the next might require large “paradigm shifts,” and these shifts might not happen without some fresh eyes entering the field (due to shortcomings of human psychology?), which might take several years at a minimum.

JakubK 6 Nov 2022 8:45 UTC
2 points
0
on: Instead of technical research, more people should focus on buying time
1. What does it mean to buy “end time”? If an action results in a world with longer timelines, then what does it mean to say that the additional time comes “later”?
2. What is “serial alignment research”? I’m struggling to distinguish it from “alignment research.”
3. Can you clarify the culture change you want to see? We should think of buying time as “better” than “(traditional) technical AI safety research and (traditional) community-building”?
Less-important comments:
- Can you be more specific about “ODA+”? Does it include Meta and Google Brain, or only the most safety-conscious labs?
- I’m confused why field building isn’t listed as one of the key benefits other than buying time. Publishing more work like the goal misgeneralization paper would make field building more successful.
- In the graph (“Figure 1”), I’m parsing “technical alignment” as “technical research on the core challenges of alignment that avoids shortening timelines,” because the impact of talented researchers depends heavily on the projects they pursue.
- PaLM would have been just as fast if they chose not to publish the PaLM paper. (The sentence about PaLM is confusing.)

JakubK 7 Nov 2022 2:08 UTC
2 points
0
in reply to: ThomasW’s comment on: Instead of technical research, more people should focus on buying time
There will always be more to do to increase certainty and reliability
I’m confused why this is an objection. I agree that the authors should be specific about what it means to “solve the problem,” but all they need is a definition like “<10% chance of AI killing >1 billion people within 5 years of the development of AGI.”

JakubK 7 Nov 2022 15:48 UTC
7 points
0
in reply to: ThomasW’s comment on: Instead of technical research, more people should focus on buying time
I worry that people will skip the post, read this comment, and misunderstand the post, so I want to point out how this comment might be misleading, even though it’s a great comment.
None of the interventions in the post are “go work at OpenAI to change things from the inside.” And only the outreach ones sound anything like “going around and convincing others.” And there’s a disclaimer that these interventions have serious downside risks and require extremely competent execution.

EDIT: one idea in the 2nd post is to join safety and governance teams at top labs like OpenAI. This seems reasonable to me? (“Go work on capabilities at OpenAI to change things” would sound unreasonable.)

JakubK 7 Nov 2022 17:07 UTC
4 points
0
on: PSA about differential technological development
I’m quite confused about the toy model (1000 person-years of research, 900 of which “can be done in parallelizable 5-year chunks,” 100 of which “factor into four chunks that take 25 serial years apiece”).
1. What is a “person-year of research”? I’m parsing as a year of progress from someone working at an optimal level with all the necessary background knowledge. “All the necessary background knowledge” is overly simplistic, as I can imagine plenty of subproblems that take X years for one person to solve, but the 2nd year relies on knowledge in the 1st, etc. Also a year of research can accomplish much more depending on interaction effects with collaborators.
2. There’s an example of 180 people each working for 5 years to do the 900 parallelizable person years, or alternatively 1800 people working for 10 years if 10% do their job correctly half the time. What if we had 1800 people doing their job correctly all the time—would that still take 5 years in this model?
3. The remaining 100 person-years are also parallelizable into 25-year chunks, no? It seems that “more parallelizable vs less parallelizable” is a better framing than “parallelizable vs serial.”
4. It seems that the core idea here is that some problems require a minimum length of time to solve due to relevant subtasks needing to happen sequentially. This still applies to the “parallelizable 5-year chunks”—they each take a minimum of 5 years.
Also, can someone please define the terms “serial time” and “serial alignment research”?
What links here?
- JakubK's comment on Thoughts on AGI organizations and capabilities work by RobBensinger (EA Forum; 11 Dec 2022 8:09 UTC; 1 point)
- JakubK's comment on A note about differential technological development by So8res (6 Nov 2022 6:24 UTC; 1 point)

JakubK 16 Nov 2022 23:30 UTC
3 points
0
in reply to: trevor’s comment on: The Alignment Community Is Culturally Broken
If halving the annual R&D of the global AI industry is equivalent to doubling the length of time before AGI, then I think that would be substantially more valuable than doubling the number of people working on alignment. I don’t think “grow alignment researchers by X” and “lengthen timelines by X” are equally valuable.

JakubK 16 Nov 2022 23:40 UTC
5 points
3
on: The Alignment Community Is Culturally Broken
I want to address the central flaw of Akash+Olivia+Thomas’s argument in the Buying Time post, which is that actually, people can improve at things.
Why do you think the Buying Time post suggested otherwise? Note that the authors did not make a blanket statement telling “bad” alignment researchers to stop working on alignment. One recommendation is for alignment researchers to consider working on technical research that concretely presents important alignment concepts or empirically demonstrates concerning alignment failures.