Steven K
steven0461
Karma: 8,760
Anonymous #1 asks:
This one is not technical: now that we live in a world in which people have access to systems like ChatGPT, how should I consider any of my career choices, primarily in the context of a computer technician? I’m not a hard-worker, and I consider that my intelligence is just a little above average, so I’m not going to pretend that I’m going to become a systems analyst or software engineer, but now code programming and content creation are starting to be automated more and more, so how should I update my decisions based on that?
Sure, this question is something that most can ask about their intellectual jobs, but I would like to see answers from people in this community; and particularly about a field in which, more than most, employers are going to expect any technician to stay up-to-date with these tools.
Here’s a form you can use to send questions anonymously. I’ll check for responses and post them as comments.
From 38:58 of the podcast:
So I do think that over time I have come to expect a bit more that things will hang around in a near human place and weird shit will happen as a result. And my failure review where I look back and ask — was that a predictable sort of mistake? I feel like it was to some extent maybe a case of — you’re always going to get capabilities in some order and it was much easier to visualize the endpoint where you have all the capabilities than where you have some of the capabilities. And therefore my visualizations were not dwelling enough on a space we’d predictably in retrospect have entered into later where things have some capabilities but not others and it’s weird. I do think that, in 2012, I would not have called that large language models were the way and the large language models are in some way more uncannily semi-human than what I would justly have predicted in 2012 knowing only what I knew then. But broadly speaking, yeah, I do feel like GPT-4 is already kind of hanging out for longer in a weird, near-human space than I was really visualizing. In part, that’s because it’s so incredibly hard to visualize or predict correctly in advance when it will happen, which is, in retrospect, a bias.
trevor has already mentioned the Stampy project, which is trying to do something very similar to what’s described here and wishes to join forces.
Right now, Stampy just uses language models for semantic search, but the medium-term plan is to use them for text generation as well: people will be able to go to chat.stampy.ai or chat.aisafety.info, type in questions, and have a conversational agent respond. This would probably use a language model fine-tuned by the authors of Cyborgism (probably starting with a weak model as a trial, then increasingly strong ones as they become available), with primary fine-tuning on the alignment literature and hopefully secondary fine-tuning on Stampy content. A question asked in chat would be used to do an extractive search on the literature, then the results would be put into the LM’s context window and it would generate a response.
Stampy welcomes volunteer developers to help with building the conversational agent and a front end for it, as well as volunteers to help write content.
- All AGI Safety questions welcome (especially basic ones) [April 2023] by (EA Forum; Apr 8, 2023, 4:21 AM; 111 points)
- All AGI Safety questions welcome (especially basic ones) [April 2023] by (Apr 8, 2023, 4:21 AM; 57 points)
- All AGI Safety questions welcome (especially basic ones) [July 2023] by (Jul 20, 2023, 8:20 PM; 38 points)
- All AGI Safety questions welcome (especially basic ones) [May 2023] by (May 8, 2023, 10:30 PM; 33 points)
- All AGI Safety questions welcome (especially basic ones) [May 2023] by (EA Forum; May 8, 2023, 10:30 PM; 19 points)
- All AGI Safety questions welcome (especially basic ones) [July 2023] by (EA Forum; Jul 19, 2023, 6:08 PM; 12 points)
- 's comment on Stampy’s AI Safety Info—New Distillations #4 [July 2023] by (Aug 16, 2023, 11:55 PM; 4 points)
- 's comment on Vaniver’s Shortform by (May 8, 2023, 10:02 PM; 4 points)
There’s another issue where “P(doom)” can be read either as the probability that a bad outcome will happen, or the probability that a bad outcome is inevitable. I think the former is usually what’s meant, but if “P(doom)” means “the probability that we’re doomed”, then that suggests the latter as a distracting alternative interpretation.
In terms of “and those people who care will be broad and varied and trying their hands at making movies and doing varied kinds of science and engineering research and learning all about the world while keeping their eyes open for clues about the AI risk conundrum, and being ready to act when a hopeful possibility comes up” we’re doing less well compared to my 2008 hopes. I want to know why and how to unblock it.
I think to the extent that people are failing to be interesting in all the ways you’d hoped they would be, it’s because being interesting in those ways seems to them to have greater costs than benefits. If you want people to see the benefits of being interesting as outweighing the costs, you should make arguments to help them improve their causal models of the costs, and to improve their causal models of the benefits, and to compare the latter to the former. (E.g., what’s the causal pathway by which an hour of thinking about Egyptology or repairing motorcycles or writing fanfic ends up having, not just positive expected usefulness, but higher expected usefulness at the margin than an hour of thinking about AI risk?) But you haven’t seemed very interested in explicitly building out this kind of argument, and I don’t understand why that isn’t at the top of your list of strategies to try.
As far as I know, this is the standard position. See also this FAQ entry. A lot of people sloppily say “the universe” when they mean the observable part of the universe, and that’s what’s causing the confusion.
I have also talked with folks who’ve thought a lot about safety and who honestly think that existential risk is lower if we have AI soon (before humanity can harm itself in other ways), for example.
It seems hard to make the numbers come out that way. E.g. suppose human-level AGI in 2030 would cause a 60% chance of existential disaster and a 40% chance of existential disaster becoming impossible, and human-level AGI in 2050 would cause a 50% chance of existential disaster and a 50% chance of existential disaster becoming impossible. Then to be indifferent about AI timelines, conditional on human-level AGI in 2050, you’d have to expect a 1⁄5 probability of existential disaster from other causes in the 2030-2050 period. (That way, with human-level AGI in 2050, you’d have a 1⁄2 * 4⁄5 = 40% chance of surviving, just like with human-level AGI in 2030.) I don’t really know of non-AI risks in the ballpark of 10% per decade.
(My guess at MIRI people’s model is more like 99% chance of existential disaster from human-level AGI in 2030 and 90% in 2050, in which case indifference would require a 90% chance of some other existential disaster in 2030-2050, to cut 10% chance of survival down to 1%.)
“Safewashing” would be more directly parallel to “greenwashing” and sounds less awkward to my ears than “safetywashing”, but on the other hand the relevant ideas are more often called “AI safety” than “safe AI”, so I’m not sure if it’s a better or worse term.
Yes, my experience of “nobody listened 20 years ago when the case for caring about AI risk was already overwhelmingly strong and urgent” doesn’t put strong bounds on how much I should anticipate that people will care about AI risk in the future, and this is important; but it puts stronger bounds on how much I should anticipate that people will care about counterintuitive aspects of AI risk that haven’t yet undergone a slow process of climbing in mainstream respectability, even if the case for caring about those aspects is overwhelmingly strong and urgent (except insofar as LessWrong culture has instilled a general appreciation for things that have overwhelmingly strong and urgent cases for caring about them), and this is also important.
after a tech company singularity,
I think this was meant to read “2. after AGI,”
Note that the full 2021 MIRI conversations are also available (in robot voice) in the Nonlinear Library archive.
Many entries on the Nonlinear Library
Maybe Towards Data Science
edit: also FLI’s AI alignment podcast
As I see it, “rationalist” already refers to a person who thinks rationality is particularly important, not necessarily a person who is rational, like how “libertarian” refers to a person who thinks freedom is particularly important, not necessarily a person who is free. Then literally speaking “aspiring rationalist” refers to a person who aspires to think rationality is particularly important, not to a person who aspires to be rational. Using “aspiring rationalist” to refer to people who aspire to attain rationality encourages people to misinterpret self-identified rationalists as claiming to have attained rationality. Saying something like “person who aspires to rationality” instead of “aspiring rationalist” is a little more awkward, but it respects the literal meaning of words, and I think that’s important.
Great report. I found the high decision-worthiness vignette especially interesting.
I haven’t read it closely yet, so people should feel free to be like “just read the report more closely and the answers are in there”, but here are some confusions and questions that have been on my mind when trying to understand these things:
Has anyone thought about this in terms of a “consequence indication assumption” that’s like the self-indication assumption but normalizes by the probability of producing paths from selves to cared-about consequences instead of the probability of producing selves? Maybe this is discussed in the anthropic decision theory sequence and I should just catch up on that?
I wonder how uncertainty about the cosmological future would affect grabby aliens conclusions. In particular, I think not very long ago it was thought plausible that the affectable universe is unbounded, in which case there could be worlds where aliens were almost arbitrarily rare that still had high decision-worthiness. (Faster than light travel seems like it would have similar implications.)
SIA and SSA mean something different now than when Bostrom originally defined them, right? Modern SIA is Bostrom’s SIA+SSA and modern SSA is Bostrom’s (not SIA)+SSA? Joe Carlsmith talked about this, but it would be good if there were a short comment somewhere that just explained the change of definition, so people can link it whenever it comes up in the future. (edit: ah, just noticed footnote 13)
SIA doomsday is a very different thing than the regular doomsday argument, despite the name, right? The former is about being unlikely to colonize the universe, the latter is about being unlikely to have a high number of observers? A strong great filter that lies in our future seems like it would require enough revisions to our world model to make SIA doom basically a variant of the simulation argument, i.e. the best explanation of our ability to colonize the stars not being real would be the stars themselves not being real. Many other weird hypotheses seem like they’d become more likely than the naive world view under SIA doom reasoning. E.g., maybe there are 10^50 human civilizations on Earth, but they’re all out of phase and can’t affect each other, but they can still see the same sun and stars. Anyway, I guess this problem doesn’t turn up in the “high decision-worthiness” or “consequence indication assumption” formulation.
My impression (based on using Metaculus a lot) is that, while questions like this may give you a reasonable ballpark estimate and it’s great that they exist, they’re nowhere close to being efficient enough for it to mean much when they fail to move. As a proxy for the amount of mental effort that goes into it, there’s only been three comments on the linked question in the last month. I’ve been complaining about people calling Metaculus a “prediction market” because if people think it’s a prediction market then they’ll assume there’s a point to be made like “if you can tell that the prediction is inefficient, then why aren’t you rich, at least in play money?” But the estimate you’re seeing is just a recency-weighted median of the predictions of everyone who presses the button, not weighted by past predictive record, and not weighted by willingness-to-bet, because there’s no buying or selling and everyone makes only one prediction. It’s basically a poll of people who are trying to get good results (in terms of Brier/log score and Metaculus points) on their answers.
Metaculus (unlike Manifold) is not a market and does not use play money except in the same sense that Tetris score is play money.
Anonymous #2 asks: