An idea that I had later which I didn’t end up saying to Ozzie at the retreat was something like a “Good Judgment Project for high schoolers”, in the same way that there are math contests and programming contests and so on. I would be really interested in seeing what happens if we can identify people who would be superforecasters as adults when they’re still teens or in undergrad or whatever, and then point them towards a career in forecasting / have them work together to build up the art, and this seems like a project that’s “normal” enough to catch on while still doing something interesting.
Vaniver
Karma: 27,975
(there wasn’t a clear case for how this would happen AFAICT, just ‘i dunno neural net magic might be able to help.’ I don’t expect neural-net magic to help here in the next 10 years but I could see it helping in the next 20 or 30. I’m not sure if it happens much farther in advance than “actual AGI” though)
I thought Ozzie’s plan here was closer to “if you have a knowledge graph, you can durably encode a lot of this in ways that transfer between questions”, and you can have lots of things where you rapidly build out a suite of forecasts with quantifiers and pointers. I thought “maybe NLP will help you pick out bad questions” but I think this is more “recognizing common user errors” than it is “understanding what’s going on.”
Conversation on forecasting with Vaniver and Ozzie Gooen
Both low-carb and keto are improved by eating lots of leafy green vegetables, I think. The main question is something like “are you getting more sugar or fiber out of this?”. Plant-free diets, of course, recommend against eating vegetables.
A quick google search gave me an estimate of a 300:1 cost ratio of iron to silver (1 lb iron costing ~300 lb silver) in the 14th century.
You have it inverted; your link says:
That is, a (pure!) silver pound would buy 300 pounds of iron or steel.
The first half of #1 I got right, but I think the second half was more wrong than right. While this might be fast enough to be useful in a crisis, it looks like the design is focused more on getting useful information out of regions rather than the ‘gross functionality’ target I mentioned there.
I think the big result here was that they came up with a way to do deep insertion of wires designed for biocompatibility and longevity, which is impressive and along a dimension I wasn’t tracking too much in my prediction. In retrospect, I might have updated too much on the article I read beforehand, which gave me the sense that this was closer to ‘a medical startup that got Musk’s money’ than ‘the thing Musk said he was trying to do, which will try to be useful along the way’, which is what the white paper looks more like.
White paper they released.
I don’t have a solid sense of this yet, in large part because of how much of it is experiential.
I think I would count the 5-second level as gesturing in this direction; I also note the claim that HPMOR lets people ‘experience’ content from the Sequences instead of just read it. Some friends who did (old-style) boxing described it as calibrating their emotional reactions to danger and conflict in a way that seems related.
I’ve been experimenting with conceptualizing some of my long-standing dilemmas as questions of the form “does this desire have merit?” as opposed to something closer to “should I do A or B?”, but it’s too soon to see if that’s the right approach.
See also: Role are Martial Arts For Agency.
Commentary On “The Abolition of Man”
Length of the control vector seems important; there’s lots of ways to use gross signals to control small vectors that don’t scale to controlling large vectors. Basically, you could imagine that question as something like “could you dance with it?” (doable in 2014) or “could you play a piano with it?” (doable in 2018), both of which naively seem more complicated than an (x,y) pair (at least, when you don’t have visual feedback).
I predict with moderate confidence that we will not see:
‘Augmented reality’-style overlays or video beamed directly to the visual cortex.
Language output (as text or audio or so on) or input.
Pure tech or design demos without any demonstrations or experiments with real biology.
I predict with weak confidence that we won’t see results in humans. (This prediction is stronger the more invasive the results we’re seeing; a superior EEG they could show off in humans, but repair or treatment of strokes will likely only be in mice.)
(Those strike me as the next milestones along the ‘make BCIs that are useful for making top performers higher performing’ dimension, which seems to be Musk’s long-term vision for Neuralink.)
They’ve mostly been focusing on medical applications. So I predict we will see something closer to:
High-spatial-fidelity brain monitoring (probably invasive?), intended to determine gross functionality of different regions (perhaps useful in conjunction with something like ultrasound to do targeted drug delivery for strokes).
Neural prostheses intended to replace the functionality of single brain regions that have been destroyed. (This seems more likely for regions that are somehow symmetric or simple.)
Results in rats or mice.
I notice I wanted to put ‘dexterous motor control’ on both lists, so I’m somehow confused; it seems like we already have prostheses that perform pretty well based on external nerve sites (like reading off what you wanted to do with your missing hand from nerves in your arm) but I somehow don’t expect us to have the spatial precision or filtering capacity to do that in the brain. (And also it just seems much riskier to attach electrodes internally or to the spinal cord than at an external site, making it unclear why you would even want that.) The main question here for me is something closer to ‘bandwidth’, where it seems likely you can pilot a drone using solely EEG if the thing you’re communicating is closer to “a location that you should be at” than “how quickly each of the four rotors should be spinning in what direction.” But we might have results where rats have learned how to pilot drones using low-level controls, or something cool like that.
Specifically, ‘urgent’ is measured by the difference between the time you have and the time it will take to do. If I need the coffee to be done in 15 minutes and the bread to be done in an hour, but if I want the bread to be done in an hour I need to preheat the oven now (whereas the coffee only takes 10 minutes to brew start to finish) then preheating the oven is urgent whereas brewing the coffee has 5 minutes of float time. If I haven’t started the coffee in 5 minutes, then it becomes urgent. See critical path analysis and Gantt charts and so on.
This might be worth a post? It feels like it’d be low on my queue but might also be easy to write.
I mostly agree with your analysis; especially the point about 1 (that the more likely I think my thoughts are to be wrong, the lower cost it is to share them).
I understand that there are good reasons for discussions to be private, but can you elaborate on why we’d want discussions about privacy to be private?
Most examples here have the difficulty that I can’t share them without paying the costs, but here’s one that seems pretty normal:
Suppose someone is a student and wants to be hired later as a policy analyst for governments, and believes that governments care strongly about past affiliations and beliefs. Then it might make sense for them to censor themselves in public under their real name because of potential negative consequences of things they said when young. However, any statement of the form “I specifically want to hide my views on X” made under their real name has similar possible negative consequences, because it’s an explicit admission that the person has something to hide.
Currently, people hiding their unpopular opinions to not face career consequences is fairly standard, and so it’s not that damning to say “I think this norm is sensible” or maybe even “I follow this norm,” but it seems like it would have been particularly awkward to be first person to explicitly argue for that norm.
Do we have those generally trusted arbiters? I note that it seems like many people who I think of as ‘generally trusted’ are trusted because of some ‘private information’, even if it’s just something like “I’ve talked to Carol and get the sense that she’s sensible.”
[Note: this, and all comments on this post unless specified otherwise, is written with my ‘LW user’ hat on, not my ‘LW Admin’ or ‘MIRI employee’ hat on, and thus is my personal view instead of the LW view or the MIRI view.]
As someone who thinks about AGI timelines a lot, I find myself dissatisfied with this post because it’s unclear what “The AI Timelines Scam” you’re talking about, and I’m worried if I poke at the bits it’ll feel like a motte and bailey, where it seems quite reasonable to me that ’73% of tech executives thinking that the singularity will arrive in <10 years is probably just inflated ‘pro-tech’ reasoning,′ but also it seems quite unreasonable to suggest that strategic considerations about dual use technology should be discussed openly (or should be discussed openly because tech executives have distorted beliefs). It also seems like there’s an argument for weighting urgency in planning that could lead to ‘distorted’ timelines while being a rational response to uncertainty.
On the first point, I think the following might be a fair description of some thinkers in the AGI space, but don’t think this is a fair summary of MIRI (and I think it’s illegible, to me at least, whether you are intending this to be a summary of MIRI):
This bears similarity to some conversations on AI risk I’ve been party to in the past few years. The fear is that Others (DeepMind, China, whoever) will develop AGI soon, so We have to develop AGI first in order to make sure it’s safe, because Others won’t make sure it’s safe and We will. Also, We have to discuss AGI strategy in private (and avoid public discussion), so Others don’t get the wrong ideas. (Generally, these claims have little empirical/rational backing to them; they’re based on scary stories, not historically validated threat models)
I do think it makes sense to write more publicly about the difficulties of writing publicly, but there’s always going to be something odd about it. Suppose I have 5 reasons for wanting discussions to be private, and 3 of them I can easily say. Discussing those three reasons will give people an incomplete picture that might seem complete, in a way that saying “yeah, the sum of factors is against” won’t. Further, without giving specific examples, it’s hard to see which of the ones that are difficult to say you would endorse and which you wouldn’t, and it’s not obvious to me legibility is the best standard here.
But my simple sense is that openly discussing whether or not nuclear weapons were possible (a technical claim on which people might have private information, including intuitions informed by their scientific experience) would have had costs and it was sensible to be secretive about it. If I think that timelines are short because maybe technology X and technology Y fit together neatly, then publicly announcing that increases the chances that we get short timelines because someone plugs together technology X and technology Y. It does seem like marginal scientists speed things up here.
Now, I’m paying a price here; it may be the case that people have tried to glue together technology X and technology Y and it won’t work. I think private discussions on this are way better than no discussions on this, because it increases the chances that those sorts of crucial facts get revealed. It’s not obvious that public discussions are all that much better on these grounds.
On the second point, it feels important to note that the threshold for “take something seriously” is actually quite small. I might think that the chance that I have Lyme disease is 5%, and yet that motivates significant action because of hugely asymmetric cost considerations, or rapid decrease in efficacy of action. I think there’s often a problem where someone ‘has short timelines’ in the sense that they think 10-year scenarios should be planned about at all, but this can be easily mistaken for ‘they think 10-year scenarios are most likely’ because often if you think both an urgent concern and a distant concern are possible, almost all of your effort goes into the urgent concern instead of the distant concern (as sensible critical-path project management would suggest).
On 3, I notice this part of your post jumps out to me:
Of course, I’d have written a substantially different post, or none at all, if I believed the technical arguments that AGI is likely to come soon had merit to them
One possibility behind the “none at all” is that ‘disagreement leads to writing posts, agreement leads to silence’, but another possibility is ‘if I think X, I am encouraged to say it, and if I think Y, I am encouraged to be silent.’
My sense is it’s more the latter, which makes this seem weirdly ‘bad faith’ to me. That is, suppose I know Alice doesn’t want to talk about biological x-risk in public because of the risk that terrorist groups will switch to using biological weapons, but I think Alice’s concerns are overblown and so write a post about how actually it’s very hard to use biological weapons and we shouldn’t waste money on countermeasures. Alice won’t respond with “look, it’s not hard, you just do A, B, C and then you kill thousands of people,” because this is worse for Alice than public beliefs shifting in a way that seems wrong to her.
It is not obvious what the right path is here. Obviously, we can’t let anyone hijack the group epistemology by having concerns about what can and can’t be made public knowledge, but also it seems like we shouldn’t pretend that everything can be openly discussed in a costless way, or that the costs are always worth it.
One side note: I’ve been surprised by how much the presentation differed between the copy I originally read (Brian Walker’s translation) and various “get I Ching readings online” sites that I’ve gone to over the years. It might be worth looking at a few different translations to find the one that fits you best.
It definitely makes sense to track “am I discovering anything new?”, as measured by “I changed my plans” or “I explored fruitfully” or “my emotional orientation towards X improved” (instead of merely changing). It seems worth comparing to other retrospective / prospective analyses you might try; in the same way that one diet should be compared against other diets (not just on grounds of nutrition, but also enjoyment and convenience and so on).
I also attempted to track “how much of a stretch was the scenario/perspective/etc.?”, where sometimes it would be right-on and other times I could kind of see it and other times my sense was “nope, that’s just not resonating at all.” If something is resonating too much, either you have a run of luck that’s unseasonably long or you’re getting prompts that aren’t specific enough to be wrong. If you’re trying to train the skill of discernment, you need both to notice when things are right and wrong, and thinking that it’s right is worthless unless sometimes you also think it’s wrong.
People don’t customize their houses all that much, to the degree they do it doesn’t get them very good returns on well being per dollar spent, and they pay larger well being costs from the aforementioned commute and career inflexibility problems.
I feel conflicting desires here, to point out that this sometimes happens, and to worry that this is justifying a bias instead of correcting it. For example, I switched from ‘wanting to rent’ to ‘wanting to buy’ when I realized that I would benefit a lot from having an Endless Pool in my house, and that this wasn’t compatible with renting (unless I could find a place that already had one, or whose owner wanted one, or so on). But also that this convinced me doesn’t mean that most people who are convinced are correctly convinced. It might be better for me to do whatever ‘looking seriously at the price differential’ and deciding to invest in the policy of walking to the local pool more instead; but I think actually money isn’t the main thing here. (Like, for a while I thought it was better to be in Austin than in the Bay because a software engineer would earn about $10k/yr more all things considered, and then after thinking about it realized that I was happy paying $10k/yr to be in the Bay instead.)
Basically, whether you think it’s primarily related to alignment vs. rationality. (Everything on the AF is also on LW, but the reverse isn’t true.) The feedback loop if you’re posting too much or stuff that isn’t polished enough is downvotes (or insufficient upvotes).