DMs open.
Cleo Nardo
Karma: 5,968
Three thoughts on civilisational handoff
Boo.
It’s so reasonable for labs to coordinate on banning neuralese. Let’s not say “Features that future AIs will have” as if this is some immutable inevitable event. It’s not, we can do better.
Alternative titles:
“Features of future AIs that we could collectively decide to avoid”
“Features of future AIs that governments should consider regulating”
“Features of future AIs that specific people will be accountable for deciding”
“Features of future AIs that a wise society would navigate with caution”
“Features of future AIs that only a woated netnegativecel would build without broad consensus”
“Features of future AIs whose pros and cons you should discuss with your lab employee friends”
Etc
You think “EAs should’ve been more cautious about waking up various external groups like the public, academia, industry, governments” and “EAs should’ve been less power-seeking because other people will notice those power-seeking patterns and there will be a backlash”. Maybe there’s a tension there.
Memes do actual epistemic work (e.g. this post, I claim) and on the margin I‘d prefer more memes (in absolute, not proportional, terms). But some cultures only incentivise memes, which would suck for this community bc much of our work is by necessity unfunny.
Fwiw I think our community rewards unfunny stuff, so I’m not too worried. For example, the top LW post is Alex Turner’s which has zero jokes.
The dots separate the first group (which were shared around a few slack meme channels) from the second group (which I added before sharing publicly).
I don’t think this is a big component of the explanation.
Alex Turner: I wasted thousands of hours proving my P theorems. (I still believe P but for different reasons.)
Arguments for P
Yeah I think if someone was truly evaluating projects on INT then this is a decent case for bumping T slightly. In fact, my guess is EAs are already overweighting T because people care about career capital and status. So correcting down on T would increase their impact.
Moreover, I don’t understand the impact case for individuals focusing on their own career capital versus something like “[people with my values] capital”. Like, it’s important for the EA/AIS community overall to have big legible wins, but maybe this implies individual EAs should be focused on the 1-10% range.
“Replaceable, wishes he wasn’t” vs “irreplaceable, wishes he wasn’t” 🙇♂️
Maybe you should do things with a lower probability of success. I think most people doing projects which have a 50%+ chance of succeeding, which is probably a good idea for your career and status. But it might be easier to farm EV in the 1-10% range. This is all very abstract so I’m not sure this is helpful advice to anyone.
(Here’s smth I wrote a month ago.)
Plan A might be a kinda rough situation, given you’d have millions of people lobbying for the project to go faster. Like, 30% of the pop (60-89) is taking an additional 30% chance of death if the singularity happens after 20 years compared with 10 years.
One solution is to cryo everyone 60+. I’m not entirely dissuaded by “cryo is too sci fi for them” because the original worry is that the cohort is lobbying for an early singularity! Maybe I should invest in some cryo startups. I bet even Leopold hasn’t backchained this far.
A similar solution is a “soft cryo”, i.e. medically induced comas. It seems pretty plausible that during an intelligence explosion, doctors will say “Keep that patient on life support for another year, the medicine will be better.”
Another solution is simplying paying risk premiums to the 60-89 age range, to compensate them / stop them lobbying for early foom.
AI has recently solved a number of conjectures. I haven’t checked this, but it seems that these are almost all resolved false, via the construction of counterexamples.
I had the same problem in the UK. I bought them instead online, at www.visiondirect.co.uk/ which doesn’t need prescription. You’ll need to tell the website which contact lens prescription you want to buy, but Claude can calculate that from your glasses prescription. Then practice at home, and you’ll quickly upskill.
I spent ~2 hours trying before I had put both contacts in and taken both of them out. This time was split over like two sitting.
Much of Damon Binder’s work is based on a methodology of speculative engineering, i.e. thinking about the feasibility of various ways to solve a problem. See:
How much this kind of work update [smart, reasonable] people on the questions he discusses?
How reliable is this methodology? I’m not sure.
It seems IMO easy for this methodology to conclude “X can’t happen because I enumerated the ways you’d do X and none of them work” and then X does happen because you missed something.
Or, “X can happen because here’s a method which works” but then X can’t happen because the method actually doesn’t work.
Maybe this is a skill issue, i.e. if I was as smart as Damon Binder I could see that, yes, that really is an exhaustive enumeration of the ways X could happen. And yes, that really is the feasibility of each method.
What’s the best/worst examples of this methodology?
I’m interested in cases where speculative engineering concluded X is feasible (at some point in the tech tree) but it wasn’t — or X isn’t feasible but it was.
What is the relation between Damon Binder and the God of Straight Lines? Where do they disagree? Which diety should I trust more?
I don’t think this proposal should be high priority for labs, compared to (a) ensuring that your monitoring system doesn’t have more direct gaps/holes, and (b) making sure your monitors actually catch the things you’re worried about in realistic settings.
Your proposal might be high priority because it has more bang for your buck because the other things seem much harder. Maybe I’m underestimating how hard this proposal would be to implement.
Yeah, I meant Carl Shulman. Well-spotted, thanks.
Yep, one hope is “EAs are good at forecasting how things will go and that maintains our influence. Both because we have a reputation as good forecasters so people listen to us, and because we make good decisions based on those forecasts (i.e. a similar arbitrage play as the “AI will be a bigger deal than people think, let’s pile into it”)”.
But I’m worried that [the top few] EAs are approaching the Horizon Point beyond which their impressive forecasting breaks down. Like, 10 years ago, Carl Schulman and Paul Christiano [and people in that reference class] could see in such clarity how things would go in 10 years time. But today, they can’t see another 10 years into the future. Like, maybe a few years into the future? Even there I’m sceptical. They might have run out of “alpha”.
For a concrete example, Daniel’s “what 2026 looks like” was pretty on-the-nose. But AI 2027 isn’t nearly so impressive — the distributions are so much broader, and Daniel himself is like “here’s a bunch of ways things could go.” AI 2026 didn’t have multiple scenarios!
One problem here is that our alpha was predicting the change in the technical landscape. But it’s not just technical landscape that will change. It‘s also the political landscape, which has been dormant for most of 2019-2026. And soon the geopolitical landscape will start rumbling. And the economic. And the cultural. I think these things are still pretty dormant, compared to how much they will start flipping the overall strategic landscape. (See here.)
IIUC, this is why Daniel’s current forecasts are choose-your-adventure, compared with his 2021 forecasts about 2021-2026.
Like, do we have good forecasts now which weren’t ~ “priced in” 5 years ago among Carl/Paul/Kokotajlo/etc? I feel like the uncertainties they had 5 years ago are pretty much still unresolved.
Farmers should sell their grain to the highest bidder, because after a bad harvest this induces more supply of grain via imports. The world is better when farmers act like this.
Lab employees should publicly refuse to work on neuralese, because this would reduce the chance of human extinction. The world is better when lab employees act like this.
I won’t be shouting “lab employees will fail to coordinate on refusing neuralese development” when I’m actually pretty unsure and haven’t presented any justification for this claim. It’s a pretty reasonable to anticipate people will coordination on not doing bad things.