I think all AI research makes AGI easier, so “non-AGI AI research” might not be a thing. And even if I’m wrong about that, it also seems to me that most harms of AGI could come from tool AI + humans just as well. So I’m not sure the question is right. Tbh I’d just stop most AI work.
cousin_it(Vladimir Slepnev)
Interesting, your comment follows the frame of the OP, rather than the economic frame that I proposed. In the economic frame, it almost doesn’t matter whether you ban sexual relations at work or not. If the labor market is a seller’s market, workers will just leave bad employers and flock to better ones, and the problem will solve itself. And if the labor market is a buyer’s market, employers will find a way to extract X value from workers, either by extorting sex or by other ways—you’re never going to plug all the loopholes. The buyer’s market vs seller’s market distinction is all that matters, and all that’s worth changing. The great success of the union movement was because it actually shifted one side of the market, forcing the other side to shift as well.
I think this is a good topic to discuss, and the post has many good insights. But I kinda see the whole topic from a different angle. Worker well-being can’t depend on the goodness of employers, because employers gonna be bad if they can get away with it. The true cause of worker well-being is supply/demand changes that favor workers. Examples: 1) unionizing was a supply control which led to 9-5 and the weekend, 2) big tech jobs became nice because good engineers were rare, 3) UBI would lead to fewer people seeking jobs and therefore make employers behave better.
To me these examples show that, apart from market luck, the way to improve worker well-being is coordinated action. So I mostly agree with banning 80 hour workweeks, regulating gig work, and the like. We need more such restrictions, not less. The 32-hour work week seems like an especially good proposal: it would both make people spend less time at work, and make jobs easier to find. (And also make people much happier, as trials have shown.)
I think the main question is how to connect technological progress (which is real) to moral progress (which is debatable). People didn’t expect that technological progress would lead to factory farming or WMDs, but here we are.
I’m worried about centralization of power and wealth in opaque non-human decision-making systems, and those who own the systems.
This has been my main worry for the past few years, and to me it counts as “doom” too. AIs and AI companies playing by legal and market rules (and changing these rules by lobbying, which is also legal) might well lead to most humans having no resources to survive.
I feel like instead of flipping out you could just say “eh, I don’t agree with this community’s views on gender, I’m more essentialist overall”. You don’t actually have to convince anyone or get convinced by them. Individual freedom and peaceful coexistence is fine. The norm that “Bayesians can’t agree to disagree” should burn in a fire.
I’m no longer sure the question makes sense, and to the extent it makes sense I’m pessimistic. Things probably won’t look like one AI taking over everything, but more like an AI economy that’s misaligned as a whole, gradually eclipsing the human economy. We’re already seeing the first steps: the internet is filling up with AI generated crap, jobs are being lost to AI, and AI companies aren’t doing anything to mitigate either of these things. This looks like a plausible picture of the future: as the AI economy grows, the money-hungry part of it will continue being stronger than the human-aligned part. So it’s only a matter of time before most humans are outbid / manipulated out of most resources by AIs playing the game of money with each other.
Amazing post. I already knew that filtered evidence can lead people astray, and that many disagreements are about relative importance of things, but your post really made everything “click” for me. Yes, of course if what people look at is correlated with what they see, that will lead to polarization. And even if people start out equally likely to look at X or Y, but seeing X makes them marginally more likely to look at X in the future rather than Y, then some people will randomly polarize toward X and others toward Y.
I think we’re using at most 1% of the potential of geniuses we already have. So improving that usage can lead to 100x improvement in everything, without the worries associated with 100x population. And it can be done much faster than waiting for people to be born. (If AI doesn’t make it all irrelevant soon, which it probably will.)
I left in 2011. My advice is to leave soon. And not even for reasons of ethics, business, or comfort. More like, for the spirit. Even if Russia is quite comfortable now, in broad strokes the situation is this: you’re young, and the curtain is slowly closing. When you’re older, would you rather be the older person who stayed in, or the person who took a chance on the world?
Unfortunately, the game of power is about ruling a territory, not improving it. It took me many years to internalize this idea. “Surely the elite would want to improve things?” No. Putin could improve Russia in many ways, but these ways would weaken his rule, so he didn’t. That’s why projects like Georgism or charter cities keep failing: they weaken the relative position of the elite, even if they plausibly make life better for everyone. Such projects can only succeed if implemented by a whole country, which requires a revolution or at least a popular movement. It’s possible—it’s how democracy was achieved—but let’s be clear on what it takes.
Not sure I understand. My question was, what kind of probability theory can support things like “P(X|Y) is defined but P(Y) isn’t”. The snippet you give doesn’t seem relevant to that, as it assumes both values are defined.
I think you’re describing a kind of robotic tank, which would be useful for many other things as well, not just clearing mines. But designing a robotic tank that can’t be disabled by an ATGM (some modern mines are already ATGMs waiting to fire) seems like a tall order to me. Especially given that ATGM tech won’t stand still either.
OpenAI has already been the biggest contributor to accelerating the AI race; investing in chips is just another step in the same direction. I’m not sure why people keep assuming Altman is optimizing for safety. Sure, he has talked about safety, but it’s very common for people to give lip service to something while doing the opposite thing. I’m not surprised by it and nobody should be surprised by it. Can we just accept already that OpenAI is going full speed in a bad direction, and start thinking what we can/should do about it?
I think the problem is not about updatelessness. It’s more like, what people want from decision theory is fundamentally unachievable, except in very simple situations (“single player extensive-form games” was my take). In more complex situations, game theory becomes such a fundamental roadblock that we’re better off accepting it; accepting that multiplayer won’t reduce to single player no matter how much we try.
If you say things like “P(X|Y) is defined but P(Y) isn’t”, doesn’t that call for a reformulation of all probability theory? Like, if I take the interpretation of probability theory based on sigma-algebras (which is quite popular), then P(Y) gotta be defined, no way around it. The very definition of P(X|Y) depends on P(X∧Y) and P(Y). You can say “let’s kick out this leg from this table”, but the math tells me pretty insistently that the table can’t stand without that particular leg. Or at least, if there’s a version of probability theory where P(Y) can be undefined but P(X|Y) defined, I’d want to see more details about that theory and how it doesn’t trip over itself. Does that make sense?
Hmm. But in the envelope experiment, once Alice commits to a decision (e.g. choose A), her probabilities are well-defined. So in Sleeping Beauty, if we make it so the day is automatically disclosed to Alice at 5pm let’s say, it seems like her probabilities about it should be well-defined from the get go. Or at least, the envelope experiment doesn’t seem to shed light why they should be undefined. Am I missing something?
The goal is for them to take over unlimited land territory
I’m not sure that goal is achievable. When the drones you’re describing become mature, another pretty horrible technology might become mature as well: smart mines and remote mine-laying systems. A smart mine is cheaper than a drone (because it sits still instead of flying around), much easier to hide and harder to detect (same reason), they can see stuff and talk to each other just like drones can, and they can be deployed at a distance in large numbers.
So that’s the picture I’m imagining. Thousands of mine-laying shells burst over a territory and make it un-takeable, a kind of hostile forest. Your drones will fly over it and see nothing. But the moment your people or vehicles enter the territory, something jumps out of the ground 50 meters away and they’re dead. Or a column of your troops enters, the mines wait and then kill them all at once. Stuff like that.
Not sure there’s any real counter to this. Even in peacetime, removing unexploded dumb bombs and mines from long-past wars (e.g. in Laos) takes more time and money than laying them in the first place. And if the mines fight back, the task of demining probably becomes unrealistic altogether. Especially as the defender can just keep dropping in more mines.
And if you want to talk specifically about “this awakening is happening in the first day of the experiment”, then such probability is undefined for the Sleeping Beauty setting.
Yeah, I don’t know if “undefined” is a good answer.
To be fair, there are some decision-theoretic situations where “undefined” is a good answer. For example, let’s say Alice wakes up with amnesia on 10 consecutive days, and each day she’s presented with a choice of envelope A or envelope B, one of which contains money. And she knows that whichever envelope she chooses on day 1, the experimenter will put money in the other envelope on days 2-10. This case is truly undefined: the contents of the envelopes on the desk in front of Alice are eerily dependent on how Alice will make the choice. For example, if she always chooses envelope A, then she should believe that the money is in envelope A with probability 10% and in B with probability 90%. But she can’t use that knowledge to say “oh I’ll choose B then”, because that’ll change the probabilities again.
But the Sleeping Beauty problem is not like that. Alice doesn’t make any decisions during the experiment that could feed back into her probabilities. If each day we put a sealed envelope in front of Alice, containing a note saying which day it is, then Alice really ought to have some probability distribution over what’s in the envelope. Undefined doesn’t cut it for me yet. Maybe I should just wait for your post :-)
Sure, but there’s an important economic subtlety here: to the extent that work is goal-aligned, it doesn’t need to be paid. You could do it independently, or as partners, or something. Whereas every hour worked doing the employer’s bidding, and every dollar paid for it, must be due to goals that aren’t aligned or are differently weighted (for example, because the worker cares comparatively more about feeding their family). So it makes more sense to me to view every employment relationship, to the extent it exists, as transactional: the employer wants one thing, the worker another, and they exchange labor for money. I think it’s a simpler and more grounded way to think about work, at least when you’re a worker.