As of June 2024, I have signed no contracts or agreements whose existence I cannot mention.
gilch
Karma: 2,583
I don’t know why we think we can colonize Mars when we can’t even colonize Alaska. Alaska at least has oxygen. Where are the domed cities with climate control?
Specifically, while the kugelblitz is a prediction of general relativity, quantum pair production from strong electric fields makes it infeasible in practice. Even quasars wouldn’t be bright enough, and those are far beyond the energy level of a single Dyson sphere. This doesn’t rule out primordial black holes forming at the time of the Big Bang, however.
It might still be possible to create micro black holes with particle accelerators, but how easy this is depends on some unanswered questions about physics. In theory, such an accelerator might need to be a thousand light years across at most, but this depends on achievable magnetic field strength. (Magnetars?) On the other hand, if compactified extra dimensions exist (like in string theory), the minimum required energy would be lower. One that small would evaporate almost instantly though. It’s not clear if it could be kept alive long enough to get any bigger.
What are efficient Dyson spheres probably made of?
There are many possible Dyson sphere designs, but they seem to fall into three broad categories: shells, orbital swarms, and bubbles. Solid shells are probably unrealistic. Known materials aren’t strong enough. Orbital swarms are more realistic but suffer from some problems with self-occlusion and possibly collisions between modules. Limitations on available materials might still make this the best option, at least at first.
But efficient Dyson spheres are probably bubbles. Rather than being made of satellites, they’re made of statites, that is, solar sails that don’t orbit, but hover. Since both gravitational acceleration and radiant intensity follow the inverse square law, the same design would function at almost any altitude above the Sun, with some caveats. These could be packed much more closely together than the satellites of orbital swarms while maybe using less material. Eric Drexler proposed 100 nm thick aluminum films with some amount of supporting tensile structure. Something like that could be held open by spinning, even with no material compressive structure. Think about a dancer’s dress being held open while pirouetting and you get the idea.
The radiation needs to be mostly reflected downwards for the sails to hover, but it could still be focused on targets as long as the net forces keep the statites in place. Clever designs could probably approach 100% coverage.
What percent of the solar system can be converted into Dyson-sphere material? Are gas giants harvestable?
Eventually, almost all of it, but you don’t need to to get full coverage. Yes, they’re harvestable; at the energy scales we’re talking about, even stellar material is harvestable via star lifting. The Sun contains over 99% of the mass of the Solar System.
How long would it take to harvest that material?
I don’t know, but I’ll go with the 31 years and 85 days for an orbital swarm as a reasonable ballpark. Bubbles are a different design and may take even less material, but either way, we’re talking about exponential growth in energy output that can be applied to the construction. At some point the energy matters more than the matter.
What would the radius of a Dyson sphere be? (i.e. how far away from the sun is optimal). How thick?
I’d say as close to the Sun as the materials can withstand (because this takes less material), so probably well within the orbit of Mercury. Too much radiation and the modules would burn up. Station keeping becomes more difficult when you have to deal with variable Solar wind and coronal mass ejections, and these problems are more severe closer in.
The individual statite sails would be very thin. Maybe on the order of 100 nm for the material, although the tensile supports could be much thicker. I don’t know how many sails an optimal statite module would use (maybe just 1). But the configuration required for focus and station keeping probably isn’t perfectly flat, so a minimal bounding box around a module could be much thicker still.
An energy efficient Dyson Sphere probably looks like a Matrioshka brain, with outer layers collecting waste heat from the inner layers. Layers could be much farther apart than the size of individual modules.
If the sphere is (presumably) lots of small modules, how far apart are they?
Statites could theoretically be almost touching, especially with active station keeping, which is probably necessary anyway. What’s going to move them? Solar wind variation? Micrometor collisions? Gravitational interactions with other celestial bodies? Remember, statites work about the same regardless of altitude, so there can be layers with some amount of overlap.
“if an AI is moderately ‘nice’, leaves Earth alone but does end up converting the rest of the solar system into a Dyson sphere, how fucked is Earth?
Very, probably. And we wouldn’t have to wait for the whole (non-Sun) Solar System to be converted before we’re in serious trouble.
Rob Miles’ YouTube channel has some good explanations about why alignment is hard.
We can already do RLHF, the alignment technique that made ChatGPT and derivatives well-behaved enough to be useful, but we don’t expect this to scale to superintelligence. It adjusts the weights based on human feedback, but this can’t work once the humans are unable to judge actions (or plans) that are too complex.
If we don’t mind the process being slow, this could be achieved by a single “crawler” machine that would go through the matrix field by field and do the updates. Since the machine is finite (albeit huge), this would work.
No following. We can already update the weights. That’s training, tuning, RLHF, etc. How does that help?
We have a goal A, that we want to achieve and some behavior B, that we want to avoid.
No. We’re talking about aligning general intelligence. We need to avoid all the dangerous behaviors, not just a single example we can think of, or even numerous examples. We need the AI to output things we haven’t thought of, or why is it useful at all? If there’s a finite and reasonably small number of inputs/outputs we want, there’s a simpler solution: that’s not an AGI—it’s a lookup table.
You can think of the LLM weights as a lossy compression of the corpus it was trained on. If you can predict text better than chance, you don’t need as much capacity to store it, so an LLM could be a component in a lossless text compressor as well. But these predictors generated by the training process generalize beyond their corpus to things that haven’t been written yet. It has an internal model of possible worlds that could have generated the corpus. That’s intelligence.
Get a VPN. It’s good practice when using public Wi-Fi anyway. (Best practice is to never use public Wi-Fi. Get a data plan. Tello is reasonably priced.) Web filters are always imperfect, and I mostly object to them on principle. They’ll block too little or too much, or more often a mix of both, but it’s a common problem in e.g. schools. Are you sure you’re not accessing the Wi-Fi of the business next door? Maybe B&N’s was down.
Takeover, if misaligned, also counts as doom. X-risk includes permanent disempowerment, not just literal extinction. That’s according to Bostrom, who coined the term:
One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.
A reasonably good outcome might be for ASI to set some guardrails to prevent death and disasters (like other black marbles) and then mostly leave us alone.
My understanding is that Neuralink is a bet on “cyborgism”. It doesn’t look like it will make it in time. Cyborgs won’t be able to keep up with pure machine intelligence once it begins to take off, but maybe smarter humans would have a better chance of figuring out alignment before it starts. Even purely biological intelligence enhancement (e.g., embryo selection) might help, but that might not be any faster.
Aschenbrenner also wrote https://situational-awareness.ai/. Zvi wrote a review.
There’s some UC San Francisco research to back up this view. California has the nation’s biggest homeless population mainly due to unaffordable housing, not migration from elsewhere for a nicer climate.
I notice that https://metaphor.systems (mentioned here earlier) now redirects to Exa. Have you compared it to Phind (or Bing/Windows Copilot)?
Relatively healthy people do occasionally become homeless due to misfortune, but they usually don’t stay homeless. It could be someone from the lower class living paycheck to paycheck who has a surprise expense they’re not insured for and can’t make rent. It could be a battered woman and her children escaping domestic abuse. They get services, they get back on their feet, they get housed. Ideally, the social safety nets would work faster and better than they do in practice, but the system basically works out for them.
The persistently homeless are a different story. They’re often mentally ill and/or addicted to drugs. They don’t have the will or capacity to take advantage of the available services. In the past, such people were often institutionalized. These institutions often didn’t act in their best interests, but it at least kept them off the streets so they couldn’t harass the rest of us. Your proposal probably wouldn’t help for them.
Why do we need more public bathrooms? I’m skeptical because if there was demand for more bathrooms, then I’d expect the market to produce them.
I wouldn’t expect so, why would you think that? Markets have a problem handling unpriced externalities without regulation. (Tragedy of the commons.) Pollution is a notable example of market failure, and the bathrooms issue is a special case of exactly this. Why pay extra to dispose of your waste properly if you can get away with dumping it elsewhere? As a matter of public health, it’s better for everyone if this type of waste goes in the sewers and not in the alley, even if the perpetrators can’t afford a coffee. How would you propose we stop the pollution? Fining them wouldn’t help, even if we could catch them reliably, which would be expensive, because they don’t have any money to take. Jailing them would probably cost taxpayers more than maintaining bathrooms would. Taxpayers are already paying for the sewer system (a highly appropriate use of taxation). This is just an expansion of the same.
The table isn’t legible with LessWrong’s dark theme.
I feel like these would be more effective if standardized, dated and updated. Should we also mention gag orders? Something like this?
As of June 2024, I have signed no contracts or agreements whose existence I cannot mention.
As of June 2024, I am not under any kind of gag order whose existence I cannot mention.
Last updated June 2024. I commit to updating at least annually.Could LessWrong itself be compelled even if the user cannot? Should we include PGP signatures or something?
I thought it was mostly due to the high prevalence of autism (and the social anxiety that usually comes with it) in the community. The more socially agentic rationalists are trying.
But probably he should be better at communication e.g. realizing that people will react negatively to raising the possibility of nuking datacenters without lots of contextualizing.
Yeah, pretty sure Eliezer never recommended nuking datacenters. I don’t know who you heard it from, but this distortion is slanderous and needs to stop. I can’t control what everybody says elsewhere, but it shouldn’t be acceptable on LessWrong, of all places.
He did talk about enforcing a global treaty backed by the threat of force (because all law is ultimately backed by violence, don’t pretend otherwise). He did mention that destroying “rogue” datacenters (conventionally, by “airstrike”) to enforce said treaty had to be on the table, even if the target datacenter is located in a nuclear power who might retaliate (possibly risking a nuclear exchange), because risking unfriendly AI is worse.
The argument chain you presented (Deep Learning → Consciousness → AI Armageddon) is a strawman. If you sincerely think that’s our position, you haven’t read enough. Read more, and you’ll be better received. If you don’t think that, stop being unfair about what we said, and you’ll be better received.
Last I checked, most of us were agnostic on the AI Consciousness question. If you think that’s a key point to our Doom arguments, you haven’t understood us; that step isn’t necessarily required; it’s not a link in the chain of argument. Maybe AI can be dangerous, even existentially so, without “having qualia”. But neither are we confident that AI necessarily won’t be conscious. We’re not sure how it works in humans but seems to be an emergent property of brains, so why not artificial brains as well? We don’t understand how the inscrutable matrices work either, so it seems like a possibility. Maybe gradient descent and evolution stumbled upon similar machinery for similar reasons. AI consciousness is mostly beside the point. Where it does come up is usually not in the AI Doom arguments, but questions about what we ethically owe AIs, as moral patients.
Deep Learning is also not required for AI Doom. Doom is a disjunctive claim; there are multiple paths for getting there. The likely-looking path at this point would go through the frontier LLM paradigm, but that isn’t required for Doom. (However, it probably is required for most short timelines.)
You are not wrong to complain. That’s feedback. But this feels too vague to be actionable.
First, we may agree on more than you think. Yes, groupthink can be a problem, and gets worse over time, if not actively countered. True scientists are heretics.
But if the science symposium allows the janitor to interrupt the speakers and take all day pontificating about his crackpot perpetual motion machine, it’s also of little value. It gets worse if we then allow the conspiracy theorists to feed off of each other. Experts need a protected space to converse, or we’re stuck at the lowest common denominator (incoherent yelling, eventually). We unapologetically do not want trolls to feel welcome here.
Can you accept that the other extreme is bad? I’m not trying to motte-and-bailey you, but moderation is hard. The virtue lies between the extremes, but not always exactly in the center.
What I want from LessWrong is high epistemic standards. That’s compatible with opposing viewpoints, but only when they try to meet our standards, not when they’re making obvious mistakes in reasoning. Some of our highest-karma posts have been opposing views!
Do you have concrete examples? In each of those cases, are you confident it’s because of the opposing view, or could it be their low standards?
the problem “How do we stop people from building dangerous AIs?” was “research how to build AIs”.
Not quite. It was to research how to build friendly AIs. We haven’t succeeded yet. What research progress we have made points to the problem being harder than initially thought, and capabilities turned out to be easier than most of us expected as well.
Methods normal people would consider to stop people from building dangerous AIs, like asking governments to make it illegal to build dangerous AIs, were considered gauche.
Considered by whom? Rationalists? The public? The public would not have been so supportive before ChatGPT, because most everybody didn’t expect general AI so soon, if they thought about the topic at all. It wasn’t an option at the time. Talking about this at all was weird, or at least niche, certainly not something one could reasonably expect politicians to care about. That has changed, but only recently.
I don’t particularly disagree with your prescription in the short term, just your history. That said, politics isn’t exactly our strong suit.
But even if we get a pause, this only buys us some time. In the long(er) term, I think either the Singularity or some kind of existential catastrophe is inevitable. Those are the attractor states. Our current economic growth isn’t sustainable without technological progress to go with it. Without that, we’re looking at civilizational collapse. But with that, we’re looking at ever widening blast radii for accidents or misuse of more and more powerful technology. Either we get smarter about managing our collective problems, or they will eventually kill us. Friendly AI looked like the way to do that. If we solve that one problem, even without world cooperation, it solves all the others for us. It’s probably not the only way, but it’s not clear the alternatives are any easier. What would you suggest?
I can think of three alternatives.
First, the most mundane (but perhaps most difficult), would be an adequate world government. This would be an institution that could easily solve climate change, ban nuclear weapons (and wars in general), etc. Even modern stable democracies are mostly not competent enough. Autocracies are an obstacle, and some of them have nukes. We are not on track to get this any time soon, and much of the world is not on board with it, but I think progress in the area of good governance and institution building is worthwhile. Charter cities are among the things I see discussed here.
Second might be intelligence enhancement through brain-computer interfaces. Neuralink exists, but it’s early days. So far, it’s relatively low bandwidth. Probably enough to restore some sight to the blind and some action to the paralyzed, but not enough to make us any smarter. It might take AI assistance to get to that point any time soon, but current AIs are not able, and future ones will be even more of a risk. This would certainly be of interest to us.
Third would be intelligence enhancement through biotech/eugenics. I think this looks like encouraging the smartest to reproduce more rather than the misguided and inhumane attempts of the past to remove the deplorables from the gene pool. Biotech can speed this up with genetic screening and embryo selection. This seems like the approach most likely to actually work (short of actually solving alignment), but this would still take a generation or two at best. I don’t think we can sustain a pause that long. Any anti-AI enforcement regime would have too many holes to work indefinitely, and civilization is still in danger for the other reasons. Biological enhancement is also something I see discussed on LessWrong.
What other courses would you consider equivalent?