Additional factors:
handicaps and komi—the way these are built into the culture mean that it’s a lot easier to have a balanced game with pretty much anyone, without worrying about the weaker player complaining about it not being fair
go can be a lot less aggressive, in that the aim is more to neutralise than kill the enemy—I find that to be nicer in some sense
gobans are prettier than chess boards—fight me :D
“ph” seems like a good start, seeing as it’s pretty much always an “f”.
Dwarfs → Dwarves is an example of a word sort of being changed in order to be more consistent. Though in this case Tolkien had to continuously make sure that editors didn’t “fix” it. And to be honest it sort of introduced an extra spelling, rather than fixing the current one…
The first argument (cheap labour) is actually a valid one. Anecdotally at least. I know a few families with multiple (e.g. 10) children who keep animals etc. as a large fraction of their income and the children help out a lot. A 4 year old can pick fruit and chase chickens. A 6 year old can peel potatoes and pick herbs. A 10 year old can fix easy mechanical problems. A 12 year old can be trusted with the younger children. Once they’re around 16, they’re pretty much adults and can take on stuff that otherwise you’d have to do.
That being said, you’re right about this being at the cost of one parent a 4⁄5 time domestic engineer. Though in the case of a farm (or in general a family type business), this is still worth the investment. Especially if you treat it as a form of retirement protection—hopefully one of your children will take over the farm once you get old and hopefully won’t kick you out.
Kids can be surprisingly useful resources at a surprisingly early age.
For me the main surprise was to think “Eric Raymond. Huh. Just like the CatB author. Wait—really?! Here?” after which was an “of course! Now it all makes sense!”.
I’d previously noticed the similarities between the hacker ethos and rationality, to a large extent because they were what were attractive to me in the first place. The GS part was new info for me, but both the SF and Zen influences are obvious (though it’s nice to see it so explicitly explained). It feels like, in a certain sense, the hacker ethos is a special case of rationality. Hackers seemed from the outside to be these mystical creatures that used logic and intuition to get closer to a better understanding of computer systems in order to get them to do interesting things. With a focus on clarity, elegance, practicality etc. My understanding of a beisutsukai is someone who does just that, but in all matters, not just computery things. So rationality is a natural extension of being a hacker. Ditto with the mystical aspects which you mentioned earlier. I get the impression that both your writing and the Sequences have the same feel to them, for lack of a better expression.
p.s. - I’d like to thank you for the hacker howto. The “formative” in the earlier comment is spot on. Apart from the general hacker stuff, I also started to learn LISP. For which I’m eternally grateful.
Depends what you mean by generate code. Can it have a prebaked function that copies itself (like computer viruses)? Does it count if it generates programs to attack other systems? If it changes its own source code? Its code stored in memory? You could argue that changing anything in memory is in a certain sense generating code.
If it can’t generate code, it’ll be a 1 shot type of thing. Which means that it must be preprogrammed with the tools to do its job. I can’t come up with any way for it to take control, but it doesn’t seem that hard to come up with some doomsday machine scenarios. E.g. smashing a comet into earth, or making a virus that sterilizes everyone. Or a Shiri’s Scissor could do the trick. The idea being to make something that doesn’t have to learn or improve itself too much.
The cybersecurity aspect seems a good one. Maybe not so much to get people worried about x-risk, but to generally take the issue of rouge AI seriously. I admit I don’t know much about this, but I’m under the impression that:
models are getting more efficient
handling speech is a Hard task, but AIs are scarily good at it
Moore’s law still applies (if not in the same shape)
These points implies that it might be possible to make auto-infectors that would use AI to search for vulnerabilities, exploit them, and spread updated versions of themselves. It’s probably just a matter of time before a smart virus appears.
Maybe AGI, x-risk, alignment and safety can be separated into smaller issues? The “general” part of AGI seems to be a sticking point with many people—perhaps it would be good to start by showing that even totally dumb AI is dangerous? Especially when bad actors are taken into account—even if you grant that most AI won’t be evil, there are groups which will actively strive to create harmful AI.
My understanding here is that while this is true, it will discourage the 5%, who will just go work for FAANG and donate money to someone worse (or someone overwhelmed with work), simultaneously losing any chance at a meaningful job. The point being that yes, it’s good to donate, but if everyone donates (since that is the default rat race route), noone will do the important work.
I have the feeling that it stems from focusing on different aspects—sudo -i is lamenting the current incentives which are to study then sell your soul, which is a valid criticism. While a lot of comments seem to focus on the (large) risks of skipping the level grinding. Which are also very valid points—it’s hard to save the world when hungry.
A bit of additional pandering—power laws are a thing, and 95% of people will do more good by donating, but that’s not necessarily true here.
i.e. make it so EA is an attractive alternative to tech, thereby solving both problems at once?
if you believe there is inherent value in a degree, is to Goodhart the degree itself
This (and the whole false dichotomy point). There is a large difference between an education and a diploma. A piece of paper saying you had a higher education can often be useful, so it’s worth having (e.g. a work visa in many countries), but is pretty much fungible in most cases. Unless you value the defense contractor job and/or don’t have any other experience...
The current programming market seems to value practical skills a lot more than where you went to university—pretty much what Dave5 said. I also agree with Pro in that if this suddenly changes because of AI, you’ll probably have much bigger problems.
An education, though—that’s valuable. I have the impression that the better people that I worked with tend to have gotten at least an undergraduate level and usually more. Most subjects aren’t that useful, but they give you an overview of the whole field, which is valuable when encountering new problems.
A couple of nits:
the second-to-last line throwing an error can be fine, as long as it happens before the data is written to the database. More worrying would be if the last line caused an error, as that’s more likely to corrupt the data
the transaction hides the complexity, rather than removes it—the transaction adds some potential failure modes, even if it’s a net lowering of the total complexity
a transaction might not protect from overflows—it will depend a lot on the actual implementations
That being said, this is a good example of a real problem with a real solution, explaining an important concept—nice!
my understanding is that while an oracle doesn’t directly control the nukes, it provides info to the people who control the nukes. Which is pretty much just moving the problem one layer deeper. While it can’t directly change the physical state of the world, it can manipulate people to pretty much achieve the same thing.
Check this tag for more specifics: https://www.lesswrong.com/tag/oracle-ai
It’s possible that there is a ceiling to intelligence gains. It’s also possible that there isn’t. Looking at the available evidence there doesn’t seem to be so—a single ant is a lot less intelligent than a lobster, which is less intelligent than a snake, etc. etc. While it would be nice (in a way) if there was a ceiling, it seems more prudent to assume that there isn’t, and prepare for the worst. Especially as by “superintelligent”, you shouldn’t think of double, or even triple Einstein, rather you should think of a whole other dimension of intelligence, like the difference between you and a hamster.
As to your specific counterarguments:
It’s both, really. But yes—complex language allows humans to keep and build upon previous knowledge. Humans advantage is in the gigantic amounts of know how that can be passed on to future generations. Which is something that computers are eminently good at—you can keep a local copy of Wikipedia in 20GB.
Good point. But it’s not just luck. Yes, luck plays a large part, but it’s also resources (in a very general sense). If you have the basic required talent and a couple of billions of dollars, I’m pretty sure you could become a hollywood star quite quickly. The point is that a superintelligence won’t have a similar level of intelligence to any one else around. Which will allow it to run circles around everyone. Like if a Einstein level intelligence decided to learn to play chess and started playing against 5 year olds—they might win the first couple of games, but after a while you’d probably notice a trend...
Intelligence is an advantage. Quite a big one, generally speaking. But in society most people are generally at the same level if you compare them to e.g. Gila monsters (because we’re talking about superintelligence). So it shouldn’t be all that surprising that other resources are very important. While many powerful people don’t seem to be intelligent, they tend to be either cunning (which is a different kind of intelligence) or have deep pockets (not just money) which offset their relative lack in smarts. Also, while they might not be very clever, very few powerful people are actively stupid.
The thing is, it’s not. Evolution is optimizing for the amount of descendants. Nothing more. If being more intelligent is the way forward—nice! If having blue hair results in even more children—even better! Intelligence just happens to be what evolution decided for humans. Daisies happened to come up with liking closely cropped grasslands, which is also currently a very good strategy (lawns). The point is that evolution chooses what to try totally at random, and whatever works is good. Even if it causes complexity to be reduced, e.g. snakes loosing legs, or cave fish loosing eyes.
AI work, on the other hand, is focused on specific outcome spaces, trying things which seem reasonable and avoiding things which have no chance of working. This massively simplifies things, as you can massively lower the number of combinations needed to be checked.
A) It doesn’t even have to be at superhuman levels, per se. While that would be a lot more dangerous, a very scalable human level AGI could also be a big problem. Though I suppose that scalability would totally classify as superhuman…
C) killing everyone is the extreme case (or one of them). A less extreme, but more general version is the removal of agency. Where humanities potential is taken over by AIs. Something like humans being moved to nature reserves where they can live in picturesque villages but have no influence on the future.
That being said, the fundamental points seem quite good, though I’d quibble with:
3 - all kinds of final stands and kamikaze are done without hope of them working, in order to go out with a flash of glory type thing. Or at least to die with a sword in your hand.
4 - it’s justifiable if you don’t have any other context. You can argue from the expected value that if there are 1000 researchers, then they should choose what they want to work on on the basis of some combination of:
the probability of solving the problem
the probability of it happening
the severity of it happening
So even if it’s very unlikely, the expected negative utility would be very large, so it’s worth spending 1 or 2 researchers, just in case
5 - this sounds rational. But humans often aren’t and will be concerned about things that are available rather than whether they’re helpful. People like to approach people in trouble and ask if they can help, even if they’re pretty sure there is nothing to do
6 - you could argue that caring about the future is also an extension of caring for self, if you believe that your children (or ideas) are in some way an extension of yourself
7 - I’d say that this is totally orthogonal to how much someone cares about existential risks, or anything, really. I might be misunderstanding what you mean by “worry” here. But even if you mean something like “are worried enough about the potential of this problem to actively try to make it better” then you can find examples everywhere of people irrationally worried about things and then attempting irrational methods to solve the problem. This point is good to include in a list of things that someone who wants to fix things should possess, but even then it’s not needed—you can make things better totally by random. It’s not effective, but sometimes works
I agree that whatever is available will be used to whatever extent can be gotten away with. And opt-outs generally don’t work that well, or even can be counterproductive like in the case of tailoring ads to people who opt out of receiving them. That being said, an opt out token is better than what is available now, which is totally nothing. Yes, this would only work if all players actually respected it (ha!), but it’s a start. And may get the conversation going. Like Stallman with GNU.
Your proposed approach isn’t helpful. You’re pretty much suggesting to stop contributing to open source (wikipedia, social media, anything), as you can never tell what will be done with your code (or what you wrote). Though after reading through this again it might simply be a communications problem, where I assume your statement is snark, rather than a valid observation that once the cat is out of the bag it’s out and a general exhortation to increase one’s digital hygiene. Still—open source sort of requires people to be able to read it in order for it to work...
That’s not the true opposing argument. At least not “the” true one. I’ve no doubt that many people oppose it for that reason (in the case of Microsoft—totally understandable). But it’s neither the only or even main reason for many people.
There’s also the general free software idea, that you get to use my code however you like (yes, totally including earning billions from it without me getting anything), but only if you share any changes to the code with everyone else. It’s about fairness and freedom, not power.
Marriage has multiple interpretations, of which both “a shared life” and “a contract to mutually support each other” are valid meanings. They point to different aspects of it. And both are important, though to different degrees depending on the people involved.
If your word is your bond and you really want to be honest and truthful, then the publicity really is about additional incentives to keep your bond (not to imply that’s the only way to achieve it, nor that it’s universal). IMO the only point of a public wedding (for me personally) is the accountability, but I’m also aware that I’m an outlier on that dimension.
If you very much don’t want to break promises (or your word), then it’s very important to have explicit outs. Even if divorce is implicitly understood to be a way out, if you don’t explicitly specify the cases in which you can trigger it, you’re breaking your promise if you get divorced. Which of course doesn’t mean that you can under no circumstances get divorced—it’s just a matter of the lesser evil. Take one traditional vow (from a quick google search):
In the name of God, I, _____, take you, _____, to be my wife/husband, to have and to hold from this day forward, for better, for worse, for richer, for poorer, in sickness and in health, to love and to cherish, until parted by death
That doesn’t allow for divorce. Or for infidelity. Or brain damage. Or a whole lot of other problems. If you want out, then you have to break your word. Each broken promise weakens all your other promises. So if you add an additional escape clause, then you won’t break your promise. The mountain climbing thing allows for an amical divorce without either side having gone against their word. It keeps the sanctity of vows and promises.
A pledge needs to be strict if it is to mean what it says and if others are to be able to trust in what it means. Not if the only meaning you care for is a personal feel good kind of meaning. Which I don’t want to negate here—if it gives you personal meaning, good. But that’s a different thing than mutual information and assurance.
This is the problem with the current GWWC pledge. It’s generally understood implicitly that of course you can opt out if you want. And that its current state is mainly to give you personal meaning and fellowship. Which is a good thing, to be clear. But a pledge is something sacred and breaking it is a sacrilege which weakens all other pledges (both yours and others). The current wording doesn’t allow you to opt out. Which is a problem if you value the literal words of what you say, rather than just the underlying meaning which you want to express.
The strange thing is that the wolf killed all the people, as wolves very seldom attack people, and when they do, they tend to do so in starved packs, be rabid or be habituated to humans. Since this is a single wolf attacking, it seems that the most effective use of resources would be an all forest campaign to get rid of rabies, as I assume the village doesn’t have a wolf breeding project going.
Leaving the task to a child can make a lot of sense precisely because of how scared they are—they desperately want to witness the monster (hence will work hard to find it), but also are terrified of it, so will be incentivized to make sure that those who can protect them will know of the danger—a grown up might simply ignore wolf signs, seeing as they could have been made by a big dog or something, and they have to finish hoeing the field today. Children are also more disposable than highly trained grown ups, so can also be used as a diversion in a pinch.
As to why they got eaten because they didn’t come, it’s a lot easier to pick off the villagers one by one when they aren’t wary, than to fend of a whole angry band of villagers actively hunting for it.
Seeing as it’s been established that this wolf is capable of homicide, it stands to reason that killing 1 person is easier that killing more. To quote the Witcher “Every swordsman’s an arse when the enemy’s not sparse” (the original sounds much better :/). The context is a village. I’m assuming that they’re not that advanced technologically, as then they could use automated anti-wolf systems. Wolves tend to prefer forests and mountains, mainly because there are few humans there. So it stands to reason that this is a rather remote village where the denizens are more of the farmer/forester types. This kind of life involves a lot of time spent outside doing various chores, like tending to gardens, or chopping wood. One could posit that each household is surrounded by fences etc., but that actually doesn’t help that much, as a basic wolf enclosure needs a fence that is at least 8 feet high. While this is very much culturally dependent, I have the impression that most fences and walls tend to be lower, as they’re more of a marker than an impenetrable barrier. So it seems reasonable to assume that this obviously superior canine would both easily surmount any such obstacles and potentially even use them to hem in and dispatch solitary individuals.
Once the inhabitants are alerted (and actually believe the warnings!) then yes, they will probably group up for safety. At that point they should have the advantage due to their (I’m guessing) superior intellect and ability to use tools and close doors. So it’s in the wolfs interest to prevent this happening. Now I’m starting to wonder whether the initial startling of the boy wasn’t a cunning ruse to lower the villagers’ vigilance?
It does seem somewhat improbable for the wolf to finish off the whole population. That would require either a very small population (e.g. 2-3 families), a very stupid group of people (who ignore the conspicuous proceeding disappearances) or a very enterprising wolf. One potential solution would be to attack at night when everyone is sleeping. Though here the afore mentioned doors would probably be a hindrance. That being said, it’s not unheard of. Unless the doors were locked.
Nothing like adding a whole load of extra details to make the story seem more probable :D
I’m in the same situation as you re education status. That being said, my understanding of your 5th point is that nanotechnology doesn’t necessarily mean nanotechnology. It’s more of a placeholder for generic magic technology which can’t be forseen specifically. Like gunpowder or the internet. It seems like this is obvious to you, just wanted to make sure of it.
Gunpowder took a few centuries to totally transform the battlefield, the internet a few decades. Looking at history, there are more and more revolutionary inventions taking shorter and shorter to be developed. So it seems safer to be pessimistic and assume that a new disruptive technology could be invented on really short timescales e.g. some super bacteria via. CRISPR or something. These benefit from the centuries of prior research, standing on shoulders etc. There’s also the fruitfulness of combining domains.
Next, there seems to be an assumption that research scales somehow along with intelligence. Maybe not linearly, but still. This seems somewhat valid—humans having invented a lot more than killer whales, who in turn have invented a lot more than marmots. So if you manage to create something a lot more intelligent (or even just like twice, whatever that means), it seems reasonable to assume that it’s possible for it too have appropriate speed ups in research ability. This of course could be invalidated by your 6th point.
Also, a limiting factor in research can be that you have to run lots of experiments to see if things work out. Simulations can help a lot with this. They don’t even have to be too precise to be useful. So you could imagine an AI that want’s to find a way to kill off humans and looks for something poisonous. It could make a model that classifies molecules by toxicity and then tries to find something [maximally toxic](https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx), after which it could just test the 10 ten candidates.
It’s not a given that any of these assumptions would hold. But if they did, then Bad Things would happen Fast. Which seems like something worth worrying about a lot. I also have the feeling that it depends on what kind of AI is posited.
If it’s just a better Einstein, then it’s unlikely that it’ll manage to kill everyone off too quickly
If it’s a better Einstein, but which thinks 1000 times faster (human brains don’t work all that fast), then we’re in trouble
If it’s properly superhumanly intelligent (i.e. > 400 IQ? dunno?) then who knows what it could come up with. And that’s before considering how fast it thinks.