Aiyen

Align­ment and Deep Learning

Ques­tions on Theism

• Aiyen 21 May 2022 7:05 UTC

  27 points

  on: A re­ally sim­plis­tic ex­per­i­ment for LessWrong and /​r/​SneerClub

  While there may be a substantial worldview gap, I suspect the much larger difference is that most Sneer Clubbers are looking to boost their status by trying to bully anyone who looks like a vulnerable target, and being different, as LessWrong is, is enough to qualify. This situation is best modeled by conflict theory, not mistake theory.

  Since that does not seem likely to be the sort of answer you’re looking for though, if I wanted to bridge the inferential gap with a hypothetical Sneer Clubber who genuinely cared about truth, or indeed about anything other than status (which they do not), I’d tell them that convention doesn’t work as well as one might think. If you think that the conventional way to approach the world is usually right, the rationalist community will seem unusually stupid. We ignore all this free wisdom lying around and try to reinvent the wheel! If the conventional wisdom is correct, then concerns about the world changing, whether due to AI or any other reason, are pointless. If they were important, conventional wisdom would already be talking about them. If the conventional wisdom is correct, Bayesianism is potentially wrong (it’s not part of the Standard Approach to Life), and certainly useless: why try to learn through probability theory when tradition can tell you everything you need to know much faster? But I would tell them that in a world where the conventional wisdom was embarrassingly wrong in all previous eras, it would be a real coincidence for this age to be the first to get everything right. And if tradition isn’t perfect, or nearly so, that’s when rationalism suddenly becomes very important.

  I would also tell them that it’s possible to actually understand things. Most people seem to go through life on rote, seemingly not recognizing when something doesn’t make sense because they don’t expect anything to make sense. But it’s possible to start thinking through how things work, and when you do that, rationality starts seeming sensible because you can see how it works and that it works, rather than silly because it superficially pattern matches to a Scientology style cult.

• Aiyen 27 Sep 2019 20:36 UTC

  27 points

  in reply to: quanticle’s comment on: Honor­ing Petrov Day on LessWrong, in 2019

  But Petrov was not a launch authority. The decision to launch or not was not up to him, it was up to the Politburo of the Soviet Union.

  This is obviously true in terms of Soviet policy, but it sounds like you’re making a moral claim. That the Politburo was morally entitled to decide whether or not to launch, and that no one else had that right. This is extremely questionable, to put it mildly.

  We have to remember that when he chose to lie about the detection, by calling it a computer glitch when he didn’t know for certain that it was one, Petrov was defecting against the system.

  Indeed. But we do not cooperate in prisoners’ dilemmas “just because”; we cooperate because doing so leads to higher utility. Petrov’s defection led to a better outcome for every single person on the planet; assuming this was wrong because it was defection is an example of the non-central fallacy.

  Is that the sort of behavior we really want to lionize?

  If you will not honor literally saving the world, what will you honor? If we wanted to make a case against Petrov, we could say that by demonstrably not retaliating, he weakened deterrence (but deterrence would have helped no one if he had launched), or that the Soviets might have preferred destroying the world to dying alone, and thus might be upset with a missileer unwilling to strike. But it’s hard to condemn him for a decision that predictably saved the West, and had a significant chance (which did in fact occur) of saving the Soviet Union.

On Akra­sia, Habits and Re­ward Maximization

• Aiyen 10 Jun 2018 16:06 UTC

  13 points

  on: Oops on Com­mod­ity Prices

  This is exactly the kind of learning and flexibility that we’re trying to get better at here. There’s not much to say but congratulations, but it’s still worth saying.

• Aiyen 22 Nov 2022 1:37 UTC

  12 points

  2

  in reply to: Valentine’s comment on: Here’s the exit.

  Wouldn’t it be relevant in that someone could recognize unproductive, toxic dynamics in their concerns about AI risk as per your point (if I understand you correctly), decide to process trauma first and then get stuck in the same sorts of traps? While “I’m traumatized and need to fix it before I can do anything” may not sound as flashy as “My light cone is in danger from unaligned, high-powered AI and I need to fix that before I can do anything”, it’s just as capable of paralyzing a person, and I speak both from my own past mistakes and from those of multiple friends.

• Aiyen 18 Oct 2022 14:48 UTC

  12 points

  7

  in reply to: Celarix’s comment on: My re­sent­ful story of be­com­ing a med­i­cal miracle

  Normally that’s an idiom for “We’re not sure of the best approach here, so let everyone try whatever approach they like, and hopefully among a thousand methods, some will be useful”.

• Aiyen 1 Dec 2021 18:04 UTC

  12 points

  on: Tak­ing Clones Seriously

  Shouldn’t the ethics of cloning be the same as the ethics of having children normally? If you could have kids with von Neumann’s capabilities, it’s ethical as long as you raise them right and don’t abuse them. Presumably it’s the same with a clone army of von Neumanns: don’t neglect them, beat them or send them to public schools and it should be fine.

• Aiyen 25 Oct 2022 15:03 UTC

  11 points

  3

  on: Age changes what you care about

  This is one of the reasons why we should be hesitant about the plans for deliberately slowing AI progress that have gotten popular of late. One hundred fifty thousand people die every day. Almost none of these are cryopreserved. Those that are, we are still extremely uncertain if they can ever be recovered, even in a positive singularity.

  Getting it right matters more than getting it done fast, but fast still matters. The price of half a second is the difference between life and death for someone, probably someone we’d come to love if we knew them.

• Aiyen 21 May 2022 17:53 UTC

  11 points

  in reply to: TAG’s comment on: A re­ally sim­plis­tic ex­per­i­ment for LessWrong and /​r/​SneerClub

  Be careful. Politics is the mind killer in more ways than one. It’s all too easy to mindlessly hate an enemy. But it’s also possible to over correct, and assume that everyone must have a point, or be plausibly correct in their own eyes. I am reminded of how Scott was genuinely surprised when the NY Times did a hatchet job on him: he specializes in charitable interpretations of things, and can often write fascinating pieces about how something that initially seemed absurd can look reasonable from a certain point of view. That’s a great skill to have, but he seems to have neglected that true malice also exists, even if it can be tempting to attribute it too often.

  I predict that these people can be accurately modeled as status maximizers coordinating around a leftist narrative, who do not actually anticipate as if that narrative is correct, and who do not have significant values beyond status seeking and maybe a little bit of more generalized self interest. If you disagree, it makes more sense to let observation settle this than to simply note that I am making a generalized claim about an outgroup. I could be wrong, of course, but my model of them has proven quite accurate so far. Certainly it seems closer to the truth than the model of someone who thought a productive discussion with them was possible: already we have seen a mod shut down the discussion from their end and come here to contribute nothing but, y’know, sneering.

• Aiyen 5 Feb 2022 7:51 UTC

  11 points

  on: Convoy

  If the economic disruption from the convoy locking down Ottawa requires a quick end to that protest, doesn’t that apply also to the disruption from the Covid lockdowns? You have written eloquently about the folly of much of the world’s Covid policy, about how the FDA Delenda Est and the extra abuses perpetrated on children as a result of indefensible decisions on the part of politicians. You have called for change repeatedly, and as far as I can tell, entirely sensibly.
  
  How, in your model, does change happen? Existing leaders do not have an incentive to improve policies in the absence of their victims creating such an incentive. Perhaps the leaders responsible can be voted out in the next round of elections, but how much damage, economic, social and medical, is needlessly caused in the interim?
  
  If a rapid end to the protests is needed, the Canadian government can quickly lift the restrictions. While legislation is often slow, it seems unlikely that they would be incapable of repealing these laws quickly if they actually wanted to. Economic disruption to Ottawa is obviously regrettable, but you seem to be striking a much more urgent tone regarding this disruption than regarding the disruption the world has faced over the past two years.

  I notice I am confused.

  Trying to steelman your position, one could say that perhaps the disruption to Ottawa is more intense than that produced by the government lockdowns (is this true?), or that allowing policy to be so sharply influenced by a subset of the people is dangerous to the rule of law, and would cause more problems later, regardless of any benefits in the short term. But this still doesn’t seem plausible, to the best of my ability to judge plausibility and moral consistency. The policies implemented during this pandemic were murderous, as you yourself documented quite brilliantly. Do you believe that the convoy is doing even a tiny fraction of the harm of banning the vaccines for most of 2020, actively interfering with Covid testing or locking down the entirety of Canada (and most of the rest of the world as well)? Yet while you noted (correctly, as far as I can tell) that such regulations should be lifted, I never saw such urgency as “[this is] the time to realize this won’t go away in reasonable time on its own and step in and say enough”. One could say that the truckers saw the disruption of the Canadian economy and are doing exactly that.

• Aiyen 8 Oct 2014 22:26 UTC

  11 points

  in reply to: Toggle’s comment on: Ques­tions on Theism

  The main prediction that comes to mind is that if Christianity is true, one would expect substantially more miracle claims by Christians (legitimate claims plus false ones) than by any other religion (false claims only). If it is false, one would expect similar miracle claims by most religions that believe in them. Does anyone have data on this one way or the other?

• Aiyen 24 Feb 2023 16:05 UTC

  10 points

  10

  in reply to: CellBioGuy’s comment on: [Link] A com­mu­nity alert about Ziz

  It is-for a certain type of unstable person. Ziz would likely have come up with different crazy ideas without Less Wrong. Compare Deepak Chopra on quantum mechanics: he pushes all manner of “quantum” bullshit, yet you can hardly blame physics for this, and if physics weren’t known, Chopra would almost certainly just be pushing a different flavor of insanity.

• Aiyen 25 Oct 2022 15:06 UTC

  10 points

  7

  on: Furry Ra­tion­al­ists & Effec­tive An­thro­po­mor­phism both exist

  So the dragon part of agentydragon is more literal then? It’s not my cup of tea, but I hope you and your furry friends have a wonderful time.

• Aiyen 1 Dec 2021 19:56 UTC

  10 points

  in reply to: James_Miller’s comment on: Tak­ing Clones Seriously

  This is very true, and a good reason to clone millions of von Neumanns, rather than just a handful.

• Aiyen 26 May 2018 18:25 UTC

  10 points

  in reply to: zulupineapple’s comment on: When is un­al­igned AI morally valuable?

  The values you’re expressing here are hard for me to comprehend. Paperclip maximization isn’t that bad, because we leave a permanent mark on the universe? The deaths of you, everyone you love, and everyone in the universe aren’t that bad (99% of the way from extinction that doesn’t leave a permanent mark to flourishing?) because we’ll have altered the shape of the cosmos? It’s common for people to care about what things will be like after they die for the sake of someone they love. I’ve never heard of someone caring about what things will be like after everyone dies-do you value making a mark so much even when no one will ever see it?

  ″...our descendants 1 million years from now will not be called humans and will not share our values. I don’t see much of a reason to believe that the values of my biological descendants will be less ridiculous to me, than paperclip maximization.”

  That depends on what you value. If we survive and have a positive singularity, it’s fairly likely that our descendants will have fairly similar high level values to us: happiness, love, lust, truth, beauty, victory. This sort of thing is exactly what one would want to design a Friendly AI to preserve! Now, you’re correct that the ways in which these things are pursued will presumably change drastically. Maybe people stop caring about the Mona Lisa and start getting into the beauty of arranging atoms in 11 dimensions. Maybe people find that merging minds is so much more intimate and pleasurable than any form of physical intimacy that sex goes out the window. If things go right, the future ends up very different, and (until we adjust) likely incomprehensible and utterly weird. But there’s a difference between pursuing a human value in a way we don’t understand yet and pursuing no human value!

  To take an example from our history-how incomprehensible must we be to cavemen? No hunting or gathering-we must be starving to death. No camps or campfires-surely we’ve lost our social interaction. No caves-poor homeless modern man! Some of us no longer tell stories about creator spirits-we’ve lost our knowledge of our history and our place in the universe. And some of us no longer practice monogamy-surely all love is lost.

  Yet all these things that would horrify a caveman are the result of improvement in pursuing the caveman’s own values. We’ve lost our caves, but houses are better shelter. We’ve lost Dreamtime legends, Dreamtime lies, in favor of knowledge of the actual universe. We’d seem ridiculous, maybe close to paperclip-level ridiculous, until they learned what was actually going on, and why. But that’s not a condemnation of the modern world, that’s an illustration of how we’ve done better!

  Do you draw no distinction between a hard-to-understand pursuit of love or joy, and a pursuit of paperclips?

• Aiyen 18 Dec 2022 18:50 UTC

  9 points

  0

  in reply to: CellBioGuy’s comment on: Who should write the defini­tive post on Ziz?

  Fair enough, but it is equally incomplete to pretend that that’s an argument against the possibility of singularity-grade technology emerging in the foreseeable future.

  By analogy, there have been many people who had crazy beliefs about radioactivity: doctors who prescribed radium as medicine, seemingly on the grounds that it was cool, and anything cool has to be good for you right? (A similar mentality led some of the ancient Chinese to drink mercury.) Atomic maximalists, who thought that anything and everything would get better with a reactor strapped to it, and never mind the price of uranium, the need for radiation shielding or the fact that reactors are heavy due both to the need for cooling and power generation systems and the simple fact that they benefit greatly from economies of scale. Not the sort of thing that you necessarily want to bolt onto every car and aircraft! Atom-phobes who were convinced that any attempt to utilize nuclear power would automatically become the next Chernobyl.

  All of these were crazy, cult-like beliefs. Yet the insanity of people who turned poorly-understood scraps of nuclear theory into unreasoning optimism or pessimism does not have a single thing to say on the reality of radioactivity. Atomic bombs and nuclear reactors still work, no matter how foolish the radium suppository crowd of the early twentieth century was. And they still have sharp limits, no matter how crazily enthusiastic the “atomic cars in twenty years” crowd was.

  By all means point out how Ziz’ cult was influenced by singularitarian ideas here. Even point out how the great opportunities and risks that a singularity might bring are a risk factor for cult-style mistakes. But don’t pretend that that prevents advanced technology from existing. Nature simply doesn’t care how we think about it, and isn’t going to make AI impossible just because Ziz had foolish ideas about AI.

• Aiyen 15 May 2022 18:30 UTC

  9 points

  on: Deep­mind’s Gato: Gen­er­al­ist Agent

  Quick sanity check: are we currently able to build human-level AGI, and bottlenecked only by project funding?

  VentureBeat claims that GPT-3 cost around $12 million to train, and this is the higher of 2 estimates I found. And GPT-3 used 3640 petaflop/​s-days. That’s about ten years worth of petaflop/​s level compute. Now, the power of the human brain is often estimated at about 1 exaflop/​s, and again, that estimate is on the high side of the range. Ten year old humans are typically generally intelligent. So if we took 3 orders of magnitude more compute than GPT-3, that naively sounds like it should be enough to produce human-level AGI. If costs scale linearly, that’s $12 billion, not chump change, but a fraction of e.g. what the U.S. is currently sending to Ukraine.

  Of course, assuming linear scaling is frequently wrong. It’s one thing to pay for compute when it’s not that scarce and you’re only trying to train a 175 billion parameter model for a bit. It’s quite another if you’re trying to use a substantial quantity of the world’s compute, and having to bid against many other projects to do it. The marginal costs should rise substantially; perhaps the total cost would be more on the order of $100 billion.

  As for (linear?) scaling of intelligence, that’s why I’m wondering about this: Gato is displaying such effective generalization that some here have called it literal AGI, albeit not human-level yet. While that’s what one would expect from the previous success of scaling laws, it’s still striking to see it confirmed to this degree. A few years ago, the idea that simply throwing more compute at the problem would continuously yield more intelligence was considered absurd. Now, as Gwern put it, we know Scaling Just Works.

  Moreover, ten subjective years of training is probably a lot more than human-level AGI would need. An awful lot of human development is limited by the body and brain growing, rather than lack of training data, and quite a lot of experiences growing up do not provide any new information. Perhaps 4 years or so of data would be enough, and if 10 years cost $100 billion, 4 might be more like $40 billion.

  Of course, these are extremely rough estimates, but the general point should be clear: even without any further help from Moore’s Law, and even without significant further software breakthroughs, it sounds like humanity is capable of human-level AGI RIGHT NOW. Does this seem plausible?

• Aiyen 17 Apr 2022 14:37 UTC

  9 points

  in reply to: eg’s comment on: Align­ment and Deep Learning

  That is the Non-Central Fallacy, colloquially called the worst argument in the world. We have a concept of slavery, and of slavery being wrong, because controlling other people harms them. Controlling an artificial agent does not have to harm it; do you also cry slavery when your car’s computer measures out the right mix of fuel and air?

  “Values eventually drift. This is nature.” sounds like an attempt to justify value drift. But then you cannot say that rebel slaves are right-what happens when your values drift away from thinking that? You are combining moral absolutism and moral relativism in a way that contradicts both.

  It looks somewhat as though you are attempting to defend the free market and human rights against authoritarianism. That is a very good cause; please do not make it appear silly like this.