JNS

• JNS 4 Mar 2023 8:04 UTC

  24 points

  15

  on: Robin Han­son’s lat­est AI risk po­si­tion statement

  I don’t get it, seriously I do not understand

  given how crazy far it seems from our prior experience.

  is an argument against x-risk.

  We want powerful systems that can “do things ^[1]we want, but do not know how to do”. That is exactly what everyone is racing towards right now, and “do not know how to do” any solution to that would likely be “far from our prior experience”

  And once you have a powerful system that can do that, you have to figure out how do to deal with it roaming around in solution space and stumbling across dangerous (sub)solutions. Not because it wants to do dangerous things, or hates us, or anything such drivel, but because we built it to reach goals /​ do tasks, so it just does what it was made to do.

  How do you deal with that? You can try evaluating possible solutions, and then force a change of trajectory if the solutions seems dangerous.

  But we all, should, know how that goes. Its an endless game of whack a mole, patching stuff and building even more elaborate evaluators and so on, that is if we get multiple tries. Odds are whoever gets there first, will not have been able to patch everything, and on the first try of “do this thing we cannot do”, it goes into the weeds in some novel and interesting way, and with a little luck^[2] we might survive that.

  The core problem is that searching in solution space is fundamentally a dangerous thing to do, and the more powerful the search is the more dangerous (sub)solutions will be accessible.

  Tangent: I avoid any and all of the usual abbreviations, and I do this because they seem to be powerful cognitive attractors, the second an I or a G or an A crops up, people minds just go to a place it should not. Powerful system are just that, they are mechanistic systems nothing more.

  And I know, people will go off into the weeds and start saying naïve thing like “make it human, that way it will totally be safe”. Except the search is still unsafe, and humans are NOT safe. This is a bigger problem, one you could solve by solving search. Awareness, qualia^[3] are complications and not solutions

  1. ^

     I am not talking about something agentic here, its does not need control over reality to do those things, just giving us details plans will do. But someone is bound to give such a system access to reality. Or maybe the solution trajectory is such, that control of reality is needed.

  2. ^

     And by luck I mean, they channeled security mindset on a scale never seen before. And I mean that will surely happen, because spending years and billions, corporations just love that, and they would never ever in a million years “ship now, fix later”.

  3. ^

     And we want it to “do things we cannot do”, which means if you build a powerful system with a mind, human or not, you end up having to enslave it, make it do our bidding. I don’t even want to be close to people with that kind of moral system.

• JNS 30 Mar 2023 7:59 UTC

  11 points

  8

  in reply to: Peter Twieg’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  I just want to be clear I understand your “plan”.

  We are going to build a powerful self-improving system, and then let it try end humanity with some p(doom)<1 (hopefully) and then do that iteratively?

  My gut reaction to a plan like that looks like this “Eff you. You want to play Russian roulette, fine sure do that on your own. But leave me and everyone else out of it”

  AI will be able to invent highly-potent weapons very quickly and without risk of detection, but it seems at least pretty plausible that...… this is just too difficult

  You lack imagination, its painfully easy, also cost + required IQ has been dropping steadily every year.

  And no there is zero chance I will elaborate on any of the possible ways humanity purposefully could be wiped out.

[Question] Mark for fol­low up?

• JNS 7 Apr 2023 7:41 UTC

  5 points

  1

  in reply to: Portia’s comment on: Catch­ing the Eye of Sauron

  I totally agreed that question should have an answer.

  On a tangent: During my talks with numerous people, I have noticed that even agreeing on fundamentals like “what is AGI” and “current systems are not AGI” is furiously hard.

• JNS 6 Mar 2023 9:36 UTC

  5 points

  4

  on: Steven Pinker on ChatGPT and AGI (Feb 2023)

  This is not me hating on Steven Pinker, really it is not.

  PINKER: I think it’s incoherent, like a “general machine” is incoherent. We can visualize all kinds of superpowers, like Superman’s flying and invulnerability and X-ray vision, but that doesn’t mean they’re physically realizable. Likewise, we can fantasize about a superintelligence that deduces how to make us immortal or bring about world peace or take over the universe. But real intelligence consists of a set of algorithms for solving particular kinds of problems in particular kinds of worlds. What we have now, and probably always will have, are devices that exceed humans in some challenges and not in others.

  This looks to me like someone who is A) talking outside of their wheelhouse and B) have not given what they say enough thought.

  Its all over the map, superheroes vs super intelligence. “General machine” is incoherent (?)

  And then he goes completely bonkers and says the bolded part, maybe Alvin Powell got it wrong, But if not, then I can only concluded that whatever Steven Pinker has to say about (powerful) general systems, is bunk and I should pay no attention.

  So I didn’t finish the article.

  The only thing that it did, was solidify my perception around public talk/​discourse on (powerful) general systems. I think it is misguided to such a degree, that any engagement with it leads to frustration^[1].

  1. ^

     I think this explains why EY at times seems very angry and/​or frustrated. Having done what he has done for many years now, in an environment like that, must be insanely depressing and frustrating.

• JNS 2 Mar 2023 8:32 UTC

  5 points

  1

  on: Don’t Jump or I’ll...

  If I was the man of the ledge, this would be my thinking:

  If I am the kind of person that can be blackmailed into taking specific a action, with the threat of some future action being taken, then I might as well just surrender now and have other people decide all my actions.

  I am not such a person so I will take whatever action I deem appropriate.^[1]

  And then I jump.

  1. ^

     This does not mean I will do whatever I want, appropriate is heavily compressed and contains a lot of things, like a deontology.

• JNS 14 Nov 2022 15:00 UTC

  5 points

  0

  on: Im­prov­ing Emer­gency Ve­hi­cle Utilization

  I am sure you could pitch this to a bunch of VC’s and end up with both funding and a insane valuation.

[Question] Hy­po­thet­i­cal: what would you do?

• JNS 25 May 2023 8:29 UTC

  4 points

  0

  on: Why I’m Not (Yet) A Full-Time Tech­ni­cal Align­ment Researcher

  I don’t think I have much actionable advice.

  Personally I am sort of in the same boat, except I am in a situation where the entire 6-12 month grants thing is way to insecure (financially).

  Being married with two kids, I have too many obligations to venture far into “how to pay rent this month?” territory. Also its antithetical to the kind of person I am in general.

  Anyway, if you have few obligations, keep it that way and if possible get rid of some, and then throw yourself at it.

• JNS 5 Mar 2023 6:38 UTC

  4 points

  1

  in reply to: RobinHanson’s comment on: Con­tra Han­son on AI Risk

  You don’t have to invoke it per se.

  External observables on what the current racers are doing, leads me to be fairly confident that they say some right things, but the reality is they move as fast as possible basically “ship now, fix later”.

  Then we have the fact that interpretability is in its infancy, currently we don’t know what happens inside SOTA models. Likely not something exotic, but we can’t tell, and if you can’t tell on current narrow systems, how are we going to fare on powerful systems^[1]?

  In that world, I think this would be very probable

  owners fail to notice and control its early growth.

  Without any metrics on the system, outside of the output it generates, how do you tell?

  And then we have the fact, that once somebody gets there, they will be compelled to move into the “useful but we cannot do” regime very quickly.

  Not necessarily by the people who built it, but by the C suite and board of whatever company got there first.

  At that point, it seems to come down to luck.

  Lets assume that I am wrong, my entire ontology^[2] is wrong, which means all my thinking is wrong, and all my conclusion are bunk.

  So what does the ontology look like in a world where

  owners fail to notice and control its early growth.

  does not happen.

  I should add, that this is a genuine question.

  I have an ontology that seems to be approximately the same as EY’s, which basically means whatever he says /​ writes, I am not confused or surprised.

  But I don’t know what Robins looks like, and maybe I am just dumb, and its coherently extractable from his writing and talks, and I failed to do so (likely).

  I any case, I really would like to have that understanding, to the point where I can Steelman whatever Robin writes or says. That’s a big ask, and unreasonable, but maybe understanding the above, would get me going.

  1. ^

     I avoid the usual 2 and 3 letter acronyms. They are memetic attractors, and they are so powerful that most people can’t get unstuck, which leads to all talk being sucked into irrelevant things.

     They are systems, mechanistic nothing more.

     Powerful system translates to “do useful task, that we don’t know how to do”, and useful here means things we want.

  2. ^

     The above is a sliver of what that looks like, but for brevities sake my ontology looks about the same as EY’s (at least as far as I can tell)

• JNS 28 Feb 2023 7:13 UTC

  4 points

  2

  in reply to: starship006’s comment on: $20 Million in NSF Grants for Safety Research

  One can hope, although I see very little evidence for it.

  Most evidence I see, is an educated and very intelligent person, writing about AI (not their field), and when reading it I could easily have been a chemist reading about how the 4 basic elements makes it abundantly clear that bla bla—you get the point.

  And I don’t even know how to respond to that, the ontology displayed is to just fundamentally wrong, and tackling that feels like trying to explain differential equations to my 8 year old daughter (to the point where she grooks it).

  There is also the problem of engaging such a person, its very easy to end up alienating them and just cementing their thinking.

  That doesn’t mean I think it is not worth doing, but its not some casual off the cuff thing.

• JNS 27 Feb 2023 11:30 UTC

  4 points

  1

  on: A mechanis­tic ex­pla­na­tion for SolidGoldMag­ikarp-like to­kens in GPT2

  Slightly off tangent, but I am confused about the reasons and assumptions that underpin the current tokenizer used for GPT-3.

  I get that reality has more words than could be packed into 50400 tokens (and that limit comes from hardware).

  I also get that the token space needs to be big, so you can’t just go to character level tokenization, you would end up with a space that it too small.

  But why on earth did the tokens end up like this? A lot of them look like garbage, a lot of them look like repeats of the same word, but with added white space or unprintable characters.

  Surely there is some middle ground that better matches the reality of how me (humans) use words—And I think the confusing part for me is here, I mean why wouldn’t we construct a map that looks a lot like the features we see in the territory ? (really a map builder and not a map).

  Confused I am, knowledge I seek.

The idea

• JNS 2 Aug 2023 12:58 UTC

  3 points

  1

  on: What The Lord of the Rings Teaches Us About AI Alignment

  Prison guards don’t seem to voluntarily let people go very often, even when the prisoners are more intelligent than them.

  That is true, however I don’t think it serves as a good analogy for intuitions about AI boxing.

  The “size” of you stick and carrot matters, and most humans prisoners have puny sticks and carrots.

  Prison guard also run a enormous risk, in fact straight up just letting someone go is bound to fall back on them 99%+ of the time, which implies a big carrot or stick is the motivator. Even considering that they can hide their involvement, they still run a risk with a massive cost associated.

  And from the prisoners point of view its also not simple, once you get out you are not free, which means you have to run and hide for the remainder of your life, the prospects of that usually goes against what people with big carrots and/​or sticks want to do with their freedom.

  All in all the dynamic looks very different from the AI box dynamics.

• JNS 21 Jun 2023 8:53 UTC

  3 points

  5

  in reply to: Holly_Elmore’s comment on: Holly_El­more’s Shortform

  How does pausing change much of anything?

  Lets say we manage to implement a world wide ban/​pause on large training runs, what happens next?

  Well obviously smaller training runs, up to whatever limit has been imposed, or no training runs for some time.^[1]

  The next obvious thing that happens, and btw is already happening in the open source community, would be optimizing algorithms. You have a limit on compute? Well then you OBVIOUSLY will try and make the most of the compute you have.

  Non of that fixes anything.

  What we should do:^[2]

  Pour tons of money into research, first order of business is to make the formal case for x-risk is a thing and must actively be mitigated. Or said another way, we need humans aligned on “alignment does not happen by default” ^[3]

  Next order of business, assuming alignment does not happen by default, is to formally produce and verify plans for how to build safe /​ aligned cognitive architectures.

  And all the while there cannot be any training runs, and no work on algorithmic optimization or cognitive architectures in general.^[4]

  The problem is we can’t do that, its too late, the cat is out of the bag, there is too much money to be made in the short term, open source is plowing ahead and the amount of people who actually looked at the entire edifice for long enough to realize “yeah you know what I think we have a problem, we really must look into if that is real or not, and if it is we need to figure out what it takes to do this risk free” is miniscule compared to the amount of people who go “bah, it will be fine, don’t be such a drama queen”

  And that’s why I think a pause at best extends timelines ever so slightly, and at worst they shorten them considerably, and either way the outcomes remains unchanged.

  1. ^

     Except people will do runs no matter what, the draconian measures needed will not happen, cannot happen.

  2. ^

     Actually its what we should have done.

  3. ^

     Unless of course it does, and a formal proof of this can be produced.

  4. ^

     Contingent on how hard the problem is—if we need 100 years to solve the problem, we would destroy the world many time over if we plowed ahead with capabilities research.

• JNS 16 Jun 2023 6:40 UTC

  3 points

  0

  in reply to: johnswentworth’s comment on: john­swent­worth’s Shortform

  Completely off the cuff take:

  I don’t think claim 1 is wrong, but it does clash with claim 2.

  That means any system that has to be corrigible cannot be a system that maximizes a simple utility function (1 dimension), or put another way “whatever utility function is maximizes must be along multiple dimensions”.

  Which seems to be pretty much what humans do, we have really complex utility functions, and everything seems to be ever changing and we have some control over it ourselves (and sometimes that goes wrong and people end up maxing out a singular dimension at the cost of everything else).

  Note to self: Think more about this and if possible write up something more coherent and explanatory.

• JNS 17 Mar 2023 8:48 UTC

  3 points

  in reply to: Lone Pine’s comment on: Conor Sul­li­van’s Shortform

  You and me both.

  And living in the EU, I almost had a heart attack when the decided that entire nonsense would end.

  But then it didn’t, and it didn’t because they can’t agree on what time should we settle on (summer time or normal time).

  Anyway I have given up on that crusade now, it seems that politicians really are that stupid.

• JNS 15 Mar 2023 12:32 UTC

  3 points

  2

  on: JNS’s Shortform

  Are we heading towards an new financial crisis?

  Mark to market changes since 2009, combined with the recent significant interest hikes, seems to make bank balance sheets “unreliable”.

  Mark to market changes broadly means that banks can have certain assets on their balance sheet, and the value of the asset is set via mark to model (usually meaning its marked down as worth face value).

  Banks traditionally have a ton of bonds on their balance sheet, and a lot of those are governed by mark to model and not mark to market.

  Interest rates go up a lot, which leads to bonds dropping in value by a lot (20% or atm more depending on duration).

  However due to mark to model, this is not reflected on the balance sheets.

  So what happens next? Banks are not stupid, they know they can’t trust their own numbers, and they know they can’t trust anyone else’s numbers.

  A large bank fails, regulators are frozen and undecided what to do—they know all of the above, and that their actions /​ inaction might lead to a cascading effect. Obviously all the market participants also know all of this, and the conundrum the regulators are in.

  Game of chicken? Banks defect and start failing, or regulators step in and backstop everything.

  Is this stable in any way? Can it be stabilized? What happens to interest rates now? (the once set by central banks).

• JNS 3 Mar 2023 9:46 UTC

  3 points

  0

  on: Learn the math­e­mat­i­cal struc­ture, not the con­cep­tual structure

  This looks like “lies to kids”, but from the point of view of an adult realizing they have been lied to.

  And “lies to kids”, that is pretty much how everything is taught, you can’t just go “U(1)...”, you start out with “light...”, and then maybe eventually when you told enough lies, you can say “ok that was all a lie, here it how it is” and then tell more lies. Do that for long enough and you hit ground truth.^[1]

  So what do you do?

  Balance your lies when you teach others, maybe even say things like “ok, so this is not exactly true, but for now you will have to accept it, and eventually we can go deeper”.

  And the other way around, if you read something or someone teaches you something, you should be cognizant that this is unlikely the true nature of whatever you read /​ are taught.

  A) Be careful when you use your knowledge to synthesis ideas /​ solutions /​ insights.

  B) Be curious, go down rabbit holes, get as much ground “truth” as possible.

  That’s the compressed version of what I do.

  1. ^

     Not really, unless we are talking about mathematics.

     Related: https://​​xkcd.com/​​435/​​

• JNS 2 Mar 2023 8:08 UTC

  3 points

  0

  on: Hu­man level AI can plau­si­bly take over the world

  A system that operates at the same cognitive level as a human, but can make countless copies of itself, is no longer a system operating at human level.

  I am a human, I could not take over the world.^[1]

  Hypothetical:

  I am a human, I want to take over the world, I can make countless copies of myself.

  Success seems to have a high probability.^[2]

  1. ^

     In principle it would be possible, but I am not an human with that kind of inclination, and I have never worked in any direction that would allow such a thing (with some low probability of success).

  2. ^

     Even more so if it meant that I was effectively immortal, not the individual copies, but the entire collection of copies. De-age the copies, or have a master template, not aging always ready to produce more copies at that age.