Cedar

[Question] [Align­ment] Is there a cen­sus on who’s work­ing on what?

• Cedar 4 Apr 2022 20:02 UTC

  12 points

  in reply to: Joseph Miller’s comment on: Google’s new 540 billion pa­ram­e­ter lan­guage model

  It basically said “thanks I hate it” in response to that joke

Burn­ing Up­time: When your Sand­box of Em­pa­thy is Leaky and also an Hourglass

• Cedar 7 Apr 2022 4:36 UTC

  10 points

  on: What I Was Think­ing About Be­fore Alignment

  Every single piece has been tweaked toward the same end goal

  I feel like there’ll be a better way to say this sentence once we figure out the answer to your first question,

  How can we recognize adaptive systems in the wild? What universal behaviors indicate an adaptive optimizer?

  It most definitely seems to make sense to say that systems can have goals, in a “if it looks like a duck it makes sense to call it a duck” kind of way. But at the same time, every single piece hasn’t been tweaked to the same end goal as the system. They are each tweaked towards their own survival, and that’s somewhat aligned with the system’s survival.

  Something I wish there are more lesswrong posts for (or at least I wish I’ve seen more lesswrong posts for) is posts exploring alignment in the context of :

  1. Organisms and their smaller replicator components (organisms < cells, cells < transposons& endoviruses & organelles)

  2. Social thingies and their smaller sorta-replicator components (religions < religious ideas, companies < replicating Management ideas)

  If you have your favorite post that falls into the above genre or mentions something to that effect, please absolutely link me to it! I’d love to read more.

  I’m only halfway through the A-Z sequence, so I’d also very much appreciate it if you could point to things in there to get me excited about progressing through it!

[Question] How “grifty” is the Fore­sight In­sti­tute? Are they mak­ing but­ton soup?

• Cedar 20 Nov 2021 3:11 UTC

  7 points

  on: How To Get Into In­de­pen­dent Re­search On Align­ment/​Agency

  Ty John for writing this up. This post and the comments really helps me find my own place and direction in terms of doing what I want to do.

  Currently in academia and VERY unhappy about the bullshit I have to ingest and create. But I’m still waiting on my social, political, and financial safety nets before I can do anything remotely brave, kindda like tryactions mentioned in his comment.

  So the most I’ll do is probably just read and write and talk to people on the side.

  Speaking of talking to people...

  My current research involves (manually) using CPU architectural artifacts to break sandboxing and steal data. I’ve been wondering whether I could do something on the lines of “make a simple AI that tries to break out of sandboxes, then make an unbreakable sandbox to contain it”.

  Do shoot me a message if you have any thoughts, or is just curious. I would love to chat.

Use the Nato Alphabet

• Cedar 10 Jan 2023 2:23 UTC

  6 points

  1

  on: Notes on writing

  Don’t think about big words or small words—think about which particular word you mean.

  I think this is good advice for most people who are used to being forced to use big words by school systems, but I personally follow a different version of this.

  I see compression as an important feature of communication, and there are always tradeoffs to be made between

  1. Making my sentence short

  2. Conveying exactly what I want to say.

  And sometimes I settle with transferring a “good enough” version of my idea, because communicating all the hairy details takes too much time /​ energy /​ social credit. I’m always scared of taking too much of people’s attention or overrunning their working memory.

• Cedar 28 Mar 2022 19:10 UTC

  6 points

  on: Search­ing for outliers

  About footnote #5 on arranged vs love marriages...

  While the distribution for compatibility when finding a partner is long tailed, if the distribution for marginal gain on investing into an existing relationship is also as long tailed, then the comparative advantage sorta cancels out and you’re not as much better off looking harder vs developing the relationships you already have.

  This sorta fits my experience too, where being vulnerable and putting in the work is occasionally given me large improvements in relationship satisfaction that I never expected based on the median improvement I get for each unit of work.

• Cedar 2 May 2022 18:54 UTC

  5 points

  on: Nar­ra­tive Syncing

  I kinda like the term “cheerleading” instead of narrative syncing. Kinda like “I define a cheer and y’all follow me and do the cheer”.

  Shameless plug here: I’m trying to get into alignment but struggling to get motivated emotionally. If any of you wants to do mutual cheerleading over a discord chat or something, please PM me. also PM me if you just want to hang out and chat and figure out whether you want information or cheerleading. I’d be glad to help with that too by rubber duckie-ing with you.

  I’m doing this because I mistook my need for cheerleading as a need for information a while ago and had a very confusing 1-hour chat with a rationalist where he kept trying to give me information and I kept trying to look for inclusion/​acceptance signals. I learned a lot both by listening to him and by reflecting upon that experience but I fear I’ve ended up wasting his time and I kinda feel sad about that. This is why I’m putting my email here.

• Cedar 29 Apr 2022 15:43 UTC

  5 points

  in reply to: Holly_Elmore’s comment on: Virtue sig­nal­ing is some­times the best or the only met­ric we have

  Just to make sure we’re on the same page, we seem to be both agreeing that

  1. Good virtual signaling is possible and should be attempted

  2. Moloch worship is possible, should be avoided, and may be the reason for why many people hate /​ avoid /​ devaluate virtual signaling.

  And you seem to be saying, yes Moloch is real, yes things can go very bad. But we should still be trying to build good norms and standards to accurately signal virtues.

  Does my summary feel fair and correct?

• Cedar 16 Apr 2022 23:05 UTC

  5 points

  on: Clem’s Memo

  This is truly epic.

  I don’t know where this conditioned response comes from, but I tear up a little when I read things like this.

  I’m scared that when the next threat to our civilization comes, the masses are so enmeshed in bullshit and the leaders so focused on producing it, that nobody rises to take action like this.

• Cedar 28 Mar 2022 13:55 UTC

  5 points

  on: Xida Ren’s Shortform

  Informational Topology and Causal Networks

  I recently found InfoTopo https://​​sites.google.com/​​view/​​pierrebaudot/​​menu/​​infotopo-software

  And it was so weird and exciting I immediately started being scared of losing it, and promptly archived it with the waybackmachine. https://​​web.archive.org/​​web/​​20220328135016/​​https://​​sites.google.com/​​view/​​pierrebaudot/​​menu/​​infotopo-software

  But what this package does is it lets you do cool and weird things visualizing the informational relationships between different variables, discrete or complete.

  I’m a bit overwhelmed in my life currently, but I would love it if somebody could take a look at it and comment, and maybe draw some connections with Judea Pearl’s works. I feel like I’m on the threshold of something great and awesome but I’m kindda stuck trying to understand this and not knowing where my attention should start.

• Cedar 14 May 2022 12:56 UTC

  4 points

  on: The Last Paperclip

  : (

  That’s not how the story went in my mind.

  It felt obvious to me that once the probes are starting to make each other into paperclips, some sort of natural selection would take over where probes that prioritize survival over paperclip making would murder the rest and survive. And there’d be a new cycle of life.

• Cedar 19 Apr 2022 17:29 UTC

  4 points

  in reply to: jefftk’s comment on: Only Ask­ing Real Questions

  I agree with you in terms of the fair and respectful systems grounding. My own experience with mazey-ness has been that they cause me intense anxiety and distress, and I’d imagine having that earlier in childhood being a very bad thing.

  In terms of mazey organizations being common… I feel like it really depends on where you are and where you’re from. In the social-economic section of China where I come from, for example, asking fake questions, sacrificing your own standards to fit in, manipulation, and “measure effort by who self-flagellates the hardest” are so common that they’re just the assumed backdrop for every conversation about career stuff and academic stuff. And I think it has a certain momentum on you that persists even when you leave. For example, I find myself having a tendency to be attracted by vaguely shiny and prestigious things, and that’s accounted for me landing in a sell-side quant position (VERY mazey) and a PhD program (somewhat mazey).

  But yes all in all I agree that if surviving without bullshit is at all possible, developing a strong bullshit allergy is an awesome thing to do for your kids.

• Cedar 17 Apr 2022 1:58 UTC

  4 points

  on: Only Ask­ing Real Questions

  My parents really cared about making things fair and giving expectations of fairness, and I really enjoyed it. It contributed to many of my qualities I care about (commitment to objective reality, attempt at consistency, integration of different parts of the self).

  But sometimes I wonder if the opposite would be better, especially for surviving in maze-ey environment a la moral mazes.

• Cedar 29 Dec 2022 17:49 UTC

  3 points

  0

  on: Let’s think about slow­ing down AI

  • AI is pretty safe: unaligned AGI has a mere 7% chance of causing doom, plus a further 7% chance of causing short term lock-in of something mediocre

  • Your opponent risks bad lock-in: If there’s a ‘lock-in’ of something mediocre, your opponent has a 5% chance of locking in something actively terrible, whereas you’ll always pick good mediocre lock-in world (and mediocre lock-ins are either 5% as good as utopia, −5% as good)

  • Your opponent risks messing up utopia: In the event of aligned AGI, you will reliably achieve the best outcome, whereas your opponent has a 5% chance of ending up in a ‘mediocre bad’ scenario then too.

  • Safety investment obliterates your chance of getting to AGI first: moving from no safety at all to full safety means you go from a 50% chance of being first to a 0% chance

  • Your opponent is racing: Your opponent is investing everything in capabilities and nothing in safety

  • Safety work helps others at a steep discount: your safety work contributes 50% to the other player’s safety

  This is more a personal note /​ call for somebody to examine my thinking processes, but I’ve been thinking really hard about putting hardware security methods to work. Specifically, spreading knowledge far and wide about how to:

  1. allow hardware designers /​ manufacturers to have easy, total control over who uses their product for what for how much throughout the supply chain

  2. make it easy to secure AI related data (including e.g. model weights and architecture) and difficult to steal.

  This sounds like it would improve every aspect of the racey-environment conditions, except:

  Your opponent is racing: Your opponent is investing everything in capabilities and nothing in safety

  The exact effect of this is unclear. On the one hand, if racey, zero-sum thinking actors learn that you’re trying to “restrict” or “control” AI hardware supply, they’ll totally amp up their efforts. On the other hand, you’ve also given them one more thing to worry about (their hardware supply).

  I would love to get some frames on how to think about this.

• Cedar 10 Aug 2022 3:09 UTC

  3 points

  on: Su­per­vise Pro­cess, not Outcomes

  I’m new to alignment and I’m pretty clueless.

  What’s Ought’s take on the “stop publishing all capabilities research” stance that e.g. Yudkowsky is taking in this tweet? https://​​twitter.com/​​ESYudkowsky/​​status/​​1557184416786423809

• Cedar 18 Apr 2022 15:31 UTC

  3 points

  in reply to: RHollerith’s comment on: Code Gen­er­a­tion as an AI risk setting

  Yes please!

• Cedar 9 Apr 2022 15:02 UTC

  3 points

  on: Utility Max­i­miza­tion = De­scrip­tion Length Minimization

  Related reading linking mutual information to best possible classifier:

  https://​​arxiv.org/​​pdf/​​1801.04062.pdf

  This one talks about estimating KL divergence and Mutual Information using neural networks, but I’m specifically linking it to show y’all Theorem 1,

  Theorem 1 (Donsker-Varadhan representation). The KL divergence admits the following dual representation:

  $D_{KL}\left(P||Q\right)=su{p}_{T:Q-R}{E}_P\left[T\right]-log\left(E_Q\right[E_T\left]\right)$

  This links the mutual information to the best possible regression. But I haven’t figured out exactly how to parse /​ interpret this.