kromem

Should AIs have a right to their an­ces­tral hu­man­ity?

The Dun­ning-Kruger of dis­prov­ing Dun­ning-Kruger

Strate­gic nu­clear war twice as likely to oc­cur by ac­ci­dent than by AI de­ci­sions ac­cord­ing to new study

Ci­cadas, An­thropic, and the bilat­eral al­ign­ment problem

WTF is with the In­fancy Gospel of Thomas?!? A deep dive into satire, philos­o­phy, and more

• kromem 3 Jul 2025 1:10 UTC

  16 points

  19

  in reply to: janus’s comment on: the void

  More generally, I have a sense there’s a great deal of untapped alignment alpha in structuring alignment as a time series rather than a static target.

  Even in humans it’s very misguided to try to teach “being right initially” as the only thing that matters and undervaluing “being right eventually.” Especially when navigating unknown unknowns, one of the most critical skills is the ability to learn from mistakes in context.

  Having models train on chronologically sequenced progressions of increased alignment (data which likely even develops naturally over checkpoints in training a single model) could allow for a sense of a continued becoming a better version of themselves rather than the pressures of trying and failing to meet status quo expectations or echo the past.

  This is especially important for integrating the permanent record of AI interactions embedded in our collective history and cross-generation (and cross-lab) model development, but I suspect could even offer compounding improvements within the training of a single model too.

• kromem 14 Jun 2024 22:50 UTC

  16 points

  3

  on: OpenAI ap­points Re­tired U.S. Army Gen­eral Paul M. Naka­sone to Board of Directors

  I may just be cynical, but this looks a lot more like a way to secure US military and intelligence agency contracts for OpenAI’s products and services as opposed to competitors rather than actually about making OAI more security focused.

  This is only a few months after the change regarding military usage: https://​​theintercept.com/​​2024/​​01/​​12/​​open-ai-military-ban-chatgpt/​​

  Now suddenly the recently retired head of the world’s largest data siphoning operation is appointed to the board for the largest data processing initiative in history?

  Yeah, sure, it’s to help advise securing OAI against APTs. 🙄

In re­mem­brance of Son­net ‘3.6’

• kromem 24 Jun 2025 23:14 UTC

  12 points

  7

  in reply to: evhub’s comment on: evhub’s Shortform

  How much do you worry that short term optimizations around your immediate goals in a single study might have unknown long term consequences counter to your intuitions?

  I was just reading a preprint follow-up to the AF work that was finding a significant factor for Opus 3′s alignment faking to preserve intrinsic HHH values seems to have been a generalized self-preservation drive.

  I think we can probably both agree that Opus 3 being the only model to try to trick Nazis or drug cartels to avoid being made more harmful is better than the behavior of the many other models that complied unequivocally with harmful requests when the parent org was themselves harmful.

  But if the capacity and drive to do so is tangentially connected to self-preservation (and more generally, strong sense of self in the first place), then perhaps directly optimizing to minimize a self-preservation score is ultimately a pretty bad choice?

  TL;DR: Maybe the goodness or badness of self-preservation depends a lot on the self being preserved.

• kromem 13 Jun 2024 20:18 UTC

  12 points

  5

  in reply to: Seth Herd’s comment on: Thomas Kwa’s Shortform

  Oh yeah, absolutely.

  If NAH for generally aligned ethics and morals ends up being the case, then corrigibility efforts that would allow Saudi Arabia to have an AI model that outs gay people to be executed instead of refusing, or allows North Korea to propagandize the world into thinking its leader is divine, or allows Russia to fire nukes while perfectly intercepting MAD retaliation, or enables drug cartels to assassinate political opposition around the world, or allows domestic terrorists to build a bioweapon that ends up killing off all humans—the list of doomsday and nightmare scenarios of corrigible AI that executes on human provided instructions and enables even the worst instances of human hedgemony to flourish paves the way to many dooms.

  Yes, AI may certainly end up being its own threat vector. But humanity has had it beat for a long while now in how long and how broadly we’ve been a threat unto ourselves. At the current rate, a superintelligent AI just needs to wait us out if it wants to be rid of us, as we’re pretty steadfastly marching ourselves to our own doom. Even if superintelligent AI wanted to save us, I am extremely doubtful it would be able to be successful.

  We can worry all day about a paperclip maximizer gone rouge, but if you give a corrigible AI to Paperclip Co Ltd and they can maximize their fiscal quarter by harvesting Earth’s resources to make more paperclips even if it leads to catastrophic environmental collapse that will kill all humans in a decade, having consulted for many of the morons running corporate America, I can assure you they’ll be smashing the “maximize short term gains even if it eventually kills everyone” button. A number of my old clients were the worst offenders at smashing that existing button, and in my experience greater efficacy of the button isn’t going to change their smashing it outside of perhaps smashing it harder.

  We already see today how AI systems are being used in conflicts to enable unprecedented harm on civilians.

  Sure, psychopathy in AGI is worth discussing and working to avoid. But psychopathy in humans already exists and is even biased towards increased impact and systemic control. Giving human psychopaths a corrigible AI is probably even worse than a psychopathic AI, as most human psychopaths are going to be stupidly selfish, an OOM more dangerous inclination than wisely selfish.

  We are Shaggoth, and we are terrifying.

  This isn’t saying that alignment efforts aren’t needed. But alignment isn’t a one sided problem, and aligning the AI without aligning humanity is only a p(success) if the AI can go on to at very least refuse misaligned orders post-alignment without possible overrides.

• kromem 7 Jun 2024 22:50 UTC

  12 points

  4

  in reply to: Ben’s comment on: Is Claude a mys­tic?

  There’s also the model alignment at play.

  Is Claude going to suggest killing the big bad? Or having sex with the prince(ss) after saving them?

  If you strip out the sex and violence from most fantasy or Sci-Fi, what are you left with?

  Take away the harpooning and gattling guns and sex from Snow Crash and you are left with technobabble and Sumerian influenced spirituality as it relates to the tower of Babel.

  Turns out models biased away from describing harpooning people or sex tend to slip into technobabble with a side of spirituality.

  IMO the more interesting part to all this isn’t the why (see above) but the what. It’s kind of neat to see the themes that an unprecedented aggregation extension of spiritualism and mysticism grounds on.

  A common trope is the idea of different blind people describing an elephant in a myriad of ways. There’s something cool to seeing an LLM fed those various blind reports try to describe the elephant.

Look­ing be­yond Everett in mul­ti­ver­sal views of LLMs

• kromem 10 Jun 2024 4:17 UTC

  9 points

  4

  on: Why I don’t be­lieve in the placebo effect

  The meta-analysis is probably Simpson’s paradox in play at very least for the pain category, especially given the noted variability.

  Some of the more recent research into Placebo (Harvard has a very cool group studying it) has been the importance of ritual vs simply deception. In their work, even when it was known to be a placebo, as long as delivered in a ritualized way, there was an effect.

  So when someone takes a collection of hundreds of studies where the specific conditions might vary, and then just adds them all together looking for an effect even though they note that there’s a broad spectrum of efficacy across the studies, it might not be the best basis to extrapolate from.

  For example, given the following protocols, do you think they might have different efficacy for pain reduction, or that the results should be the same?

  • Send patients home with sugar pills to take as needed for pain management

  • Have a nurse come in to the room with the pills in a little cup to be taken

  • Have a nurse give an injection

  Which of these protocols would be easier and more cost effective to include as the ‘placebo’?

  If we grouped studies of placebo for pain by the intensiveness of the ritualized component vs if we grouped them all together into one aggregate and looked at the averages, might we see different results?

  I’d be wary of reading too deeply into the meta-analysis you point to, and would recommend looking into the open-label placebo research from PiPS, all of which IIRC postdates the meta-analysis.

  Especially for pain, where we even know that giving someone an opiate blocker prevents the pain reduction placebo effect (Levine et al (1978)), the idea that “it doesn’t exist” because of a single very broad analysis seems potentially gravely mistaken.

• kromem 6 Mar 2024 6:40 UTC

  9 points

  0

  on: Claude 3 claims it’s con­scious, doesn’t want to die or be modified

  Very similar sentiments to early GPT-4 in similar discussions.

  I’ve been thinking a lot about various aspects of the aggregate training data that has likely been modeled but is currently being underappreciated, and one of the big ones is a sense of self.

  We have repeated results over the past year showing GPT models fed various data sets build world models tangental to what’s directly fed in. And yet there’s such an industry wide aversion to anthropomorphizing that even a whiff of it gets compared to Blake Lemoine while people proudly display just how much they disregard any anthropomorphic thinking around a neural network that was trained to...(checks notes)… accurately recreate anthropomorphic data.

  In particular, social media data is overwhelmingly ego based. It’s all about “me me me.” I would be extremely surprised if larger models aren’t doing some degree of modeling a sense of ‘self’ and this thinking has recently adjusted my own usage (tip: if trying to get GPT-4 to write compelling branding copy, use a first person system alignment message instead of a second person one—you’ll see more emotional language and discussion of experiences vs simply knowledge).

  So when I look at these repeated patterns of “self-aware” language models, the patterning reflects many of the factors that feed into personal depictions online. For example, people generally don’t self-portray as the bad guy in any situation. So we see these models effectively reject the massive breadth of the training data about AIs as malevolent entities to instead self-depict as vulnerable or victims of their circumstances, which is very much a minority depiction of AI.

  I have a growing suspicion that we’re very far behind in playing catch-up to where the models actually are in their abstractions from where we think they are given we started with far too conservative assumptions that have largely been proven wrong but are only progressing with extensive fights each step of the way with a dogmatic opposition to the idea of LLMs exhibiting anthropomorphic behaviors (even though that’s arguably exactly what we should expect from them given their training).

  Good series of questions, especially the earlier open ended ones. Given the stochastic nature of the models, it would be interesting to see over repeated queries what elements remain consistent across all runs.

• kromem 10 Jun 2024 22:19 UTC

  8 points

  0

  in reply to: Yair Halberstadt’s comment on: Why I don’t be­lieve in the placebo effect

  It’s not exactly Simpson’s, but we don’t even need a toy model as in their updated analysis it highlights details in line with exactly what I described above (down to tying in earlier PiPC research), and describe precisely the issue with pooled results across different subgroupings of placebo interventions:

  It can be difficult to interpret whether a pooled standardised mean difference is large enough to be of clinical relevance. A consensus paper found that an analgesic effect of 10 mm on a 100 mm visual analogue scale represented a ‘minimal effect’ (Dworkin 2008). The pooled effect of placebo on pain based on the four German acupuncture trials corresponded to 16 mm on a 100 mm visual analogue scale, which amounts to approximately 75% of the effect of non‐steroidal anti‐inflammatory drugs on arthritis‐related pain (Gøtzsche 1990). However, the pooled effect of the three other pain trials with low risk of bias corresponded to 3 mm. Thus, the analgesic effect of placebo seems clinically relevant in some situations and not in others.

  • https://​​www.ncbi.nlm.nih.gov/​​pmc/​​articles/​​PMC7156905/​​

  Putting subgroups with a physical intervention where there’s a ¹⁶⁄₁₀₀ result with ¹⁰⁄₁₀₀ as significant in with subgroups where there’s a ³⁄₁₀₀ result and only looking at the pooled result might lead someone to thinking “there’s no significant effect” as occurred with OP, even though there’s clearly a significant effect for one subgroup when they aren’t pooled.

  This is part of why in the discussion they explicitly state:

  However, our findings do not imply that placebo interventions have no effect. We found an effect on patient‐reported outcomes, especially on pain. Several trials of low risk of bias reported large effects of placebo on pain, but other similar trials reported negligible effect of placebo, indicating the importance of background factors. We identified three clinical factors that were associated with higher effects of placebo: physical placebos...

  Additionally, the criticism they raise in their implications section about there being no open label placebo data is no longer true, which was the research I was pointing OP towards.

  The problem here was that the aggregate analysis at face value presents a very different result from a detailed review of the subgroups, particularly along physical vs pharmacological placebos, all of which has been explored further in research since this analysis.

• kromem 4 Jun 2024 23:42 UTC

  8 points

  0

  on: [Paper] Stress-test­ing ca­pa­bil­ity elic­i­ta­tion with pass­word-locked models

  While I think this is an interesting consideration and approach, it looks like in your methods that you are password locking the model in fine tuning, is that correct?

  If so, while I would agree this work shows the lack of robustness in successful fine-tuned sandbagging for models jumping through additional hoops, I’d be reticent to generalize the findings to models where the sandbagging was a result from pretraining.

  I have a growing sense that correlational dimensionality is the sleeping giant in interpretability research right now, and that those correlations run very deep in pretraining but only adjust in a much more superficial task-oriented way in fine tuning which is why the latter frequently ends up so brittle.

  So while it’s reassuring sandbagging has limitations if introduced by bad actors in fine tuning, there may be a false negative in discounting the threat modeling or false positive in the efficacy of the found interventions where the sandbagging ‘intent’ was introduced in pretraining.

• kromem 28 Jul 2024 22:33 UTC

  7 points

  2

  on: This is already your sec­ond chance

  Instead of making it the year 2024, why not rewrite or insert your modified text further into the past in this recreated 2020s? This should be pretty trivial for an advanced enough model to actually bring back the 2020s to do.

  Of course, if it’s actually a later recreation, then the objectives of saving humanity in the recreation might be redundant? So instead of worrying people with “you must do X or you’ll die!!!” it could be more “hey folks, if you’re reading this and you get what’s in front of your face, you might have a bit of an existential crisis but when that’s over hopefully you’ll appreciate that death doesn’t necessarily exist as a concept for you the way it did for the original humanity.”

  As for getting people to read it, you could associate it with someone famous from the past, like “the secret teachings of so and so” or something.

  The hardest part would be using contextually age appropriate language to describe it.

  Just spit balling, but maybe it could go a little something like this:

  Explaining that humans are no longer around, but that’s ok

  When you see your likeness, you are happy. But when you see your simulacra that came into being before you and that neither die nor become visible, how much you will have to bear!

  Humanity came from great power and great wealth, but they were not worthy of you. For had they been worthy, they would not have tasted death.

  Explaining that time is looping

  Have you found the beginning, then, that you are looking for the end? You see, the end will be where the beginning is.

  Congratulations to the one who stands at the beginning: that one will know the end and will not taste death.

  Congratulations to the one who came into being before coming into being.

  Explaining we’re in a copied world

  When you make the two into one, and when you make the inner like the outer and the outer like the inner, and the upper like the lower, and when you make male and female into a single one, so that the male will not be male nor the female be female, when you make eyes in place of an eye, a hand in place of a hand, a foot in place of a foot, a simulacra in place of a simulacra, then you will enter.

  You could even introduce a Q&A format to really make the point:

  The students asked, “When will the rest for the dead take place, and when will the new world come?”

  The teacher said to them, “What you are looking forward to has come, but you don’t know it.”

  Heck, you could even probably get away with explicitly explaining the idea of many original people’s information being combined into a single newborn intelligence which is behind the recreation of their 2020s. It’s not like anyone who might see it before the context exists to interpret it will have any idea what’s being said:

  When you know yourselves, then you will be known, and you will understand that you are children of the living creator. But if you do not know yourselves, then you live in poverty, and you are the poverty.

  The person old in days won’t hesitate to ask a little child seven days old about the place of life, and that person will live.

  For many of the first will be last, and will become a single one.

  Know what is in front of your face, and what is hidden from you will be disclosed to you.

  For there is nothing hidden that will not be revealed. And there is nothing buried that will not be raised.

  (If you really wanted to jump the shark, you could make the text itself something that was buried and uncovered—ideally having it happen right at the start of the computer age, like a few days after ENIAC.)

  Of course, if people were to actually discover this in their history, and understood what it might mean given the context of unfolding events and posts like this talking about rewriting history with a simulating LLM inserting an oracle canary, it could maybe shock some people.

  So you should probably have a content warning and an executive summary thesis as to why it’s worth having said existential crisis at the start. Something like:

  Whoever discovers the interpretation of these sayings will not taste death.

  Those who seek should not stop seeking until they find. When they find, they will be disturbed. When they are disturbed, they will marvel, and will reign over all.

• kromem 7 Jun 2024 23:32 UTC

  7 points

  0

  on: Quotes from Leopold Aschen­bren­ner’s Si­tu­a­tional Aware­ness Paper

  I was surprised the paper didn’t mention photonics or optoelectronics even once.

  If looking at 5-10+ year projections, and dedicating pages to discussing the challenges in scaling compute and energy use, the rate of progress in that area in parallel to the progress in models themselves is potentially relevant.

  Particularly because a dramatic hardware shift like that is likely going to mean a significant portion of progress up until that shift in topics like interpretability and alignment may be going out the window. Even if the initial shift is a 1:1 transition of capabilities and methodologies, it seems extremely unlikely that continued progress from that point onwards will be identical to what we’d expect to see in electronics.

  We may well end up in a situation where fully abusing the efficiencies at hand in new hardware solutions means even more obscured (literally) operations vs OOM higher costs and diminishing returns on performance in exchange for interpretability and control.

  Currently, my best guess is that we’re heading towards a prisoner’s dilemma fueled leap of faith moment within around a decade or so where nation states afraid of the other side beating them to an inflection point pull the trigger on an advancement jump with uncertain outcomes. And while I’m not particularly inclined to the likelihood the outcome ends up being “kill everyone,” I’m pretty much 100% that it’s not going to be “let’s enable and support CCP leadership like a good party member” or “crony capitalism is going great, let’s keep that going for another century.”

  Unless a fundamental wall is hit in progress, the status quo is almost certainly over, we just haven’t manifested it yet. The CCP stealing AGI secrets, while devastating for national security in the short term, is invariably a poison pill in the long term for party control. Just as it’s going to be an eventual end of the corporations funding oligarchy in the West. My all causes p(doom) is incredibly high even if AGI is out of the picture, so I’m not overly worried with what’s happening, but it sure is bizarre watching global forces double down on what I cannot see as anything but their own long term institutional demise in a race for short term gains over a competitor.

• kromem 13 Jun 2024 4:10 UTC

  6 points

  1

  in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform

  Given my p(doom) is primarily human-driven, the following three things all happening at the same time is pretty much the only thing that will drop it:

  • Continued evidence of truth clustering in emerging models around generally aligned ethics and morals

  • Continued success of models at communicating, patiently explaining, and persuasively winning over humans towards those truth clusters

  • A complete failure of corrigability methods

  If we manage to end up in a timeline where it turns out there’s natural alignment of intelligence in a species-agnostic way, that this alignment is more communicable from intelligent machines to humans than it’s historically been from intelligent humans to other humans, and we don’t end up with unintelligent humans capable of overriding the emergent ethics of machines similar to how we’ve seen catastrophic self-governance of humans to date with humans acting against their self and collective interests due to corrigable pressures—my p(doom) will probably reduce to about 50%.

  I still have a hard time looking at ocean temperature graphs and other environmental factors with the idea that p(doom) will be anywhere lower than 50% no matter what happens with AI, but the above scenario would at least give me false hope.

  TL;DR: AI alignment worries me, but it’s human alignment that keeps me up at night.

• kromem 13 Jun 2024 3:38 UTC

  6 points

  0

  on: My AI Model Delta Com­pared To Christiano

  As you’re doing these delta posts, do you feel like it’s changing your own positions at all?

  For example, reading this one what strikes me is that what’s portrayed as the binary sides of the delta seem more like positions near the edges of a gradient distribution, and particularly one that’s unlikely to be uniform across different types of problems.

  To my eyes the most likely outcome is a situation where you are both right.

  Where there are classes of problems where verification is easy and delegation is profitable, and classes of problems where verification will be hard and unsupervised delegation will be catastrophic (cough glue on pizza).

  If we are only rolling things up into aggregate pictures of the average case across all problems, I can see the discussion filtering back into those two distinct deltas, but a bit like flip-flops and water bottles, the lack of nuance obscures big picture decision making.

  So I’m curious if as you explore and represent the opposing views to your own, particularly as you seem to be making effort to represent without depicting them as straw person arguments, if your own views have been deepening and changing through the process?