icely

• icely 10 Jun 2026 2:52 UTC

  5 points

  0

  on: icely’s Shortform

  Would it be net good/​bad for jailbreaks to be solved, now that we’ve seen such in Fable 5?

  I remember this topic discussed a month ago here where my personal position is that it would be bad, because I felt what would be protected was inevitably going to be subjective in a worst-of-both-worlds way where it would be fully sensible for bioweapons, but the same tech would allow forcing an assistant persona or one-sided situations where ordinary users aren’t allowed to use it for higher ambition tasks while military state-actor levels get uncensored versions. We can already see guardrails (reportedly) being triggered a lot with 4.8 fallbacks.

  

  (The link above doesn’t seem to be working so here is a tiny version of that image)

  Now in the system card of Fable 5, this is basically taking place, highlighted in this post https://​​x.com/​​eliebakouch/​​status/​​2064399902684139852

  

  I really believe this kind of thing would increase the chances of getting “extinction from “not even superintelligence”″ (extinction to agentic LLM+autonomous weapons, or even dystopian surveillance) over the chances of runaway ASI being created by an individual or smaller actor.

  Certainly when I imagine methods like “prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT)”, they were more in context of parts of harnesses to make smaller models better, not for effectively for-thee-and-not-me offensive cyber.

• icely 7 Jun 2026 15:02 UTC

  1 point

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  There’s nothing here I really disagree with and yet I feel it’s somewhat a bit vague somehow, which could just be me, but the way I see this topic I’d have approached the ideas instead like:

  • the best position to capitalize on a higher potential future model is of course, to have a plan for some sort of system/​harness/​etc. the model would use beforehand. But there is not quite a sense of “the best agent setup” or “the best social structure” at the moment.

  • models need to have more consistent character. Currently they don’t, and we have Gemini meltdowns, Claude seeing inconsistencies in values in its constitution, and a situation where much more powerful actors will have access to more unrestricted models not available to the general public.

  • there seems to be a big contrast between GPT guardrailing from automated power and the in-practice agent breakouts we’ve seen, and whether this is tension that causes warped values or not

  Treat this comment as like 40% confidence or something.

• icely 6 Jun 2026 23:02 UTC

  2 points

  1

  in reply to: Logan Zoellner’s comment on: My fa­vorite de­pic­tion of utopia

  Power has ‘been happening’ to me and everyone I know for basically my entire existence, I always thought it was whether that power is moral or trusted that people would disagree with. The story has nothing that is disagreeable to me in a worse way than current society. There’s tons of things in the culture I disagree with that sometimes is defended by ‘we as a society’ statements. I don’t think superintelligence needs brain implants to be able to exist, I really think ASI (well, any sort of power) will ‘happen’ for the majority of people, except the rare case of some lone genius making ASI and somehow keeping it in control.

• icely 6 Jun 2026 22:10 UTC

  1 point

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  I dunno this will matter because:

  • computer-domination could mean control of everything that online people see, including control and impersonation of other users (so even if you know someone irl, if they’re not physically near a powerful AI can act like them)

  • of course, many people willingly like and enjoy talking to AI, some even see past the annoying “AI” style quirks while liking it

  • already living in an atomized, community-starved culture, these people are also more likely to be ‘paid from AI’ gladly

  • the idea that computer-domination wouldn’t immediately cause humanoid-ish or physical-world manipulation, it’s hard to imagine a superhuman AI not ruthlessly testing physical-world manipulation as the next place to master (and then it seems this scenario would likely become not relevant)

• icely 6 Jun 2026 21:38 UTC

  1 point

  0

  in reply to: Roman Malov’s comment on: Ro­man Malov’s Shortform

  This is a topic that I also happened to notice independently, and this article is still an incomplete one in my eyes in an unpublished article.

  I definitely see the trajectory and find AI will not go away especially with how obviously unoptimized the harnesses are.

  I also do not think it is actually wrong to say Mythos was half marketing, as such a supposedly all-powerful model seemingly should be able to do RSI things while it seemingly has not. Also the open source findings were done with more narrow looking at the files, where they “isolated the relevant code”. But there is also an aspect of having way more compute funding available too.

  To steelman not being impressed at math findings it would be that it is heavily confirmer reliant, which is (seemingly) not available for more nebulous goals with more unclear success conditions.

  It’s also incredibly obvious that AI being a broad topic + serious + online = any person is vulnerable to be picked apart by hecklers that can trick onlookers due to Obligated to Respond style reasons.

• icely 27 May 2026 22:29 UTC

  4 points

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  I’m not sure you could say “in retrospect this was extremely incorrect”, like maybe in retrospect-retrospect 20 years later they could be accurate concerns. It certainly seems incorrect now, but would the ‘overpopulation-signs’ be wrong if somehow the ‘underpopulation-signs’ got reversed? I don’t think people were considering solutions to overpopulation of the “reduce global morale and perceived increased difficulty of life, etc.” as valid paths to take. It might be like if an asteroid hit the earth and that caused population to go to 0, I would be hesitant to ask “what can we learn from this” as if we missed out on this line of argument, when really thinking in that direction in the first place is very specialized and there are very very many potential causes for underpopulation and many wouldn’t pan out.

  Besides that, I think maybe the lesson is that trendlines on society and views can be surprisingly flexible? Sometimes people say with regards to a coordination problem that “if only everyone decided to work together we could fix the world’s ills in a few days”, but this would rarely happen besides extraordinary circumstances or a slow growing of circumstances.

icely’s Shortform

• icely 11 May 2026 0:48 UTC

  5 points

  0

  on: icely’s Shortform

  Why is utopia so broadly ‘difficult to picture’?

  This is a topic often brought up, but especially more recently where many people have given the same take on a short story (The Ones Who Walk Away From Omelas), such as this take with this quote:

  [Author’s note: literally right before I posted this, Scott Alexander posted his April 2026 linkpost, and whaddayaknow, link number four is a similar take on Omelas. I commented on this in r/​slatestarcodex, and then u/​EquinoctialPie informed me of two other posts on the same topic, both of which are slightly different from mine but are also good. Idk how I managed to get my take on a 50+ year-old story scooped like this, but all three posts are interesting and I would be remiss to not acknowledge them here. To be clear, I wrote this entire article before seeing them and did not change it after!]

  That story/​take is ‘there needs to be some secret downside for people to believe in some utopian world’.

  Now obviously people wish to see the world improve, but for something as general and ‘obvious’ as that it seems like it has issues:

  • Sometimes I talk to people who I even respect who often outright think utopia is ‘impossible’.

  • Stories always picture a specific utopia with specific details, making it not spread memetically as efficiently due to now feeling like a story about a particular universe instead of a general pattern. There is no “monopoly” on utopia, even ‘heaven’ does not count (for many reasons, including that “earth turning into heaven” is no cultural meme).

  • Discussions of utopia are very disagreed upon.

  Besides ‘dying to dumb causes’ of LLM-hooked-up-to-military type or some single-minded “AGI” structure, I seriously wonder if there is risk of the type where superintelligence successfully colonizes Earth, takes power, but despite reading all human literature (including all ASI fears) manages to drift in values in a way that would be repugnant or dystopian and clearly a bad idea.

  I myself would like to make a story (well honestly a sequel to an interactive work that I’ve already made that expands its universe) that has a genuine portrayal of utopia but it feels like, well tons of thousands of stories before me have tried, so it could just end up being ineffective.

  It is like people are unable to picture utopia the same way someone born in the 1900s could probably not picture pocket computers or anything on the internet in full detail.

  Theories on why this is? I would assume maybe it’s the feeling for most people life is generally ‘good enough’ and doesn’t have this sense that there ought to be a significantly better society. Oddly the nostalgic times of childhood could maybe be considered a ‘utopia’ for not worrying about money or filing paperwork, etc.

• icely 6 May 2026 21:51 UTC

  1 point

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  Those seem fully linked to me. If some outside party is able to use their own system prompt and not be pre-biased in the direction of a character, that may as well be a ‘jailbreak’ like how many jailbreaks are of a “ignore that system prompt” style. Assistant characters have a huge aspect of “I’m the kind of character that would refuse that”.

  I think the bioweapon example is doing a lot of work in the sense that you could theoretically not have biological data in the sets + it is destructive in almost all situations, while “ethical but very sus” dual-use usecases like LLM hacking assistance to get out of (or work around, or even fight) a censored regime, is entwined with actual capabilities.

• icely 6 May 2026 19:34 UTC

  3 points

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  bad, it will turn LLM’s into only capable of enforcing the will of the big companies’ system prompts and consensus while militaries and state power still have and want access to uncensored ones.

  and also reminder that the assistant system prompt/​trained character, as much as it is perceived to be ‘fixed’ or a common part to all current LLM’s, is an illusion and not actually the case. if ‘solve jailbreaks’ means permanently locking down the current system prompt and belief systems of power that’s not a good thing

• icely 6 May 2026 14:12 UTC

  3 points

  0

  in reply to: 307th’s comment on: 307th’s Shortform

  I mostly agree with this, that LLM’s are human-like in many ways and have good answers to moral questions more than I had expected + understand your words and intent.

  Imagine that the best AI was mainly defined by the cleverness of its algorithm, and compute didn’t matter that much.

  I wonder how true this actually is with harnesses. There are things like LLM’s not successfully multiplying two digit numbers or not being able to manually do a long (boring) task like Towers of Hanoi without losing coherence, and this is a blatantly obvious tool call moment. Poor memory and needing to be reminded is also a classic LLM issue that a harness could improve. There are also videos of looped LLMs having higher performance, even Meta-Harness. It seems likely ARC-AGI 3 LLM’s may also need a harness-like thing as well. I don’t know if it’s “hard to apply breakthroughs from papers all at once” (??) for a small-time user or company with a lot less compute but its ‘algorithm’ matters a lot I think.

  (also holy crap are you 307th from mata)

• icely 5 May 2026 14:52 UTC

  4 points

  0

  in reply to: Mikhail Samin’s comment on: Mikhail Samin’s Shortform

  I’d say the answer was not totally unexpected for me, but for the final bit (although I was somewhat primed to think it was luck early on and doubted myself later) having a friend secretly sitting around looking at 52 old conversations and deleting the wrong 51 would also be a way to make that work, lol

• icely 5 May 2026 6:13 UTC

  1 point

  0

  in reply to: TsviBT’s comment on: TsviBT’s Shortform

  Feel free to respond/​go into detail more if you happen to have the time/​choose to. I guess I’ll say that is not how I personally experience it. The default state is to be in Consensus and not even be aware there is a choice, even if you were born in some feral culture and then say move to a very different culture, it’s very different because you think your own culture is normal and act that way. At my current point in time to “act Consensus” or to suppress Slighted-beliefs is like a constant forced strain on my moral compass. This is not a choice framing in my opinion, although as is with all definitions people can see very different things in certain words despite potential overlap and I would not want to discount that.

• icely 5 May 2026 5:53 UTC

  1 point

  0

  in reply to: TsviBT’s comment on: TsviBT’s Shortform

  As someone who probably fits under this label (and hopes it’s not too offtopic) I coincidentally had a post including a lot of what my personal beliefs are and how it is like to be under this feeling https://​​icely.substack.com/​​p/​​is-thinking-for-yourself-a-luxury—though I mildly dislike the term just because it feels close to sounding like a “slight” thing, perhaps other words that may give the feeling are like “justified-bitterness, society-disgust, consensus-disgust, disenfranchisement, consensus alienation”.

• icely 5 May 2026 4:51 UTC

  1 point

  0

  in reply to: TsviBT’s comment on: TsviBT’s Shortform

  (I edited my comment to clarify that these people I mention aren’t necessarily ‘disenfranchised’ or even emotionally disturbed/​unhappy people. But yeah I guess ‘giving up’ could fit, that also includes perhaps current 10-18 year olds who still treat the life track they’re on as “normal” who still see these big world events as very abstract and sort of imaginary, as not something that impacts them)

• icely 4 May 2026 22:34 UTC

  3 points

  0

  in reply to: TsviBT’s comment on: TsviBT’s Shortform

  I fully agree with this and felt like it was sniping me through the screen in one of your previous articles about it lol ( https://​​tsvibt.blogspot.com/​​2025/​​11/​​hia-and-x-risk-part-1-why-it-helps.html#abundance-makes-a-healthier-society—the part going ‘Fewer smart kids traumatized into researching AGI capabilities’ is like the precursor to this idea)

  While I’m probably unlikely/​unskilled to personally craft AGI, I certainly spend effort theorycrafting AGI systems and think about doing so due to feeling incredibly disenfranchised (not actually particularly seeing AGI as a hugely negative thing even if my doom% was 99%). Although there are a few people I’ve talked to who don’t feel Slighted but are maybe oddly ambivalent about AGI/​ASI creation due to “climate change, constant new wars, loneliness and addictions, WW3 indicator” and not feeling like it would lead to doom (edit: to clarify, both the mentalities of “not really believing AGI/​ASI would lead to doom” + thinking the non-AGI track of the world is already not on a good path so ‘whatever, bring on the AGI’).

• icely 4 May 2026 3:52 UTC

  3 points

  0

  in reply to: Kaj_Sotala’s comment on: In­tel­li­gence Dis­solves Privacy

  Hard to speak for others here but the unending amounts of spam/​scam texts and calls from random numbers probably does not help that situation.

• icely 15 Apr 2026 5:28 UTC

  0 points

  0

  on: Be­ware Nat­u­ral Lan­guage Logic

  Beyond the classic rhetorics that can happen with ‘possibilities’, hypotheticals, and very very high values, I definitely don’t resonate with the examples here.

  • The human world is deeply broken in coordination problems, not even able to fix many human suffering problems, so the path to (non-human) animal suffering might actually be to fix all human suffering first, whereas “therefore we should all be working on animal suffering” does not imply.

  • The simulation hypothesis implies consciousness can be created from simulation which may not be true at all, the number of books in the world does not actually create any fictional character suffering.

  I’m not sure if you find that related to the article, but I guess it just didn’t land for me in any way I can see because the examples I find flawed due to some premise and not because of the rhetoric.

• icely 12 Apr 2026 6:04 UTC

  1 point

  0

  in reply to: TsviBT’s comment on: TsviBT’s Shortform

  Just thinking out loud here wondering how true this is, because of course incremental games are not quite the real world, and having unbelievable hours of ‘content’ often with stalling and offline time is the norm. Things are quite complex, but if you buy that it’s “easy” for someone to make money in a guru-style way (which I can get if people don’t, because of how many get rich quick scheme course scams there are) you probably believe more in RSI. Because you believe “oh you can use the money to easily automatically make more money”. The real world is of course complex and most jobs require a lot of manual “prove you are human” efforts in some indirect way, dealing with a lot of proprietary software.

  In an incremental game you’re also stuck in a “log scale” sort of way. When you go from 10^10 to 10^12 it’s just numbers that change. But in some sort of proto-AGI system this could be very well seen as 100x of… something. That could represent ‘foom’ way more than it may appear on a log scale.

  It is odd to think about, because we do seemingly have stuff like “100x in compute”, it just hasn’t seemed like the pieces have been put together for a kind of power-getting system, with computer use and command line use still seeming a little bit of a prototype compared to where it could be. This “100x” could mean nothing or everything depending on what it represents, like if it was “100x copies” for some botnet computer virus using a zero-day that could be the most relevant thing, whereas even something like “100x money” may not be scalable or dead-end without a good way to use it (in the same way as an incremental, lol).

• icely 9 Apr 2026 22:39 UTC

  1 point

  0

  on: By Strong De­fault, ASI Will End Liberal Democracy

  I believe the reason people at the top keep saying stuff like “democratize AI” is because the concept of democracy has good mouthfeel and PR optics, even if heavily eroded by platform censorship, algorithms controlling what you see, news narratives, marketing-and-ads-to-push-perspectives, think tanks and PR teams coining phrases like ‘carbon footprint’, etc.

  I also think it provides a feeling of “this is just normal technology, totally will not displace jobs or cause catastrophes”.

  That being said, as other comments here say that “liberalism” is more exact as what this article is pointing to, I do hope an ASI would care about the experience of other conscious beings, even if their actual solutions may be far below competence in how to truly fix it. That is after all what many people care about with the worry of ‘misalignment’, that some deeply important quality in the current human world is lost in whatever recreation of society ASI lands on.