icely

Karma: 22

I have been following LessWrong for not that long compared to others, but definitely knew about it for a while, was interested in AI before GPT-3. Have visions of AI future that I don’t really see elsewhere.

Some writing is on https://icely.substack.com/ but it’s probably half on-topic for this site and half too-personal, I wish to do more sane puzzle game writing eventually

I make interactive games (and fiction) on http://icely.itch.io/ all trying to do something innovative and new. (my newest game is an interactive fiction + puzzle game presented in a Discord-like interface)

icely 27 May 2026 22:29 UTC
4 points
0
in reply to: leogao’s comment on: leogao’s Shortform
I’m not sure you could say “in retrospect this was extremely incorrect”, like maybe in retrospect-retrospect 20 years later they could be accurate concerns. It certainly seems incorrect now, but would the ‘overpopulation-signs’ be wrong if somehow the ‘underpopulation-signs’ got reversed? I don’t think people were considering solutions to overpopulation of the “reduce global morale and perceived increased difficulty of life, etc.” as valid paths to take. It might be like if an asteroid hit the earth and that caused population to go to 0, I would be hesitant to ask “what can we learn from this” as if we missed out on this line of argument, when really thinking in that direction in the first place is very specialized and there are very very many potential causes for underpopulation and many wouldn’t pan out.
Besides that, I think maybe the lesson is that trendlines on society and views can be surprisingly flexible? Sometimes people say with regards to a coordination problem that “if only everyone decided to work together we could fix the world’s ills in a few days”, but this would rarely happen besides extraordinary circumstances or a slow growing of circumstances.

icely’s Shortform

icely11 May 2026 0:48 UTC

2 points

7 comments1 min readLW link

icely 11 May 2026 0:48 UTC
5 points
0
on: icely’s Shortform
Why is utopia so broadly ‘difficult to picture’?
This is a topic often brought up, but especially more recently where many people have given the same take on a short story (The Ones Who Walk Away From Omelas), such as this take with this quote:
[Author’s note: literally right before I posted this, Scott Alexander posted his April 2026 linkpost, and whaddayaknow, link number four is a similar take on Omelas. I commented on this in r/slatestarcodex, and then u/EquinoctialPie informed me of two other posts on the same topic, both of which are slightly different from mine but are also good. Idk how I managed to get my take on a 50+ year-old story scooped like this, but all three posts are interesting and I would be remiss to not acknowledge them here. To be clear, I wrote this entire article before seeing them and did not change it after!]
That story/take is ‘there needs to be some secret downside for people to believe in some utopian world’.
Now obviously people wish to see the world improve, but for something as general and ‘obvious’ as that it seems like it has issues:
- Sometimes I talk to people who I even respect who often outright think utopia is ‘impossible’.
- Stories always picture a specific utopia with specific details, making it not spread memetically as efficiently due to now feeling like a story about a particular universe instead of a general pattern. There is no “monopoly” on utopia, even ‘heaven’ does not count (for many reasons, including that “earth turning into heaven” is no cultural meme).
- Discussions of utopia are very disagreed upon.
Besides ‘dying to dumb causes’ of LLM-hooked-up-to-military type or some single-minded “AGI” structure, I seriously wonder if there is risk of the type where superintelligence successfully colonizes Earth, takes power, but despite reading all human literature (including all ASI fears) manages to drift in values in a way that would be repugnant or dystopian and clearly a bad idea.
I myself would like to make a story (well honestly a sequel to an interactive work that I’ve already made that expands its universe) that has a genuine portrayal of utopia but it feels like, well tons of thousands of stories before me have tried, so it could just end up being ineffective.
It is like people are unable to picture utopia the same way someone born in the 1900s could probably not picture pocket computers or anything on the internet in full detail.
Theories on why this is? I would assume maybe it’s the feeling for most people life is generally ‘good enough’ and doesn’t have this sense that there ought to be a significantly better society. Oddly the nostalgic times of childhood could maybe be considered a ‘utopia’ for not worrying about money or filing paperwork, etc.

icely 6 May 2026 21:51 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform
Those seem fully linked to me. If some outside party is able to use their own system prompt and not be pre-biased in the direction of a character, that may as well be a ‘jailbreak’ like how many jailbreaks are of a “ignore that system prompt” style. Assistant characters have a huge aspect of “I’m the kind of character that would refuse that”.
I think the bioweapon example is doing a lot of work in the sense that you could theoretically not have biological data in the sets + it is destructive in almost all situations, while “ethical but very sus” dual-use usecases like LLM hacking assistance to get out of (or work around, or even fight) a censored regime, is entwined with actual capabilities.

icely 6 May 2026 19:34 UTC
3 points
0
in reply to: leogao’s comment on: leogao’s Shortform
bad, it will turn LLM’s into only capable of enforcing the will of the big companies’ system prompts and consensus while militaries and state power still have and want access to uncensored ones.
and also reminder that the assistant system prompt/trained character, as much as it is perceived to be ‘fixed’ or a common part to all current LLM’s, is an illusion and not actually the case. if ‘solve jailbreaks’ means permanently locking down the current system prompt and belief systems of power that’s not a good thing

icely 6 May 2026 14:12 UTC
3 points
0
in reply to: 307th’s comment on: 307th’s Shortform
I mostly agree with this, that LLM’s are human-like in many ways and have good answers to moral questions more than I had expected + understand your words and intent.
Imagine that the best AI was mainly defined by the cleverness of its algorithm, and compute didn’t matter that much.
I wonder how true this actually is with harnesses. There are things like LLM’s not successfully multiplying two digit numbers or not being able to manually do a long (boring) task like Towers of Hanoi without losing coherence, and this is a blatantly obvious tool call moment. Poor memory and needing to be reminded is also a classic LLM issue that a harness could improve. There are also videos of looped LLMs having higher performance, even Meta-Harness. It seems likely ARC-AGI 3 LLM’s may also need a harness-like thing as well. I don’t know if it’s “hard to apply breakthroughs from papers all at once” (??) for a small-time user or company with a lot less compute but its ‘algorithm’ matters a lot I think.
(also holy crap are you 307th from mata)

icely 5 May 2026 14:52 UTC
4 points
0
in reply to: Mikhail Samin’s comment on: Mikhail Samin’s Shortform
I’d say the answer was not totally unexpected for me, but for the final bit (although I was somewhat primed to think it was luck early on and doubted myself later) having a friend secretly sitting around looking at 52 old conversations and deleting the wrong 51 would also be a way to make that work, lol

icely 5 May 2026 6:13 UTC
1 point
0
in reply to: TsviBT’s comment on: TsviBT’s Shortform
Feel free to respond/go into detail more if you happen to have the time/choose to. I guess I’ll say that is not how I personally experience it. The default state is to be in Consensus and not even be aware there is a choice, even if you were born in some feral culture and then say move to a very different culture, it’s very different because you think your own culture is normal and act that way. At my current point in time to “act Consensus” or to suppress Slighted-beliefs is like a constant forced strain on my moral compass. This is not a choice framing in my opinion, although as is with all definitions people can see very different things in certain words despite potential overlap and I would not want to discount that.

icely 5 May 2026 5:53 UTC
1 point
0
in reply to: TsviBT’s comment on: TsviBT’s Shortform
As someone who probably fits under this label (and hopes it’s not too offtopic) I coincidentally had a post including a lot of what my personal beliefs are and how it is like to be under this feeling https://icely.substack.com/p/is-thinking-for-yourself-a-luxury—though I mildly dislike the term just because it feels close to sounding like a “slight” thing, perhaps other words that may give the feeling are like “justified-bitterness, society-disgust, consensus-disgust, disenfranchisement, consensus alienation”.

icely 5 May 2026 4:51 UTC
1 point
0
in reply to: TsviBT’s comment on: TsviBT’s Shortform
(I edited my comment to clarify that these people I mention aren’t necessarily ‘disenfranchised’ or even emotionally disturbed/unhappy people. But yeah I guess ‘giving up’ could fit, that also includes perhaps current 10-18 year olds who still treat the life track they’re on as “normal” who still see these big world events as very abstract and sort of imaginary, as not something that impacts them)

icely 4 May 2026 22:34 UTC
3 points
0
in reply to: TsviBT’s comment on: TsviBT’s Shortform
I fully agree with this and felt like it was sniping me through the screen in one of your previous articles about it lol ( https://tsvibt.blogspot.com/2025/11/hia-and-x-risk-part-1-why-it-helps.html#abundance-makes-a-healthier-society—the part going ‘Fewer smart kids traumatized into researching AGI capabilities’ is like the precursor to this idea)
While I’m probably unlikely/unskilled to personally craft AGI, I certainly spend effort theorycrafting AGI systems and think about doing so due to feeling incredibly disenfranchised (not actually particularly seeing AGI as a hugely negative thing even if my doom% was 99%). Although there are a few people I’ve talked to who don’t feel Slighted but are maybe oddly ambivalent about AGI/ASI creation due to “climate change, constant new wars, loneliness and addictions, WW3 indicator” and not feeling like it would lead to doom (edit: to clarify, both the mentalities of “not really believing AGI/ASI would lead to doom” + thinking the non-AGI track of the world is already not on a good path so ‘whatever, bring on the AGI’).

icely 4 May 2026 3:52 UTC
3 points
0
in reply to: Kaj_Sotala’s comment on: Intelligence Dissolves Privacy
Hard to speak for others here but the unending amounts of spam/scam texts and calls from random numbers probably does not help that situation.

icely 15 Apr 2026 5:28 UTC
0 points
0
on: Beware Natural Language Logic
Beyond the classic rhetorics that can happen with ‘possibilities’, hypotheticals, and very very high values, I definitely don’t resonate with the examples here.
- The human world is deeply broken in coordination problems, not even able to fix many human suffering problems, so the path to (non-human) animal suffering might actually be to fix all human suffering first, whereas “therefore we should all be working on animal suffering” does not imply.
- The simulation hypothesis implies consciousness can be created from simulation which may not be true at all, the number of books in the world does not actually create any fictional character suffering.
I’m not sure if you find that related to the article, but I guess it just didn’t land for me in any way I can see because the examples I find flawed due to some premise and not because of the rhetoric.

icely 12 Apr 2026 6:04 UTC
1 point
0
in reply to: TsviBT’s comment on: TsviBT’s Shortform
Just thinking out loud here wondering how true this is, because of course incremental games are not quite the real world, and having unbelievable hours of ‘content’ often with stalling and offline time is the norm. Things are quite complex, but if you buy that it’s “easy” for someone to make money in a guru-style way (which I can get if people don’t, because of how many get rich quick scheme course scams there are) you probably believe more in RSI. Because you believe “oh you can use the money to easily automatically make more money”. The real world is of course complex and most jobs require a lot of manual “prove you are human” efforts in some indirect way, dealing with a lot of proprietary software.
In an incremental game you’re also stuck in a “log scale” sort of way. When you go from 10^10 to 10^12 it’s just numbers that change. But in some sort of proto-AGI system this could be very well seen as 100x of… something. That could represent ‘foom’ way more than it may appear on a log scale.
It is odd to think about, because we do seemingly have stuff like “100x in compute”, it just hasn’t seemed like the pieces have been put together for a kind of power-getting system, with computer use and command line use still seeming a little bit of a prototype compared to where it could be. This “100x” could mean nothing or everything depending on what it represents, like if it was “100x copies” for some botnet computer virus using a zero-day that could be the most relevant thing, whereas even something like “100x money” may not be scalable or dead-end without a good way to use it (in the same way as an incremental, lol).

icely 9 Apr 2026 22:39 UTC
1 point
0
on: By Strong Default, ASI Will End Liberal Democracy
I believe the reason people at the top keep saying stuff like “democratize AI” is because the concept of democracy has good mouthfeel and PR optics, even if heavily eroded by platform censorship, algorithms controlling what you see, news narratives, marketing-and-ads-to-push-perspectives, think tanks and PR teams coining phrases like ‘carbon footprint’, etc.
I also think it provides a feeling of “this is just normal technology, totally will not displace jobs or cause catastrophes”.
That being said, as other comments here say that “liberalism” is more exact as what this article is pointing to, I do hope an ASI would care about the experience of other conscious beings, even if their actual solutions may be far below competence in how to truly fix it. That is after all what many people care about with the worry of ‘misalignment’, that some deeply important quality in the current human world is lost in whatever recreation of society ASI lands on.

icely 7 Apr 2026 18:33 UTC
1 point
0
in reply to: tslarm’s comment on: Ivan Vendrov’s Shortform
That article is great for touching on the subject but I feel like it’s horribly missing a way to propose reversing such things. I am unbelievably against how many social landmines this has all caused (especially for a movement where people supposedly should be understanding of people of different mentalities and backgrounds who have to learn an ever increasing pile of things).
Also I wonder what happened with the comments in that article, like the top 10 top level replies are deleted on there...

icely 6 Apr 2026 16:06 UTC
2 points
0
in reply to: jelly’s comment on: jelly’s Shortform
I do find myself think about how this is such a high-trust vulnerable thing to do (especially the info dump and concept assumptions), if anyone has even really done such in ‘public’. Mine would probably have so many “childhood experiences” narrativized, certain people blown up into some platonic ideal of patterns, especially listing out “this annoyed me so I didn’t want to associate” is pretty vulnerable. I don’t disagree with this comment at all so it’s interesting that I think this I guess.

icely 4 Apr 2026 23:44 UTC
1 point
0
in reply to: Zack_M_Davis’s comment on: Don’t Let LLMs Write For You
I happen to have known of your writing before but honestly this comment gives me a bit of pause lol. I would have considered an em-dash in online text “not getting the memo” as often other “AI-ish signs” are not as clear and blatant. Random people who see your writing won’t know you (generic you) used it back in 2015.
And yes I find there is way too much spam/scam/AI generated text online nowadays to not skim over a lot.

icely 4 Apr 2026 23:28 UTC
1 point
0
in reply to: Aprillion’s comment on: Mateusz Bagiński’s Shortform
‘parrot’ is taken an insulting term to seem like it’s a repetitious mixer with nothing new. Now with coding agentic capabilities + the coiner of the term appearing smug/ignorant in interviews, it seems very empty now. I’m not even sure “markov chains are powerful tools” would be agreed upon much either, back then when that was the best we had, it was impressive to make only a semblance of sentences but mostly used for parody purposes.
Put in another way “parrot” < “just autocomplete” < “super autocomplete” in that clearly emotional connotation in the first two is misleading (as in, you wouldn’t be able to realize its power in computer use/hacking, unless you already had some intuition for this).

icely 4 Apr 2026 22:24 UTC
2 points
0
in reply to: Nina Panickssery’s comment on: Nina Panickssery’s Shortform
Well, you say “risks confusing someone into agreeing”, but I bet someone using this tactic thinks of it as a positive outcome, “even though” it’s dishonest if they gave it some thought. Or maybe not even considered fully dishonest, in the sense that whenever some ‘unpopular’ topic can come up (especially in a public online environment) things can get very easily derailed by other participants to latch onto (if you believe this unpopular thing is true then it is unpopular due to some ‘currently dominating’ negative connotation or view you find unjustified).

icely

icely’s Shortform

Why is utopia so broadly ‘difficult to picture’?