Michael Samoilov

Karma: 189

Michael Samoilov 27 Sep 2025 19:02 UTC
3 points
0
in reply to: Kaj_Sotala’s comment on: And Yet, Defend your Thoughts from AI Writing
At the same time, especially if you are a complete beginner, it can still be better to start out from just copying it and not thinking about all that other stuff. [...] Getting that initial experiential understanding will then make it easier for you to apply any information related to how to do many of the “project management” tasks.
Yeah, you’re right—I think the absolutism can pull the ladder up from beginners.
I’ll say, I see the absolutism that I advocate for as “aspirational absolutism”, like how the commandment “You shall not murder” should be understood as practically absolute (ie, people do not have license to set personal thresholds of which murder is okay, because that does not technically forbid setting a threshold of “this person inconveniences me”). But everyone also recognizes that there are edge cases when its okay to murder—i.e., war, self-defense. So, my essay is phrased strongly because I am trying to protect against the first kind permissiveness, but I do recognize there’s another class of AI being fine. The concern is people thinking they’re doing the second, as a cover for doing the first.
I read your original post as saying “using prose from AI will prevent you from improving at all”. If your position is just “using prose from AI won’t allow you to jump to the AI’s skill level right away”, then I agree, but that seems like a much less interesting claim.
I do think for people who already write on the internet, as opposed to complete beginners, AI does prevent 90-95%+ of potential improvement, and pragmatically speaking my point is true. Partly because “jump to the AI’s skill level right away” doesn’t apply: I think AI’s ability to “express ideas clearly” is often bad. Claude’s suggestions for you in this case is my prototypical example, because:
- It provides a set of options, which “leads” a non-cautious author towards choosing the best one as a base & editing it. But it’s not apparent that they might all share the same fatal flaw (e.g., “[it’s] like [x]”, when that’s not necessary), and you won’t notice because it’s all that’s on offer.
  - I think this applies even to the Claude outputs you do end up accepting.
- It provides options at all, but sometimes, the advice should be to subtract.
- I also think its advice was technically wrong. When it says “that sentence is doing a lot of conceptual heavy lifting without enough setup”, that has two problems:
  1. It’s actually not true; my rendition of the point is actually more brief, needing less setup.
  2. Identifying a problem with that passage was not Claude’s contribution, it was your suggestion.
  - Therefore, someone listening to Claude will be led astray. When you say “Asking AI for prose suggestions is what caused me to think through what I was even trying to say” (ie, even though you didn’t take its advice) -- it means you have to be actively skeptical of outputs, which most people don’t have the endurance to always do—they only reach for AI when they’re already “exhausted,” so to speak. In practice, people will adopt poor artifacts of the writing which aren’t borne from clear thinking. So I still think you would learn more sitting and thinking.

Michael Samoilov 23 Sep 2025 22:35 UTC
3 points
0
in reply to: Kaj_Sotala’s comment on: And Yet, Defend your Thoughts from AI Writing
Yes, your comment is shaping my opinion to concede that examples are useful for teaching ‘binary classification tasks’. “Here’s an example of how to identify redundant words/sentences which don’t contribute to your point,” or “here’s an example how you can write with more vivid language.”
Ie, if you’re new to the gym, you can follow your friend, copy their workout, and learn how to do those exercises. However, just copying them gives you very little insight into why: what made them choose this exercise, which muscle it’s supposed to hit, next time when you come alone and the machine is taken how to ask a stranger if you can work in a set, and a million other open-ended “project management” tasks. I see those as requiring executive function, and knowing those well is a more reliable indicator of if you’ll go to the gym consistently later & improve in the long run, rather than knowledge of one exercise.
I think my unstated contention is that in writing, examples can show you how to put something more briefly, etc., but that doesn’t impart any learning in your subjective executive function and critical thinking. If a good writer did a total rewrite of your paragraph, they may have changed it on so many layers of abstraction (word, sentence, paragraph etc.) at once that it’s impossible to reverse-engineer “Which mental state would produce these changes in the first place, and how do I get that?”
For what it’s worth, when I see your example:
“So the brain can apply some of the same machinery it uses to generate feedback signals for any other task” doesn’t feel clear enough, or feels like it’d need a bit more explanation.
The way my mind parses it is:
- The idea’s expression is not clear because its content is not clear, not the other way around:
  - “some of the same machinery” is a vague proxy for a specific function you actually mean
    Convert to something like “reward circuitry”
  - “[...] any other task” is also an abstract / parent-category proxy for a specific classification
    Convert to something like “non-instrumental tasks”^[1]
- So in the end you’d be left with something like:
  - Pre: “So the brain can apply some of the same machinery it uses to generate feedback signals for any other task”
  - Post: “So the brain can aim its reward circuitry even at non-instrumental tasks.”
I think that’s clearer, obviously pending alignment with (1) what you actually meant, and (2) your target audience. But that only happens after taking effort to clarify what you’re even trying to say, which is the ‘platonic’ writing process, and has lots of friction. If I just dropped the end result without my reasoning, it’s not clear what one could learn from the example to repeat next time, besides “just make it better”. But you do learn that by writing without assistance, clarifying your thoughts (or even asking AI to help you clarify what you mean).
And FWIW, I think this version is a lot better than what Claude came up with, I only reviewed those after. Which I think substantiates that if you did resort to Claude without taking the effort yourself, you would inherit AI barnacles that you don’t actually mean to collect. The Claudisms can be too abstract to even articulate—though, did you catch how each example said “[it’s] like [x]”? I wouldn’t have, unless I looked closer. And that’s the risk someone runs resorting to AI. “[it’s] like [x]” is a construction that evidently leads to wordier explanations. But an editor telling you to avoid that construction will not inoculate yourself from the next Claudism sneaking in (and won’t tell you in which cases that construction is appropriate).
Lastly, Claude is unlikely to ever suggest that you just remove the passage, which is what you ultimately did. That too is an example of what becomes clear when you think about what you’re even trying to say, rather than asking AI to “convert” your idea into fully phrased thoughts.
1. ^
  By the way, I have no issue with using AI like a tool, e.g., asking it “term for a task you can’t make progress in, as in, waiting for a bus couldn’t be done any better or worse than the one way”. But this only works well once you know what you really need, as opposed to asking for AI’s composition from the outset.

Michael Samoilov 23 Sep 2025 21:37 UTC
3 points
2
in reply to: AlanCrowe’s comment on: And Yet, Defend your Thoughts from AI Writing
I see your point—at first I was thinking:
I don’t think AI would have trouble differentiating between the senses of “sound” (using Eliezer’s essay as an example)
But actually it seems like you’re saying:
Suppose we live in a world before people recognized a distinction between sound (audio waves) and sound (aural sensation). In this world, AI trained on the corpus of human text would not spontaneously generate this distinction (one, it doesn’t have the knowledge, and two, its dissuaded from even conjecturing it, because neologisms are taboo). But we don’t even need to ‘suppose’ this world exists—we do actually live in it now, it just applies to concepts more nuanced than “sound”.
I think neologisms are interesting because on one hand, it is annoying to see terms “astroturfed” (e.g., sonder),^[1] or have insane mismatch between their sound and meaning (e.g., “grok” which people use as “to profoundly understand”, yet sounds more like a clunky word for a caveman’s lack of understanding. Its “etymology” is quite fitting (it’s supposed to be unrelatable),^[2] but it’s a shame the term caught on).
On the other hand, I think much of the pursuit of knowledge is building towards finer and finer distinctions in our experience of reality. This necessitates new words.
For whatever reason, some morphologies seem more tasteful than others, such as ‘common extensions’ (e.g., ChatGPT → ChatGPTism), or ‘combining neoclassical compounds’ (e.g., xeno- + -cide = xenocide, from Ender’s Game), or even just ‘adding standard-word qualifiers’ (e.g., your example of splitting “defensive alliance” into “chaining alliance” and “isolating alliance”). I think most of the people who find success in coining terms probably do it in these more intuitive ways, rather than purely ‘random’ morphologies—find an excerpt from Nabeel Qureshi’s post, Reflections on Palantir:
One of my favorite insights from Tyler Cowen’s book ‘Talent’ is that the most talented people tend to develop their own vocabularies and memes, and these serve as entry points to a whole intellectual world constructed by that person. Tyler himself is of course a great example of this. Any MR reader can name 10+ Tylerisms instantly - ‘model this’, ‘context is that which is scarce’, ‘solve for the equilibrium’, ‘the great stagnation’ are all examples. You can find others who are great at this. Thiel is one. Elon is another (“multiplanetary species”, “preserving the light of consciousness”, etc. are all memes). Trump, Yudkowsky, gwern, SSC, Paul Graham, all of them regularly coin memes. It turns out that this is a good proxy for impact.
1. ^
  From the Dictionary of Obscure Sorrows, whose whole project is to coin new terms for phenomena which don’t yet have names.
2. ^
  Wikipedia:
  Robert A. Heinlein originally coined the term grok in his 1961 novel Stranger in a Strange Land as a Martian word that could not be defined in Earthling terms, but can be associated with various literal meanings such as “water”, “to drink”, “to relate”, “life”, or “to live”, and had a much more profound figurative meaning that is hard for terrestrial culture to understand because of its assumption of a singular reality.

Michael Samoilov 23 Sep 2025 7:31 UTC
5 points
4
in reply to: AlanCrowe’s comment on: And Yet, Defend your Thoughts from AI Writing
You may already be aware, but part of Eliezer’s Sequences (his post Disputing Definitions) covers how by disambiguating homonyms,^[1] we can talk about our differing (or not) intuitions of what exists in objective reality, rather than disagreements about definitions.
But I haven’t seen reason to believe that AI has this problem particularly badly, in that I think its ‘vector space’ is decent enough at understanding which sense of the ‘same’ word is being used given the surrounding context of other words. Or at least it would readily parse the ambiguous senses if asked, as opposed to a biased human who might entrench and “resist any attempt to improve language,” as you mentioned.
1. ^
  Or I’d also add, cases of the equivocation fallacy; “a word or phrase with multiple meanings is used ambiguously in an argument, shifting its meaning between premises or between a premise and the conclusion”

Michael Samoilov 23 Sep 2025 7:17 UTC
4 points
3
in reply to: The Dao of Bayes’s comment on: And Yet, Defend your Thoughts from AI Writing
Believe me, I agree that hating on AI writing because it’s AI is a mistake (the genetic fallacy) -- that is the thesis of my essay before this one, A Thoughtful Defense of AI Writing (and that perspective was controversial here on LessWrong).
That is mainly from a reader’s perspective—don’t judge pirates by if it was AI’s idea, judge pirates by if that was a good or bad inclusion in that part of your campaign.
Note, I think there’s a difference between “I am setting out to write a story with AI” (as you describe, which I think is almost entirely harmless), vs. “I am trying to express my idea, but can’t figure it out, so I will amputate part of myself to install whatever spills out of the robot” (which I see as atrophying your creative and critical thinking).
This essay argues from the writer’s perspective (as Orwell argued about trite platitudes in his time) that you risk picking up bad habits which themselves disable your critical faculties to notice and get rid them (by encouraging habitual lapses into laziness, and adding to the ‘switching cost’ of going back to free thinking).
---
Are the best editors really all famous authors as well?
The best editors are likely not famous authors, but are almost certainly good writers. On a tactical level, the alternatives/suggestions they offer would be of great writing quality. And on a strategic level, they need to know how to write the edits and feedback in a way which is clear and convincing to the author. If an editor can’t write good edits, evidently they aren’t one of the best editors. There is a reflexivity where the best knowledge work always requires good writing, because writing is just the primary medium to convey ideas.

Michael Samoilov 23 Sep 2025 6:27 UTC
7 points
2
in reply to: Kaj_Sotala’s comment on: And Yet, Defend your Thoughts from AI Writing
Sometimes I will write something, notice that what I’ve written sounds clumsy but be unable to figure out how to make it better, and then ask an LLM to suggest alternative phrasings. By your arguments, this would imply that this would prevent me from getting any better at writing.
Yes, I think it would prevent you from improving, insofar as you had room to improve tackling that yourself.
But that would imply that you also couldn’t learn from having an editor or English teacher go through what you’ve written and suggest edits.
[...]
People do learn from having a good example modeled to them as well. So an AI having suggested an improvement, in the future it’s likely that my brain will be able to spontaneously generate a similar solution by itself, without AI help.
I don’t think that recognizing an improvement will make you much better. I think that an editor or English teacher can (1) make your piece better by overwriting parts of it (which doesn’t improve your own skills) and (2) make you better by providing explanations/feedback of what’s wrong (but not by overwriting your work).
Trying to learn through indirect lessons via their rewrites is probably similar to how much you’ll improve at Tennis by watching Roger Federer play. Which is, a little, and more than if you hadn’t, but still far less than playing on the court yourself in that same time. The reason is that when you watch Federer play, it’s easy to identify the output of everything he does right, but you get no visibility into the subjective experience—what it’s like inside the mind and body—to play in such a way that results in good output.
To further illustrate when it comes to writing, consider me and my mom’s ESL experiences (footnote 1).^[1]
I would argue there is still “generative effort” within editing (an AI’s) work, because you still need to come up with with reasons for what’s wrong, and alternatives that make it better. It’s magnitudes more “mental exercise” to articulate problems than to merely feel something is wrong. However, in the opposite direction, I think it’s magnitudes less generative effort to articulate problems vs. to write for yourself and “create something from nothing”. In my view, flexing the ‘generative muscle’ is what leads to improvement.
---
In the past, for many of those sentences, I would just have concluded that I’m unhappy with this sentence but I’m not going to figure out anything better, so I’ll just leave it in and move on.
This is a related reflex I’ve been actively trying to fight in my own writing process. Often I would come to a sentence, be unable to oneshot it, then [leave a rough outline in brackets to return to later]. These days, I’ve been trying to push through, and just write something passable. Partly because leaving work to return to duplicates my workload,^[2] and partly because I believe that discipline trains my ‘generative muscle’.
1. ^
  First, me: I understand Russian but can’t speak it; classic immigrant child experience.^[3] My understanding does not convert into generative capability. It’s a different subjective experience to grope and flail at trying to pluck words from an inaccessible aether, vs. skipping that and reviewing words in front of you. Like how a multiple choice test is easier to bullshit than free text questions.
  Second, my mom: She reads plenty of English literature, and has a great sense for when its written well vs. poorly. But she is aware that her writing is, say, typical for an immigrant, and also that AI is ‘better’ but ‘soulless’. If all it took to improve was to notice deficiencies (or to know well-written examples), then couldn’t she just edit her emails until they’re good? But it’s not that easy for her.
2. ^
  In his blog post Lessons from Peter Thiel, Joe Lonsdale writes:
  6. Don’t waste time talking about what you plan to think about; instead, work through it immediately.
  Intellectual laziness can easily sneak up on you. If you are sitting there talking about problems you plan to solve later, there’s a good chance you are being inefficient. Similarly, in GTD, you don’t put off tasks that only take a couple minutes. In many cases, you can outline and solve or at least clarify any decision or problem you’re confronted with in just a few minutes.
3. ^
  In his blog post Proto-World and the Language Instinct, conlanger Mark Rosenfelder writes:
  It’s commonly thought that children absorb languages effortlessly; but language acquisition studies make it clear that this is not so (cf. François Grosjean’s Life with Two Languages, 1984). Language learning takes effort even for children, and children seek to minimize that effort by learning only those languages they absolutely must, in order to communicate. For instance, once children of immigrants discover that their parents speak the national language, they may speak it with them, and retain only passive knowledge of the parents’ native language. Full language ability is likely to continue only when there are people in the child’s life that cannot understand the national language.

Michael Samoilov 22 Sep 2025 4:35 UTC
6 points
2
in reply to: Dom Polsinelli’s comment on: And Yet, Defend your Thoughts from AI Writing
Yes, I do think there will be fewer people with original thoughts. Mostly because original thoughts come from immersing yourself deeply in a topic, and writing about topics is one of our main ways to do that.
Though I don’t think it’s a hopeless battle. If you’re bought into the concept that writing is thinking, then Paul Graham makes the following case for continuing to write by yourself in his essay Writes and Write-Nots:
So a world divided into writes and write-nots is more dangerous than it sounds. It will be a world of thinks and think-nots. I know which half I want to be in, and I bet you do too.
This situation is not unprecedented. In preindustrial times most people’s jobs made them strong. Now if you want to be strong, you work out. So there are still strong people, but only those who choose to be.
It will be the same with writing. There will still be smart people, but only those who choose to be.
I think there aren’t many fields where quantity of output will give you some killer advantage compared to quality of output. As decent AI writing becomes more common, I think it will just raise the quality benchmark, and good thinking beyond that point will be highly valued. My friend Logan makes this point in his essay about thoughtfulness, This was meant for you (LW), as it pertains to business sales.
Sales is, by its very nature, a rivalrous domain. It is a game of limited attention, finite budgets, and constant competition. Multiple vendors want the same customer.
In these environments, the signal that breaks through is rarely the loudest. It is the most precise. It is the one that says: this was meant for you, and no one else. It is a gesture that cannot be mistaken for automation or routine. That is what makes sales so interesting in the context of thoughtfulness. It is a space where commoditized effort is common, and where genuine, specific sacrifice becomes all the more powerful.
[...]
AI SDRs (or any other technology) will get arbitrarily good at solving whatever they can; always, tech will get competitors up to a frontier, and what will push a customer over the edge is differentiation beyond that established frontier.

In the short run, AI SDRs will work as dishonest signals of units of work, but people will wizen up eventually. They are a leverage multiplier.
So even if some AI writing will be required to automate labor, you will always need your own brain to know how to use it better than others.

Michael Samoilov 21 Sep 2025 19:15 UTC
27 points
5
in reply to: Zack_M_Davis’s comment on: And Yet, Defend your Thoughts from AI Writing
You’re correct, it is an allusion. In earlier drafts I did mention, [...] Bedingfield’s song was written just after the dot com boom, but applies equally to AI writing on Substack dot com. It is a clearly joking reference, but I left it included specifically for readers who notice the connection and dig into the lyrics, which were originally figurative, but map closely to my point in that section when taken literally (emphasis my own):
Staring at the blank page before you
Open up the dirty window
Let the sun illuminate the words that you could not find
Reaching for something in the distance
So close you can almost taste it
Release your inhibitions
Feel the rain on your skin
No one else can feel it for you
I do recognize that including high-variance tactics mean they won’t always land with readers. It might be relevant to revive the rest of Orwell’s quote from my epigraph:
[...] Phrases like a not unjustifiable assumption, leaves much to be desired, would serve no good purpose, a consideration which we should do well to bear in mind, are a continuous temptation, a packet of aspirins always at one’s elbow. Look back through this essay, and for certain you will find that I have again and again committed the very faults I am protesting against. [...]

Michael Samoilov 18 Sep 2025 0:31 UTC
1 point
0
in reply to: Random Developer’s comment on: A Thoughtful Defense of AI Writing
I agree the most proliferate AI use case is SEO-spam/content-farming, and that almost no human input is invested to prune the mediocrity typically outputted. I see this as an alignment problem, rather than a problem with AI writing (ie, AI writing tools) in principle. Of course, AI writing out-of-the-box is bad, and a reasonable person should stop reading when they realize that anything they read fails to meet their quality bar.
My concern is when a Bayesian heuristic shifts to Bayesian epistemology, and people believe that “badness” is a property of AI writing, when in reality AI writing can become great. Particularly if you subscribe to a Popperian epistemology rooted in “alternating conjecture and criticism” as an engine for progress—because you can choose to use AI tools in a way to emulate that cycle.
Regarding:
All it would take to improve my opinion of AI writing would be to find myself regularly surprised and delighted by finding new, correct, and non-generic information in pieces obviously written by AIs.
My view is we won’t approach this anytime soon because the human curator’s input, if done well, will transform the writing to no longer seem “obviously written by AI”.

Michael Samoilov 17 Sep 2025 23:44 UTC
3 points
0
in reply to: Seth Herd’s comment on: A Thoughtful Defense of AI Writing
I assume you mean Policy for LLM Writing on LessWrong, thanks for directing me to that, it was interesting to read. It partly boils down to: a human must contribute value-add transformations at some point(s) of the AI writing ‘supply chain’, which I totally agree with.
I think its interesting you mentioned sophistry; I know the dictionary basically flattens it to “deception” but I’ve been interested for a while in its meaning as “style over substance”. In particular, I think much of what makes someone feel subjective epiphany from an idea isn’t usually the idea’s substance, but by shaping its expression into a ‘key’ that fits their particular ‘keyhole’, or knowledge gap. Often this requires phrasing it as some form of “not X but opposite-of-X” (imagine my dismay when AI appropriated this after years of independent thinking on it...). Not to say substance doesn’t matter. I think that often, the process of converting your idea-seed into this “not-but” form will frequently refine it into something clearer and better-scoped than it was before.

Michael Samoilov 17 Sep 2025 21:55 UTC
3 points
0
in reply to: ChristianKl’s comment on: A Thoughtful Defense of AI Writing
Yeah, that’s good point. I was modeling stratifications of lay-usage I’m aware of; treating ChatGPT like a prose dispenser. But if I were to write a second edition of this post, I would include your point.
I think people know to turn AI’s generative capabilities towards selection (ie, “edit this”). What you’ve described seems like: instead of turning AI’s capability towards selection execution, its towards the selection process, (think, “working in the business” vs “working on the business”; the ‘business’ here is ‘improving the writing’). Ie, you let the AI help you gain taste for what you even mean to say, and what you think is worth elaborating on or pruning away.
In any case, I think that builds naturally on the generation/selection framework established in this essay.

Michael Samoilov 17 Sep 2025 17:08 UTC
3 points
2
in reply to: Richard_Kennaway’s comment on: A Thoughtful Defense of AI Writing
Really appreciate the comment, I enjoyed engaging with it! See below:
Not quite. You explicitly quoted this as a piece of AI writing
Ahah, this made me laugh. Yes, that’s true.
---
Falling in love isn’t just about romance. It’s about discovering new parts of yourself.
AI or human, that’s a trite cliche. I’ll take your word for it that you wrote the subsequent expansion into a whole paragraph without AI assistance, but the paragraph is no better than what an AI would come up with.
I think this illustrates my point.
- AI or human, that’s a trite cliche.
  - First: That point was counterintuitive, and therefore not trite, to me. Insights have a subjective property in which, “subverting a misconception with the truth” depends on which misconceptions an audience currently holds. The footnoted article by Esther Perel (Why Happy People Cheat) basically boils down to the same insight in a different form.^[1]
- I’ll take your word for it that you wrote the subsequent expansion into a whole paragraph without AI assistance,
  - Second: By having one’s guard up for AI writing, and having some hesitation/doubt about its authenticity, you’ll shade everything in a cynical tone and become less willing to accept ideas that you may have otherwise found insightful.
- but the paragraph is no better than what an AI would come up with.
  - Third: I argue that if you ever feel this way about something you read, you have license to stop engaging with the material. In this sense, the spectre of AI writing is actually very liberating for developing your taste, even if nothing you ever read was written with AI.
---
That’s another hazard of AI slop: picking up its dull-witted habits by osmosis.
I guess I can give you an advance preview of the next essay: this is the first point it makes. As in, this current essay covers how for reading AI writing, you should develop and trust your own taste / independent judgment of what counts as good, rather than using presence of AI as a proxy.
But when it comes to writing (topic of the next essay), the same prescription to ‘develop and trust your taste’ leads to a near-opposite perspective—don’t use AI. My closing line in this essay; “I handle [writing with AI] like poison”; segues into the following epigraph; a quote by Orwell written about the language of his time, but which maps nearly 1:1 for modern AI usage:
But if thought corrupts language, language can also corrupt thought. A bad usage can spread by tradition and imitation, even among people who should and do know better. The debased language that I have been discussing is in some ways very convenient. Phrases like a not unjustifiable assumption, leaves much to be desired, would serve no good purpose, a consideration which we should do well to bear in mind, are a continuous temptation, a packet of aspirins always at one’s elbow. Look back through this essay, and for certain you will find that I have again and again committed the very faults I am protesting against.
1. ^
  Of course, it is more insightful because it also applies the first point in a short deductive chain to then resolve the titular enigma. Deductive chains are themselves a form of insight, I elaborate somewhat in another essay of mine (Lesswrong, Linkpost).

Michael Samoilov 17 Sep 2025 16:42 UTC
3 points
0
in reply to: Viliam’s comment on: A Thoughtful Defense of AI Writing
To me that seems like the difference between text written by AI, and text written with AI. Selection as a steering mechanism for AI currently only works well with a human curator. This applies during writing (choosing between options, or choosing when to intervene with manual edits), but just as well applies after writing (seeing feedback on a piece, and learning how to improve your own curatorial instinct).
After that, the hope is that with future tools, curating a better past context window will itself lead to better future AI writing outputs. But I don’t think the tools are there yet.

Michael Samoilov 17 Sep 2025 16:30 UTC
5 points
0
in reply to: Adele Lopez’s comment on: A Thoughtful Defense of AI Writing
That’s a great point. Seems like the category of ‘informational writing’ is not so homogeneous. There are at least:
* Evaluative contexts: “I am judging the author (are they worth engaging with? anything from cover letter screening to replying on blogposts), and their writing is the only available proxy to judge the author by.”
* Expository contexts: “I am trying to learn an idea as best as I can.”

Take the example of ‘Falling in love feels good because of endogenous, not exogenous factors’. Even if written by AI, that could be useful + interesting in an expository context, for someone to whom that is new and subversive. But realizing it was written by AI would have a negative impact on any reader in an evaluative context.

I suppose I hadn’t considered the distinction at all, and assumed a focus on expository writing—though ironically, most blogposts are engaged with as evaluative instead!

Though, I think the point of the essay still stands, which is—if AI was used well enough (as judged by a reader’s independent evaluation of how interesting/good the idea is), that is still value delivered to the audience, which may not have existed otherwise.
Consider your footnote: “Imagine how obnoxious it would be if I had someone ghostwriting half of my comments and posts here”—yes, it would be annoying to realize you were debating an automaton. But if you used AI to (1) write clarifying analogies (suppose your “ghostwriting half of my comments and posts here” example was the sort of example an AI could help surface), or (2) develop your opinions beyond what you previously held, I’m not sure a reader would get too annoyed!

Michael Samoilov 26 Nov 2023 21:14 UTC
1 point
0
on: Insulate your ideas
If strong ideas that face friction come out stronger, then why would you need to insulate them behind locked doors from external stimuli? Shouldn’t they easily vanquish external stimuli and validate themselves? Unless the point is to recognize the strength of timeless ideas. But even if an idea worked well the first 999 times, it doesn’t mean it will also do so the 1,000th time—you shouldn’t strive to crystallize tried-and-true ideas into static heuristical husks: Heuristics that almost always work can still critically fail, with black swan moments.
‘Always run faster’ can make you stumble down a hill and break your ankle. ‘Always deploy capital prudently’ can make your business miss out on bold plays, as ChristianKl commented. Since even your best ideas are fallible, the goal should be to discern which part of the idea was wheat, and which was chaff. “Run fast, but not if it’ll overexert yourself, here’s an explanation to assess when that’s about to happen.” “Deploy capital prudently, but make allowance for calculated risks, and here’s an explanation for how to assess those opportunities.”
These distinctions require active criticism ‘in the arena,’ even if—especially if, rather—it’s at the risk of swaying under new information in the ephemeral territory of the immediate. It requires a willingness to suspend an idea’s universality if a better explanation carves an edge case where it doesn’t apply, or even subverts the whole paradigm (slow and steady wins the race!). But that only happens if you’re willing to part with even your best ideas, instead of jailing them behind thick walls and padlocks—which insulates them, yes, in an echochamber.

Michael Samoilov 3 Aug 2023 14:31 UTC
3 points
0
in reply to: MondSemmel’s comment on: Spiral Staircase
It’s interesting to compare the first two points: novel math derivations and remixing old artwork can seem like disparate paths to greater understanding. Yet often, ‘novel’ math derivations are more like the artistic remixes, or pastiches. Gian-Carlo Rota, MIT math & philosophy prof, referenced two ways to come across as genius: either keep a bag of tricks and apply them to new problems, or keep a bag of problems and apply them to new tricks.
Eliezer’s discussion about his work was interesting too, I hadn’t seen that before. Rota also spoke of scientific popularization as your mentioned, saying you’re more likely to be remembered for expository work than for original contributions:
Allow me to digress with a personal reminiscence. I sometimes publish in a branch of philosophy called phenomenology. After publishing my first paper in this subject, I felt deeply hurt when, at a meeting of the Society for Phenomenology and Existential Philosophy, I was rudely told in no uncertain terms that everything I wrote in my paper was well known. This scenario occurred more than once, and I was eventually forced to reconsider my publishing standards in phenomenology.
It so happens that the fundamental treatises of phenomenology are written in thick, heavy philosophical German. Tradition demands that no examples ever be given of what one is talking about. One day I decided, not without serious misgivings, to publish a paper that was essentially an updating of some paragraphs from a book by Edmund Husserl, with a few examples added. While I was waiting for the worst at the next meeting of the Society for Phenomenology and Existential Philosophy, a prominent phenomenologist rushed towards me with a smile on his face. He was full of praise for my paper, and he strongly encouraged me to further develop the novel and original ideas presented in it.
Here’s the source from which I found Rota’s speech—and the fact that I wouldn’t have known of those ideas otherwise—validates the usefulness of repackaging good ideas again! And you’re right that choosing what to curate is a form of originality; choosing the best out of several AI text generations is you applying your taste and sense of relevance, or in other words, bits of selection pressure. So both human contribution and human selectivity can indicate originality. But the same could go for sampling past human work too, in art, math, or otherwise.

Michael Samoilov 20 Apr 2023 19:43 UTC
1 point
0
on: You can tell a drawing from a painting
Coding
In a video by web developer Joshua Morony, he explains why he won’t rely on GPT-4 for his code, despite its efficiency. Me paraphrasing: In coding, knowing why and how a system was designed, edge cases accounted for, etc. is valuable because it contextualizes future decision-making. But if you allow AI to design your code, you’ll lose that knowledge. And if you plan on prompting AI to develop further, you’ll lack the judgment to direct it most coherently.
Writing
Many have written about how it’s the writing process itself that generates the great ideas comprising a final product. I’ve briefly consolidated those ideas before. This would warn against writing essays with ChatGPT. But it would also apply to, e.g., hiring an editor to compile a book from your own notes, rather than writing it yourself. Outsourcing the labour of selecting words, ordering ideas etc. also offloads the opportunity to generate new ideas and fascinating relationships between them—often much greater ideas than the ones you started with. Of course, your editor may notice all this, but the principal-agent problem may prevent them from working beyond their scope. Even if they went above and beyond, the arrangement still depends on you not caring that the ideas in the book aren’t your own, or that you don’t even understand the ideas in your own book, since you aren’t the one who laboured to earn an intuition about them.
Automating some work is fine. I ran both the above paragraphs through ChatGPT for brevity. But even then, I wrote them beforehand myself, which generated many ideas that I didn’t yet have at the outset of writing, for example the principal-agent consideration. I even ran this paragraph through ChatGPT, but chose not to adopt its summary. And in the process of rewriting it, I thought to include the principal-agent example two sentences ago. If I had just asked ChatGPT (or some editor) to write a decent response to this post, even with general direction, I doubt it would have been as thoughtful as I was. But of course, ChatGPT or an editor could have come up with a comment exactly as thoughtful, right?

Michael Samoilov 21 Feb 2023 2:37 UTC
2 points
0
on: Write to Think
Banishing the epistemic status disclaimer to the comments, since it clashes with the target audience and reading experience.
Epistemic status: briefly consolidated insights on writing to think, for newer audiences. Partly interpolates Paul Graham, Herbert Lui, Larry McEnerney.

Michael Samoilov 11 Feb 2023 19:01 UTC
8 points
7
on: Language Ex Machina
I loved this post. Its overall presentation felt like a text version of a Christopher Nolan mind-bender.
- The crescendo of clues about the nature of the spoiler: misattributed/fictional quotes; severe link rot even though the essay was just freshly published; the apparently ‘overwhelming’ level of academic, sophisticated writing style and range of references; the writing getting more chaotic as it talks about itself getting more chaotic. And of course, the constant question of what sort of spoiler could possibly ‘alter’ the meaning of the entire essay.
- I loved the feeling of Inception near the end of the essay when, in the analyst’s voice, it confirms the reader’s likely prediction that it was written by AI, only to reveal how that ‘analyst’ section was also written by AI. Or rather that the voice fluidly changes between AI and analyst, first- and third-person. And when you finally feel like you’re on solid ground, the integrity of the essay breaks down; “” tags make you contend with how no part is certainly all-human or all-AI, and so, does it even matter who wrote it.
- Returning to the spoiler and initial paragraphs after finishing the essay, and getting a profound, contextualized appreciation for what it means. You realize that the essay achieved what it told you it set out to; to convey a salient point through apparent nonsense, validating that such nonsense can be useful, as it explains the process of generating the nonsense. Or in the essay’s words, “[the] string of text can talk about itself [as it] unmask[s] the code hidden within itself.”
The post also shared concepts I now use when thinking about language. My favourite being quantum poetry, associating the artificial (and ‘next-token prediction’) to the humanistic:
Just as the presence of a particle always completely erases the ghost of its wavefunction, [...] so does the presence of a word erase the ghost of the manifold that could have been named. [...]
This is the principle of quantum poetics. The content of poetry is limited not by the poet’s vocabulary, but by the part of their soul that has not been destroyed by words they have used so far. [...] It is the quantum nature of reality that allows for unforeseeable events, stochastic processes, and the evolution of life. Similarly, it is the quantum nature of language that allows for the evolution of meaning, for creativity, for jokes, and for bottomless misunderstandings. [...]
[Generative] systems are entirely too good at hallucinating content that does not exist in the training corpus—content that creates meaningful structures that foster coherent fictive space where there is none. Ironically, this is exactly what we want in a poet—to create new worlds out of nothing but the coupling of waves of possibility drunk from memory
My main response to the essay’s content, is that still, a human in the loop seemed to still be the primary engine for most of the art in the essay. From my understanding of critical rationalism, personhood is mapped to the ability to creatively conjecture and criticize ideas to generate testable, hard to vary explanations of things.
This essay depended on a human analyst to evaluate and criticize (by some sense of ‘relevance’) which generation was valid enough to continue into the main branch of the essay. The essay also depended on a human to decide which original conjecture to write about (also by some sense of what’s ‘interesting’).
Therefore, it seems to me that AGI is still far from automating both of humans’ conjecture and criticism capacities. However, the holistic artistry the essay did push me to consider AGI’s validity more than other text I’ve read, and in that sense, it achieved what it meant to: to connect my prior thoughts to some new idea—both in the real domain—through ‘babble’ of the the imaginary domain.

Michael Samoilov 22 Jan 2023 22:55 UTC
1 point
0
on: What fills a vacuum?
When vacuums exist, it isn’t just random things filling up that time or space, but it’s things that are purposely able to fill up that vacuum.
What do you mean by “purposely?” Didn’t Part I exemplify how random, not purposeful things can fill an empty room?