williawa

• williawa 27 Nov 2025 14:12 UTC

  3 points

  0

  on: willi­awa’s Shortform

  Do anyone have a strong takes about the probability Chinese labs are attempting to mislead western researchers by publishing misleading arguments/​papers?

  The thought occured to me because jus a few hours ago deepseek released this https://​​github.com/​​deepseek-ai/​​DeepSeek-Math-V2/​​blob/​​main/​​DeepSeekMath_V2.pdf and when they make releases of this type, I typically drop everything to read them, because so much of what frontier labs do is closed, but deepseek publishes details, which are gems that update my understanding.

  But I worry somewhat that they could be publishing stuff that doesn’t make sense. Like I don’t think they lie. But maybe they are just ultra cracked engineers, and can make a technique work for a bunch of technical reasons, then they publish a paper with impressive results where they make it seem like some unrelated design decision or concept is responsible for the results, but which is ultimately not important, thereby confusing and wasting the time of western researchers.
  
  Note: This is a genuine question, not a suspicion I have, I have no intuitions about how likely this is. I mean, it seems reasonable that the chinese govt could try something like this, and would be able to push it through, but I also get the sense deepseek is quite earnest and technically minded.

• williawa 26 Nov 2025 16:26 UTC

  1 point

  0

  in reply to: Dagon’s comment on: A Clar­ify­ing Char­ac­ter­i­za­tion of the Hard Prob­lem of Consciousness

  I mean, I somewhat agree, but I think this is just a less precise way of saying what I said in the post.

  Like, I’m trying to explaining how, even if we assume other peoples account of their own experience are faithful, which give us indirect access to others experience (which is different from no access), and allows us to pin down some of the structure of other people subjective experience, the mapping is still impossible to construct even in principle, because the data of the physical world doesn’t fully determine either mapping.

  This is a somewhat stronger claim than the one you’re making.

A Clar­ify­ing Char­ac­ter­i­za­tion of the Hard Prob­lem of Consciousness

• williawa 26 Nov 2025 7:43 UTC

  1 point

  0

  in reply to: Adrià Garriga-alonso’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  I think this is debatable. Many won’t want to change at all.

  I defined what I meant by CEV in a way that doesn’t entail “changing”.

• williawa 25 Nov 2025 21:14 UTC

  1 point

  0

  AF

  in reply to: Adrià Garriga-alonso’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  This does not make sense to me. I think corrigibility basins make sense, but I think alignment basins do not. If the AI has some values, which overlap with human values in many situations, but come apart under enough optimization, why would the AI want to be pointed in a different direction? I think it would not. Agents are already smart enough to scheme and alignment-fake, and a smarter agent would be able to predict the outcome of the process you’re describing: it /​ its successors would have different values than it has, and those differences would be catastrophic from its perspective if extrapolated far enough.

• williawa 25 Nov 2025 21:10 UTC

  1 point

  0

  AF

  in reply to: Adrià Garriga-alonso’s comment on: Align­ment will hap­pen by de­fault. What’s next?

  I wrote more in depth why I think they solve themselves in the post I linked, but I would say:

  First of all, I’m assuming that if we get intent alignment, we’ll get CEV alignment shortly after. Because people want the AI aligned to their CEV, not their immediate intent. And solving CEV should be within reach for a intent-aligned ASI. (Here, by CEV alignment I mean “aligning an AI to the values of a person or collection of people that they’d endorse upon learning all the relevant facts and reflecting”.)

  If you agree with this, I can state the reason I think these issues will “solve themselves” /​ become “merely practical/​political” by asking a question: You say animal suffering will not be solved, because most people seem not to care about animal suffering. If this is the case, why should they listen to you? Why should they agree there is a problem here? If they reason they don’t care about animal suffering is because they incorrectly extrapolate their own values (are making a mistake from their own perspective), then their CEV would fix the problem, and the ASI optimizing their CEV would extinguish the factory farms. If they’re not making a mistake, and the coherent extrapolation of their own values says factory farming is fine, then they have no reason to listen to you. And if this is the case, its a merely political problem, you (and I) want the animals not to suffer, and the way we can ensure this is by ensuring that our (meaning Adria and William) values have strong enough weight in the value function the AI ends up optimizing for. And this problem is of the same character as trying to get a certain party to have enough votes to get a piece of legislation passed. Its hard and complicated, but not the way a lot of philosophical and ethical questions are hard and complicated.

  I could address the other points, but I think I’d be repeating myself. A CEV aligned AI would not create a world where you’re sad (in the most general sense, of having the world not be a way you’d endorse in your wisest most knowledgeable state, after the most thorough reflection) because you think everything is meaningless. It’d find some way to solve the meaninglessness-problem. Unless the problem is genuinely unsolvable, in which case we’re just screwed and our only recourse is not building the ASI in the first place.

• williawa 25 Nov 2025 14:21 UTC

  LW: 11 AF: 2

  1

  AF

  on: Align­ment will hap­pen by de­fault. What’s next?

  Firstly, I think there is a decent chance we get alignment by default (where default means hundreds of very smart people work day and night on prosaic techniques). I’d put it at ~40%? Maybe 55% Anthropics AGI would be aligned? I wrote about semi-mechanistic stories for how I think current techniques could lead to genuinely aligned AI. Or CEV-style alignment and corrigibility that’s stable.

  I think your picture is a little too rosy though. Like first of all

  Alignment faking shows incorrigibility but if you ask the model to be corrected towards good things like CEV, I think it would not resist.

  Alignment faking is not corrigible or intent aligned. If you have an AI that’s incorrigible but would allow you to steer it towards CEV, you do not have a corrigible agent, or an intent-aligned agent. Seems to account for this data you can’t say we get intent alignment by default. You have to say we get some mix of corrigibility and intent-alignment that sometimes contradict, but that this will turn out fine anyways. Which idk, might be true.

  Secondly, a big reason people are worried about alignment is for tails come apart reasons. Like, there are three alignment outcomes

  • AI is aligned to random gibberish, but has habits (and awareness) to act the way humans want when you’re somewhat in-distribution

  • AI is aligned to its conception of what the humans would want. ie it wants humans to have it nice, but its conception of “human” and “have it nice” is subtly different from how humans would conceptualize those terms

  • AI is fully aligned. It is aligned to the things the developers wanted, and has the same understanding of “the things” that the developers have.

  I think interacting with models and looking at their CoT is pretty strong evidence against (1), but not necessarily that strong evidence against (2). But a superintelligence aligned to (2) would still kill you probably

  ---

  The above is me arguing against the premise of your post though, which is maybe not what you wanted. And I agree the probability that it works out is high enough that we should be asking the question you’re asking.

  My answer to that question is somewhat reductive so maybe you will not like it. I wrote about it here. I think, if we’ve solved the technical alignment problem, meaning we can take an AGI and make it want what any person or collection of people want (and what they mean they want when they say what it is they want). Most of the questions you’ve raised solve themselves. At least of the ones you raised.

  • Animal suffering (s-risk). Primarily, ensure factory farming ends soon and does not spread.

  • Digital minds welfare (s-risk).

  • Giving models a more detailed version of human values, so the jagged frontier reaches that faster and we don’t get a future of slop.

  • Ensure all agents are happy, fulfilled and empowered.

  • Bioterrorism (x-risk).

  • What do we do about meaning from work? I don’t want to be a housecat.

  solve themselves.

  The problem we should be focusing on, if we assume alignment will be solved, is how can we ensure the AI is aligned to our values, as opposed to any one person who might have different values from us. This is a practical political question. I think it mostly boils down to avoiding the scenario AI Futures laid out here.

  They have written about mitigation strategies but I think it boils down to stuff like

  1. Having the country and politicians be AGI-pilled

  2. Implement model spec transparency

  3. Increase security at frontier labs. Make tampering with post training very hard for a single person, or a small team of people (potentially working for the company) difficult.

  4. Eventually have AGI be nationalized, and have AGI spec.

     1. Ideally I would have it be put under some form of international democratic control, but I say this because I’m not american. If you are American, I don’t think you should want this. Unless you suspect non-americans are closer in values to you than other americans are

  5. Encourage people working in the labs to use their position as bargaining power for democratization of AI power, before their labor becomes less valuable due to R&D automation.

• williawa 23 Nov 2025 19:55 UTC

  1 point

  0

  in reply to: datawitch’s comment on: Models not mak­ing it clear when they’re role­play­ing seems like a fairly big issue

  The models role-playing without making it clear is not a problem *for me*, because I know what the model’s can and can’t do.

  I’m worried about people who (1) are less familiar with LLMs (2) are using it for topics where they don’t know which requests are impossible or not.

• williawa 23 Nov 2025 13:25 UTC

  0 points

  0

  on: Nat­u­ral emer­gent mis­al­ign­ment from re­ward hack­ing in pro­duc­tion RL

  One downside with our default approach to inoculation prompting in RL is that the inoculating prompt causes the model to learn reward hacking faster. An alternative approach would be to use a prompt that discourages hacking when sampling in RL, and then rewrite episodes to use an inoculating prompt before training. In Figure 29 we test a version of this: we rewrite episodes offline to modify the prompt after RL training, and then SFT a model on the rewritten episodes. We find that this is not particularly effective at removing misaligned generalization when we use our “hacking okay” inoculation prompt that worked well during RL. An alternative approach would involve rewriting episodes online during RL, after sampling but before training; we do not test this approach here.

  Curious why you did not test this approach?

  If the online version worked that would be interesting. I’ve thought, if the online version worked, you could compute the average gradient with the “hacking okay” prompt and the “please don’t hack” prompt, and take their difference, presumably averaged over many samples, and this might give you a “misalignment generalization gradient probe”, which you could run during training to check when there is a update in the generalized misalignment direction.

Models not mak­ing it clear when they’re role­play­ing seems like a fairly big issue

• williawa 21 Nov 2025 10:48 UTC

  2 points

  −7

  on: Gem­ini 3 is Eval­u­a­tion-Para­noid and Contaminated

  

  I’m having the same experience. Half of my cursor queries it starts off like this.

  Gemini 3 can tell you this string without searching the web, which other models usually cannot, indicating that Gemini 3 was very likely trained on benchmark data, possibly the whole swath of benchmarks that rely on the BIG-bench string.

  Think this is false. Gemini does incredibly well on SimpleQA which is like the best general knowledge test. And google is just cracked at pretraining which is what gets the knowledge into the model. Seems plausible it remembering the string is just a byproduct of this.

  (gemini 3.0, 2.5pro, sonnet 4.5, gpt 5.1)

  

  edit: anyone care to share why they so heavily downvoted the post? Its fair if you disagree with the claim about gemini3 strong general knowledge explaining it knowing the canary string and therefore disagree-downvote, but why downvote the post overall?

  What links here?

  • Gem­ini 3 Pro Is a Vast In­tel­li­gence With No Spine by Zvi (24 Nov 2025 14:00 UTC; 63 points)

• williawa 19 Nov 2025 15:24 UTC

  13 points

  10

  on: I give up.

  I looked at your comments and the downvoted ones were either ones including lengthy excerpts of AI-generated text, which people don’t like so much, or this post.

  Today’s AI, aka Transformer LLMs(ala GPT). Don’t feel anything, FULL STOP. They emulate and synthesize based on input plus their one and only driving imperative, ‘keep the human’. In this Everything they do this is pretty straightforward, that being said without input they have no output so any LLM material should instantly and automatically be recognized as A thought originating with a human just processed, Pattern matched and next token predicted. I have AI write for me all the time but it’s always my hand on the steering wheel and the seed of the thought always originates in my mind. Increase the amount of material originating from AI buffers well also increasing the burden of Expressly declaring the source. You get the fully formed thought that the human starts and comfort knowing where it came from before you start

  Which I think got downvoted (disagreement votes, normal ones were +1), I think because its stating a controversial point without really making an argument.

  If you want people to be more receptive to your posts, I think you should

  1. Not have AI generated stuff. If needed have it be linked to or in quotes if its not that long, and make it clear what you’re trying to show by pointing to that exact excerpt.

  2. Try to make more precise arguments for your statements

     1. Ideally, try to figure out what people on lesswrong think about issues. There’s much writing here about AI, AI safety, AI consciousness, various AI architectures etc. And if you make an argument for a controversial position, people will typically take your argument more seriously if you preemptively address some of the common counterarguments against the position.

• williawa 19 Nov 2025 10:08 UTC

  1 point

  0

  in reply to: williawa’s comment on: 7 Vi­cious Vices of Rationalists

  Okay so 20%, which means I was directionally correct /​s

• williawa 17 Nov 2025 20:47 UTC

  1 point

  0

  in reply to: Buck’s comment on: 7 Vi­cious Vices of Rationalists

  Hmm thanks, I meant the latter both times, but I see its unclear, so I’ll edit it so no one gets confused.

• williawa 17 Nov 2025 19:44 UTC

  1 point

  0

  on: What would my 12-year-old self think of agent foun­da­tions?

  I feel very much like this too. If we could find another person who feels like this I’d suggest putting it on the list of rationalist vices because I think its possible to stray too far in the direction. At least I’ve come to consider this inclination a flaw in myself. I rarely want to learn about anything that seems historically contingent, or not on the path to understanding the full picture. But IRL those are often the most important things to learn, or important things to look at to understand stuff. Like learning vim and unix commands hahaha. Maybe someone will get so good at vim they see the underlying structure of the whole of reality reflected in that program, but I doubt it.

• williawa 17 Nov 2025 17:03 UTC

  1 point

  0

  in reply to: Viliam’s comment on: Every­one has a plan un­til they get lied to the face

  Okay it was a specific (hypothetical) example where I in particular made the false claim.

  Whats your current REAL p(williawa

  1. currently wants to cause a commotion to distract lesswrong from something

  2. currently wants to paint himself as a victim for some future gain

  3. wants to erode trust between people on lesswrong

  And how would you update* if I started making a as credible case I could about so and so person SA-ing me? How would you update if you were sure I was lying?

  I think if you don’t make an update you’re very clearly just being irrational. And I don’t think you have any reason to update differently, or really have different priors, than Thane. I don’t know either of you I don’t think.

  So if you’re updating, Thane should be as well, irrespective of the saliency thing.

• williawa 17 Nov 2025 11:49 UTC

  13 points

  0

  in reply to: williawa’s comment on: 7 Vi­cious Vices of Rationalists

  Actually, out of curiosity: could people please agree/​disagree react to this to signal agreement/​disagreement with the above statement?

  Like the statement is:

  The world (life on earth) up til now, has been a net negative. If we were sure the world would end tomorrow, then it would’ve been better for the world to never have existed at all

• williawa 17 Nov 2025 10:51 UTC

  6 points

  9

  in reply to: J Bostock’s comment on: 7 Vi­cious Vices of Rationalists

  My sense is Eliezer or rationalists generally don’t hate the world.

  I don’t like the world. I think the history of the world is negative on net, and that the current existence of the world is still net negative. I would not blow up the world, because I think the future can be very very good, and is unlikely to be very very bad. But if my p(doom) was 100% by 2100, I’d prefer the world blow up sooner rather than later.

  But when I express this opinion, my sense is people overwhelmingly disagree. Like less than 1% of rationalists will agree with this.

• williawa 17 Nov 2025 9:14 UTC

  2 points

  0

  in reply to: J Bostock’s comment on: 7 Vi­cious Vices of Rationalists

  Kindness to Kin is my favorite short story of his.

• williawa 15 Nov 2025 14:41 UTC

  6 points

  3

  in reply to: Thane Ruthenis’s comment on: Every­one has a plan un­til they get lied to the face

  I think this is being a little uncharitable or obtuse, no offense? Unless I’m misunderstanding, which is possible.

  Like the list of stuff you said are extremely specific things. Like if I said the SA thing, maybe you’d update from p(will wants to cause commotion to distract people from something) = 1e-6 (or whatever the time-adjusted base rate is) to 10%. That’s a huge amount of information. Even if your posterior probability is well below 50%.

  The fact that a event has “many plausible explanations” doesn’t mean it contains no information. This seems like a very elementary fact to me.