Shubhorup Biswas

• Shubhorup Biswas 17 Jun 2026 14:48 UTC

  1 point

  0

  on: A 400-year timeline of failed at­tempts to fix a lethal bug in the hu­man soft­ware of in­her­ited concepts

  The 12-penny threshold for death was finally repealed in 1827 - having been in force through an era in which the value of money dropped by a factor of about 200.

  This by itself doesn’t mean much. You could have had 0 inflation for all of this period and a sudden 200x inflation at the end of this period shortly before the law gets repealed.

Es­ti­mat­ing No-CoT Task-Com­ple­tion Time Hori­zons of Fron­tier AI Models

• Shubhorup Biswas 28 May 2026 14:02 UTC

  1 point

  0

  in reply to: Seth Herd’s comment on: Why Phys­i­cal At­trac­tive­ness Mat­ters for Men’s Dat­ing Prospects

  I’m not sure carrying an amount of muscle mass that makes you top tier in physique is actually particularly helpful for health or energy, and it takes a lot of time.

  It may be worth mentioning that short timelines gives most of the audience a comparative advantage in using steroids, i.e. the shorter your timelines, the less you care about long term health issues so that’s one way in which this audience should be more open to steroids than the general population that this sort of looksmaxxing advice is usually aimed at.

• Shubhorup Biswas 6 May 2026 23:59 UTC

  1 point

  0

  in reply to: RHollerith’s comment on: AI un­em­ploy­ment and AI ex­tinc­tion are of­ten the same

  Full control of ASI by a few model developers is IMO the main prosaic worry. The crux(between extinction and prosaic risks) is whether alignment is hard.
  
  > (if we successfully built AI that cared about us in the right way)

  I’m unsure if this means cares about human flourishing/​agency or fully obedient to its developer, but we only need the latter.
  

• Shubhorup Biswas 30 Apr 2026 23:51 UTC

  2 points

  0

  in reply to: anaguma’s comment on: Cy­borg evals

  These are not professional SWEs. Professional SWE tasks look very different to competitive programming problems and also it’s hard to create competitive programming tasks that take 8h-30h.

• Shubhorup Biswas 3 Apr 2026 3:30 UTC

  1 point

  0

  on: My cost-effec­tive­ness unit

  such AI seems worse than the aliens that would otherwise claim a large fraction of the lightcone.

  Any moral intuitions for why so? It seems to me that alien civilisations could also suffer loss of control or gradual disempowerment to AI, or any spectrum of scenarios between these and retaining ‘agency’ whatever that means. Do biologically evolved lifeforms definitely have positive moral valence?

• Shubhorup Biswas 2 Apr 2026 0:37 UTC

  12 points

  0

  in reply to: Kongo Landwalker’s comment on: My hobby: run­ning de­ranged surveys

  Steelman—the hypothetical scenario is incongruous with their current world model, so it being true would require extreme changes in their world model leading to a very weird world which they might not want to live in.

  E.g. How would the brain and memories function under immortality?

  This may also mean that the question is underspecified and the respondent might want to seek further clarification, but the website just doesn’t allow this.

• Shubhorup Biswas 21 Mar 2026 1:23 UTC

  4 points

  0

  on: Null Re­sults From An Orexin RCT

  If sleep deprivation doesn’t actually make you dumber then patching it shouldn’t make you smarter. Would the cognitive battery still be valid?

• Shubhorup Biswas 19 Mar 2026 3:19 UTC

  1 point

  0

  on: [Er­ror com­mu­ni­cat­ing with LW2 server]

  Can language models preserve their own alignment?

  Wouldn’t consider this an argument for; rather a project proposal to empirically test how much models remain “good”

• Shubhorup Biswas 19 Mar 2026 3:01 UTC

  1 point

  0

  on: [Er­ror com­mu­ni­cat­ing with LW2 server]

  Avoiding AI Deception: Lie Detectors can either Induce Honesty or Evasion

  They call this scalable oversight but they both train and evaluate the lie detector probes on labeled DolusChat examples. I don’t get why they call it scalable.
  Claude’s analysis
  https://​​claude.ai/​​share/​​d659b385-f625-4b86-9eb5-f8ce1fea33e5

• Shubhorup Biswas 19 Mar 2026 2:40 UTC

  1 point

  0

  on: [Er­ror com­mu­ni­cat­ing with LW2 server]

  Training fails to elicit subtle reasoning in current language models

  I think Sonnet 3.5 monitor/​Sonnet 3.7 attacker was a narrow intelligence gap.
  
  The paper itself says “as developers continue to scale RL reasoning compute, models may become more capable of subtle reasoning.”
  
  I see this as a minor empirical result with unclear generalisability

• Shubhorup Biswas 19 Mar 2026 1:58 UTC

  1 point

  0

  on: [Er­ror com­mu­ni­cat­ing with LW2 server]

  which proxies to train against.

  https://​​www.lesswrong.com/​​posts/​​G9HdpyREaCbFJjKu5/​​it-is-reasonable-to-research-how-to-use-model-internals-in?commentId=krg2jzDxXhei9vNLj
  
  and Daniel Kokotajlo comment about preserving at least one output stream that isn’t optimised against(this could be activations, while doing cot+output monitoring)

• Shubhorup Biswas 10 Mar 2026 23:54 UTC

  1 point

  0

  in reply to: David Africa’s comment on: Daniel Tan’s Shortform

  Unsure if this is implicit, but I’m excited about an additional affordance for diagnosis—having access to training checkpoints.

  This would be especially useful for models which may acquire some alien drives during post-training, and by the end of it can competently hide it. At the earlier stages, the drive would not be strong enough for the model to develop the instrumental goal of hiding alien drives.

• Shubhorup Biswas 4 Mar 2026 5:20 UTC

  1 point

  0

  on: The per­sona se­lec­tion model

  • Post-training can be viewed as updating this distribution using training episodes as evidence. When training an AI assistant on an (input xx, output yy) pair, hypotheses that predict the Assistant would respond with yy to xx are upweighted; hypotheses that predict the opposite are downweighted.

  • PSM does not rule out learning of new capabilities during post-training. For example, no persona learned during pre-training knows how to use Anthropic’s syntax for tool calling; that capability is learned during post-training. PSM explains this as the LLM learning that the Assistant knows how to use this syntax. The important thing is that the LLM still models the Assistant as being an enacted persona.

  
  This is more than just shifting the distribution of personas learned via pretraining(unless this can be framed as shifting the probability of a tool-calling-assistant from 0 to nonzero).
  
  Importantly, does PSM implicitly claim that personas do not shift in propensities during post-training? Why should this be so/​what conceptual difference between propensities and capabilities might cause this?

  2. IIUC PSM does not make strong predictions about what happens with a lot of post-training(the threshold I’m most interested in here is whatever amount of post-training the first AGI will have).

  In what ways might you expect PSM to breakdown if we do a lot of RL on tiny models?

  3. For the purpose of this post SFT is conceptually clubbed with postraining RL, but mechanistically, I see SFT as just pretraining but stronger(per token). I expect SFT to be able to do the same things as pretraining does, i.e. create new personas. Is this the sort of thing that you would term ‘non-persona agency’ arising from heavy amounts of post-training?

• Shubhorup Biswas 27 Feb 2026 9:48 UTC

  3 points

  0

  on: Dario Amodei – The Ado­les­cence of Technology

  My co-founders at Anthropic and I were among the first to document and track the “scaling laws” of AI systems—the observation that as we add more compute and training tasks, AI systems get predictably better at essentially every cognitive skill we are able to measure.

  
  The scaling laws paper is only about perplexity, or next-token-prediction loss, not ‘every cognitive skill we are able to measure’. To my knowledge there wasn’t much measurement of compute vs cognitive skills in 2020. IMO we don’t have well-validated scaling laws for compute vs benchmark performance, and any that exist are not a straightforward extension of the 2020 paper.
  
  EDIT: The reason I cared enough about this to comment is that the conflation of these two things launders authority in two ways

  1. The predictability of perplexity vs compute in 2020 applied to present day benchmark performance where labs are much more secretive about training details

  2. The novel contribution of Amodei et al to perplexity scaling, which people care much less about now, makes Dario seem prescient and uniquely qualified to comment on benchmark performance, which everyone cares about in 2026
     

  tl;dr: he was among the first to document and track perplexity scaling. Benchmark scaling is a different thing that (1) he was not among the first to document and track and (2) is not as empirically robust with verifiable open source data.

[Paper] How does in­for­ma­tion ac­cess af­fect LLM mon­i­tors’ abil­ity to de­tect sab­o­tage?

Aether is hiring tech­ni­cal AI safety researchers

• Shubhorup Biswas 27 Nov 2025 15:02 UTC

  1 point

  0

  on: Align­ment will hap­pen by de­fault. What’s next?

  I also don’t believe that insider trading is immoral...The reason it’s illegal is to protect retail investors

  That’s not the reason, at least according to Matt Levine and the SEC.

  Another reason insider trading is immoral is that it robs the company of proprietary information

  This is.

  That’s fair, but in that case doing it officially for The Company should be allowed, and it’s not.

  It depends on the merger agreement with LING—whether it allows either party to trade on this information. You’re ‘robbing’ proprietory information if this is disallowed. I’m unsure if merger agreements typically mention this, but if they don’t, then this is (legally)implicit.

  Every contract between 2 parties implicitly or explicitly includes whether you’re allowed to trade on information you gain from your relationship. Examples

  1. Employment contracts explicitly forbid employees from trading on material nonpublic information

  2. Customer relationships don’t—if you buy a product and discover it’s crap, you’re allowed to short the company. You’re allowed to buy Boeing puts while you’re in a plane which is currently crashing(unless their ticket policies prohibit this in the future).

  3. However if the customer is a large company and it’s a vendor relationship, the contract might forbid insider trading

  Companies are always allowed to trade on information they own/​produce—otherwise it would be illegal for companies to ever issue shares or do buybacks.

  Importantly, the law is about the information, not the stock of the company itself. It would have been completely legal for Pfizer and Moderna to trade hotel and airlines stocks before releasing their COVID vaccine trial results. This is ‘unfair’ to retail investors but that’s not what the law is about.

  This is(maybe?) also why the SEC wants jurisdiction over crypto, prediction markets and other securities—these are all avenues for robbing other companies of proprietory information.

• Shubhorup Biswas 26 Nov 2025 23:42 UTC

  1 point

  0

  in reply to: richbc’s comment on: On closed-door AI safety research

  Apologies for the late response as well.

  >For a more fine-grained example, actions like siphoning compute to run unauthorised tasks might be a signal that a model poses significantly higher catastrophic risk, but would also be something a commercial business would want to prevent for their own reasons
  
  Agreed.

  >Even assuming that the most greedy/​single-minded business leaders wouldn’t care about catastrophic risks on a global scale

  To clarify I didn’t meant that clients wouldn’t care about xrisk, but that they would not care about xrisk specifically from their vendor, or about the marginally increased x risk arising from using a certain unsafe frontier lab as a vendor(versus another). This marginally increased x risk is borne by all of humanity of which the client is a tiny slice, so they would undercare(versus the socially optimal amount of caring) about their marginal impact on this risk, as human extinction would be an externality. Similar to companies undercaring about GHG emissions.

  You propose that companies would use vendors because they advertise “10% less existential risk!”—this is similar to using a company because they promise to donate 10% of their revenues to climate charities as a USP. Except in this case frontier labs can share their safety research with each other for free.

  So even if the clients are perfectly altruistic, a frontier lab could reduce x risk by

  1. Doing AIS research

  2. Publishing AIS research

  So why would clients reward vendors for (1) but not (2)?

• Shubhorup Biswas 26 Nov 2025 17:46 UTC

  2 points

  0

  on: Aim for sin­gle piece flow

  A Tesla factory converts a pile of unassembled aluminum and some specialized parts into a ready-to-ride car in almost exactly 10 hours, all on a continuously moving assembly line that snakes itself through the Gigafactory.

  The Fremont Factory has 5 lines, 2 for Models S and X and 3 for Models 3 and Y.

  Horizontal scaling seems absolutely essential; there’s a limit to how fast you can move a single assembly line.

  And the speed at which a single line moves also affects how far it a product moves before faults are detected. In general,

  If at any point we discover a fault at any step in the assembly we immediately halt the line, preventing any quality issues from propagating to anything beyond the first faulty product.

  Fault introduction → detection time can be significant. For code changes, regression on user retention metrics may only show up after months or even years.

  I’ve seen ‘tighter feedback loops’ being promoted, but if our end goal is changing the trajectory of human civilisation, we would need proxies to mock out many sections of the feedback loop and these may be imperfect(we cannot even measure how imperfect they are relative to ground truth).