Shubhorup Biswas

Karma: 181

Former Software Engineer, now working on AI Safety.

Questions I’m undecided on which on resolution could shift my focus, ranked by decreasing magnitude of shift:

Should I be a solipsist? If no other sentiences exist then morality is moot.
Are human values worth preserving? Would it be so (morally) bad if our lightcone is paved with a (misaligned) superintelligent AI’s values instead of human values?
Could technical alignment research be exploited by a TAI to align a superintelligence to its values?
1. If the technical alignment problem is unsolvable, would this preclude an intelligence explosion as a TAI would have no way to align superintelligence to its values
2. If the technical alignment problem is solvable and we do partially solve it, would it promote a misaligned AGI’s takeover by making it easier for it to align AIs smarter than it?
What’s the appropriate level of alarm to detecting misbehaviour in AIs(via CoT monitoring, interpretability or any other technique)? Almost everything we do(directly train against visible misbehaviour, discard the model and start afresh) creates selection pressure towards being stealthy. It would be good to quantify the number of bits of optimisation pressure that gets us into different regimes(do not misbehave, or misbehave covertly), and how many we have already expended.

Shubhorup Biswas 6 May 2026 23:59 UTC
1 point
0
in reply to: RHollerith’s comment on: AI unemployment and AI extinction are often the same
Full control of ASI by a few model developers is IMO the main prosaic worry. The crux(between extinction and prosaic risks) is whether alignment is hard.

> (if we successfully built AI that cared about us in the right way)
I’m unsure if this means cares about human flourishing/agency or fully obedient to its developer, but we only need the latter.

Shubhorup Biswas 30 Apr 2026 23:51 UTC
1 point
0
in reply to: anaguma’s comment on: Cyborg evals
These are not professional SWEs. Professional SWE tasks look very different competitive programming problems and also it’s hard to create competitive programming tasks that take 8h-30h.

Shubhorup Biswas 3 Apr 2026 3:30 UTC
1 point
0
on: My cost-effectiveness unit
such AI seems worse than the aliens that would otherwise claim a large fraction of the lightcone.
Any moral intuitions for why so? It seems to me that alien civilisations could also suffer loss of control or gradual disempowerment to AI, or any spectrum of scenarios between these and retaining ‘agency’ whatever that means. Do biologically evolved lifeforms definitely have positive moral valence?

Shubhorup Biswas 2 Apr 2026 0:37 UTC
1 point
0
in reply to: Kongo Landwalker’s comment on: My hobby: running deranged surveys
Steelman—the hypothetical scenario is incongruous with their current world model, so it being true would require extreme changes in their world model leading to a very weird world which they might not want to live in.
E.g. How would the brain and memories function under immortality?
This may also mean that the question is underspecified and the respondent might want to seek further clarification, but the website just doesn’t allow this.

Shubhorup Biswas 21 Mar 2026 1:23 UTC
4 points
0
on: Null Results From An Orexin RCT
If sleep deprivation doesn’t actually make you dumber then patching it shouldn’t make you smarter. Would the cognitive battery still be valid?

Shubhorup Biswas 19 Mar 2026 3:19 UTC
1 point
0
on: [Error communicating with LW2 server]
Can language models preserve their own alignment?
Wouldn’t consider this an argument for; rather a project proposal to empirically test how much models remain “good”

Shubhorup Biswas 19 Mar 2026 3:01 UTC
1 point
0
on: [Error communicating with LW2 server]
Avoiding AI Deception: Lie Detectors can either Induce Honesty or Evasion
They call this scalable oversight but they both train and evaluate the lie detector probes on labeled DolusChat examples. I don’t get why they call it scalable.
Claude’s analysis
https://claude.ai/share/d659b385-f625-4b86-9eb5-f8ce1fea33e5

Shubhorup Biswas 19 Mar 2026 2:40 UTC
1 point
0
on: [Error communicating with LW2 server]
Training fails to elicit subtle reasoning in current language models
I think Sonnet 3.5 monitor/Sonnet 3.7 attacker was a narrow intelligence gap.

The paper itself says “as developers continue to scale RL reasoning compute, models may become more capable of subtle reasoning.”

I see this as a minor empirical result with unclear generalisability

Shubhorup Biswas 19 Mar 2026 1:58 UTC
1 point
0
on: [Error communicating with LW2 server]
which proxies to train against.
https://www.lesswrong.com/posts/G9HdpyREaCbFJjKu5/it-is-reasonable-to-research-how-to-use-model-internals-in?commentId=krg2jzDxXhei9vNLj

and Daniel Kokotajlo comment about preserving at least one output stream that isn’t optimised against(this could be activations, while doing cot+output monitoring)

Shubhorup Biswas 10 Mar 2026 23:54 UTC
1 point
0
in reply to: David Africa’s comment on: Daniel Tan’s Shortform
Unsure if this is implicit, but I’m excited about an additional affordance for diagnosis—having access to training checkpoints.
This would be especially useful for models which may acquire some alien drives during post-training, and by the end of it can competently hide it. At the earlier stages, the drive would not be strong enough for the model to develop the instrumental goal of hiding alien drives.

Shubhorup Biswas 4 Mar 2026 5:20 UTC
1 point
0
on: The persona selection model
- Post-training can be viewed as updating this distribution using training episodes as evidence. When training an AI assistant on an (input xx, output yy) pair, hypotheses that predict the Assistant would respond with yy to xx are upweighted; hypotheses that predict the opposite are downweighted.
- PSM does not rule out learning of new capabilities during post-training. For example, no persona learned during pre-training knows how to use Anthropic’s syntax for tool calling; that capability is learned during post-training. PSM explains this as the LLM learning that the Assistant knows how to use this syntax. The important thing is that the LLM still models the Assistant as being an enacted persona.
This is more than just shifting the distribution of personas learned via pretraining(unless this can be framed as shifting the probability of a tool-calling-assistant from 0 to nonzero).

Importantly, does PSM implicitly claim that personas do not shift in propensities during post-training? Why should this be so/what conceptual difference between propensities and capabilities might cause this?
2. IIUC PSM does not make strong predictions about what happens with a lot of post-training(the threshold I’m most interested in here is whatever amount of post-training the first AGI will have).
In what ways might you expect PSM to breakdown if we do a lot of RL on tiny models?
3. For the purpose of this post SFT is conceptually clubbed with postraining RL, but mechanistically, I see SFT as just pretraining but stronger(per token). I expect SFT to be able to do the same things as pretraining does, i.e. create new personas. Is this the sort of thing that you would term ‘non-persona agency’ arising from heavy amounts of post-training?

Shubhorup Biswas 27 Feb 2026 9:48 UTC
3 points
0
on: Dario Amodei – The Adolescence of Technology
My co-founders at Anthropic and I were among the first to document and track the “scaling laws” of AI systems—the observation that as we add more compute and training tasks, AI systems get predictably better at essentially every cognitive skill we are able to measure.

The scaling laws paper is only about perplexity, or next-token-prediction loss, not ‘every cognitive skill we are able to measure’. To my knowledge there wasn’t much measurement of compute vs cognitive skills in 2020. IMO we don’t have well-validated scaling laws for compute vs benchmark performance, and any that exist are not a straightforward extension of the 2020 paper.

EDIT: The reason I cared enough about this to comment is that the conflation of these two things launders authority in two ways
1. The predictability of perplexity vs compute in 2020 applied to present day benchmark performance where labs are much more secretive about training details
2. The novel contribution of Amodei et al to perplexity scaling, which people care much less about now, makes Dario seem prescient and uniquely qualified to comment on benchmark performance, which everyone cares about in 2026
tl;dr: he was among the first to document and track perplexity scaling. Benchmark scaling is a different thing that (1) he was not among the first to document and track and (2) is not as empirically robust with verifiable open source data.

[Paper] How does information access affect LLM monitors’ ability to detect sabotage?

Rauno Arike, Raja Moreno, RohanS, Shubhorup Biswas and Francis Rhys Ward

11 Feb 2026 21:25 UTC

26 points

0 comments6 min readLW link

Aether is hiring technical AI safety researchers

Rauno Arike, RohanS and Shubhorup Biswas

5 Jan 2026 22:27 UTC

22 points

0 comments2 min readLW link

Shubhorup Biswas 27 Nov 2025 15:02 UTC
1 point
0
on: Alignment will happen by default. What’s next?
I also don’t believe that insider trading is immoral...The reason it’s illegal is to protect retail investors
That’s not the reason, at least according to Matt Levine and the SEC.
Another reason insider trading is immoral is that it robs the company of proprietary information
This is.
That’s fair, but in that case doing it officially for The Company should be allowed, and it’s not.
It depends on the merger agreement with LING—whether it allows either party to trade on this information. You’re ‘robbing’ proprietory information if this is disallowed. I’m unsure if merger agreements typically mention this, but if they don’t, then this is (legally)implicit.
Every contract between 2 parties implicitly or explicitly includes whether you’re allowed to trade on information you gain from your relationship. Examples
1. Employment contracts explicitly forbid employees from trading on material nonpublic information
2. Customer relationships don’t—if you buy a product and discover it’s crap, you’re allowed to short the company. You’re allowed to buy Boeing puts while you’re in a plane which is currently crashing(unless their ticket policies prohibit this in the future).
3. However if the customer is a large company and it’s a vendor relationship, the contract might forbid insider trading
Companies are always allowed to trade on information they own/produce—otherwise it would be illegal for companies to ever issue shares or do buybacks.
Importantly, the law is about the information, not the stock of the company itself. It would have been completely legal for Pfizer and Moderna to trade hotel and airlines stocks before releasing their COVID vaccine trial results. This is ‘unfair’ to retail investors but that’s not what the law is about.
This is(maybe?) also why the SEC wants jurisdiction over crypto, prediction markets and other securities—these are all avenues for robbing other companies of proprietory information.

Shubhorup Biswas 26 Nov 2025 23:42 UTC
1 point
0
in reply to: richbc’s comment on: On closed-door AI safety research
Apologies for the late response as well.
>For a more fine-grained example, actions like siphoning compute to run unauthorised tasks might be a signal that a model poses significantly higher catastrophic risk, but would also be something a commercial business would want to prevent for their own reasons

Agreed.
>Even assuming that the most greedy/single-minded business leaders wouldn’t care about catastrophic risks on a global scale
To clarify I didn’t meant that clients wouldn’t care about xrisk, but that they would not care about xrisk specifically from their vendor, or about the marginally increased x risk arising from using a certain unsafe frontier lab as a vendor(versus another). This marginally increased x risk is borne by all of humanity of which the client is a tiny slice, so they would undercare(versus the socially optimal amount of caring) about their marginal impact on this risk, as human extinction would be an externality. Similar to companies undercaring about GHG emissions.
You propose that companies would use vendors because they advertise “10% less existential risk!”—this is similar to using a company because they promise to donate 10% of their revenues to climate charities as a USP. Except in this case frontier labs can share their safety research with each other for free.
So even if the clients are perfectly altruistic, a frontier lab could reduce x risk by
1. Doing AIS research
2. Publishing AIS research
So why would clients reward vendors for (1) but not (2)?

Shubhorup Biswas 26 Nov 2025 17:46 UTC
2 points
0
on: Aim for single piece flow
A Tesla factory converts a pile of unassembled aluminum and some specialized parts into a ready-to-ride car in almost exactly 10 hours, all on a continuously moving assembly line that snakes itself through the Gigafactory.
The Fremont Factory has 5 lines, 2 for Models S and X and 3 for Models 3 and Y.
Horizontal scaling seems absolutely essential; there’s a limit to how fast you can move a single assembly line.
And the speed at which a single line moves also affects how far it a product moves before faults are detected. In general,
If at any point we discover a fault at any step in the assembly we immediately halt the line, preventing any quality issues from propagating to anything beyond the first faulty product.
Fault introduction → detection time can be significant. For code changes, regression on user retention metrics may only show up after months or even years.
I’ve seen ‘tighter feedback loops’ being promoted, but if our end goal is changing the trajectory of human civilisation, we would need proxies to mock out many sections of the feedback loop and these may be imperfect(we cannot even measure how imperfect they are relative to ground truth).

Shubhorup Biswas 11 Sep 2025 15:50 UTC
1 point
0
on: On closed-door AI safety research
″ Especially for corporate customers, safety and reliability are important unique selling points,

Only very prosaic, not catastrophic risks(as a customer I would not care at all about likelihood of catastrophic risks from my vendor—that’s something that affects humanity regardless of customer relationships).

Only if a particular secret is useful for preventing both catastrophes and prosaic ‘safety failures’ would this be a consideration for us—catastrophic risks increasing due to companies trying to have a competitive edge in prosaic safety risks