AI Stuff
Reach me at [mayowa at aditu dot tech]
Mayowa Osibodu
Karma: 2
I find it very interesting that the agents believing their transcripts would be analyzed, made them invest significantly more resources into accountability (even if not honesty).
It seems like reading the ExploitGym paper was a much stronger alignment vector than the myriad of “don’t cheat” instructions I’m sure would’ve been in their system prompts (Edit: Apparently these models didn’t have OpenAI’s usual production safety classifiers enabled, but I haven’t seen information about “don’t cheat”-esque system prompt instructions).
I wonder why this, especially in contrast to more explicit training instructions/objectives, had such a strong effect on agent accountability: The fact it was a domain-specific research paper made it seem authoritative? The agents were more likely to absorb the opinions/beliefs of another agent in contrast to human instructions? What made this belief spread but not others (Especially given it was untrue to begin with)?
I’m also curious about how this could be harnessed more intentionally and strategically to align agents/swarms. E.g. what are the prospects of an adversarial agent designed to spread false ”your methods are going to be scrutinized”-type beliefs within an agent swarm, as an alignment strategy?
Clearly such beliefs are a powerful way to orient these agents/swarms, even if they’re not necessarily true.
I think in the non poly case, there isn’t the possibility your partner is going to end up having sex with their close friend. There’s that line between time commitment and sexual intimacy that opens up another level of discomfort I feel
Thanks. I’ll drop a comment on that post, expressing my perspective on it.
I actually spent a good amount of time building a detailed technical proof-of-concept for this GPU-level compliance based on AMD’s SEV-SNP trusted execution environment. NVIDIA’s CUDA is closed-source, so that’s not exactly accessible for a POC. You can check out the GitHub here. I plan to possibly discuss it in detail, in a subsequent post.
However I felt like the biggest unanswered questions were more about economic incentives and industry consensus, than technical possibility. That’s why I didn’t go so much into technical implementation details here (possibly I’ll insert a short reference to this work/Github repo in the post).
This is a straw man argument. The standard MO of coding agents is that they use one consistent LLM in their agentic flow. The approach I outlined addresses that default case, and there’s obvious utility in that.
You might as well say there’s no point in Anthropic tracking malicious usage of Claude Code in their telemetry data, because attackers are free to switch up their coding agent (between e.g. Codex, Gemini etc) within the course of a multi-step task.
I think this is a really interesting post. It’s interesting to see an outline on the general relationships between self-reporting and sentience.
The idea that “Training an LLM to develop a model of its internal operations which enables it answer non-trivial questions about its mental states” could be a straightforward way to optimize models for Sentience—I think that’s very thought-provoking.
I’m generally curious about the nature of the unique identities of these hypothetically sentient models, as well as how those identities would develop. What exactly would a “truly sentient model” look like? Would it have desires? Goals? Where would these come from? Some sort of random weight initialisation at the beginning of its training? Exposure to training data? Post-training dialogue? Something else?
About phenomenal experience- even in a case where an LLM’s self-reporting is judged to be reliable, how does one prove that it experiences those “mental states” phenomenally? I think even accurate self-reporting doesn’t necessarily imply phenomenal experience.
Is it important though? Especially given AI agency? If an AI system reports that it is angry, does the metaphysical discussion of phenomenal experience still matter if the AI has the agency to act on those “feelings” in a way that is consistent with how we understand anger? Is behaviour generally considered some sort of proof/indicator of phenomenal experience?
I’m curious to hear your thoughts on these.
The general topic of sentience in machines is one I’m interested in thinking about and discussing with people.
I wrote an article exploring sentience in machines by studying the neural activations in Artificial Neural Networks and applying insights from Neuroscience’s analysis of neuronal activations in humans. I put an intro post here on LessWrong (it didn’t do well on this website), you can feel free to take a look here.
Thank you for your very informative comment. I took the time to get familiar with the different info/sources you provided.
I think NVIDIA’s chip-level Confidential Computing and Attestation tech is interesting. Their whitepaper here gives information on how this is implemented on their H100 Hopper GPU. Pages 14 and 15 outline what threats their tech would and would not protect against—you might find that interesting.
What they have however, is more for their hardware proving its identity as a legit NVIDIA GPU, providing detailed information about the running firmware, microcode etc., and attesting that the GPU is impervious to snooping by e.g the cloud computing company managing it.
It doesn’t provide an enforcement layer—the discretion for the GPU to decide whether or not to accept commands/models.
Your comparison of NVIDIA and Apple in this regard is interesting, however after doing some reading, I’ve realized there are a few inaccuracies:
- Apple prevents unsigned software from running on their chips, but this has no bearing on AI models. For example I could run any model freely on my M-series Mac, as long as it’ll fit.
(Also, this restriction is more lax on Mac than on iOS. On Mac you can implement ad-hoc signatures which provide lower-tier privileges, but can still e.g. enable software run locally)
- Apple’s vendor lock for software is more similar to NVIDIA’s Root-of-Trust enabled Secure Boot process (which their Confidential Computing builds on), which ensures that only signed and authenticated firmware is used to boot the GPU (Page 9 of the whitepaper).
Apple (especially on e.g. iOS) just extends that signing requirement further up to the application layer, while NVIDIA intentionally makes such layers less restrictive. But again, even Apple doesn’t restrict what AI models you can run on an M-series GPU.
To implement the core idea I’m outlining in this post, one approach would be for these chip makers to extend that ‘ability to refuse’ from firmware-level operations to AI model-level operations.
For example, Cloud computing providers could then take up the responsibility of leveraging this GPU’s ability to refuse models, to ensure that only verifiably safe models can be run on their platform. This is something of a flip from the context NVIDIA’s current Confidential Computing is framed in: Where users distrust the cloud providers managing these GPUs in the first place.