I’m Jérémy Perret. Based in France. PhD in AI (NLP). AI Safety & EA meetup organizer. Information sponge. Mostly lurking since 2014. Seeking more experience, and eventually a position, in AI safety/governance.
Extremely annoyed by the lack of an explorable framework for AI risk/benefits. Working on that.
I am confused by the problem statement. What you’re asking for is a generic tool, something that doesn’t need information about the world to be created, but that I can then feed information about the real world and it will become very useful.
My problem is that the real world is rich, and feeding the tool with all relevant information will be expensive, and the more complicated the math problem is, the more safety issues you get.
I cannot rely on “don’t worry if the Task AI is not aligned, we’ll just feed it harmless problems”, the risk comes from what the AI will do to get to the solution. If the problem is hard and you want to defer the search to a tool powerful enough that you have to choose carefully your inputs or catastrophe happens, you don’t want to build that tool.
“Knowledge,” said the Alchemist, “is harder to transmit than anyone appreciates. One can write down the structure of a certain arch, or the tactical considerations behind a certain strategy. But above those are higher skills, skills we cannot name or appreciate. Caesar could glance at a battlefield and know precisely which lines were reliable and which were about to break. Vitruvius could see a great basilica in his mind’s eye, every wall and column snapping into place. We call this wisdom. It is not unteachable, but neither can it be taught. Do you understand?”
Quoted from Ars Longa, Vita Brevis.
I second Charlie Steiner’s questions, and add my own: why collaboration? A nice property of an (aligned) AGI would be that we could defer activities to it… I would even say that the full extent of “do what we want” at superhuman level would encompass pretty much everything we care about (assuming, again, alignment).
Hi! Thank you for writing this and suggesting solutions. I have a number of points to discuss. Apologies in advance for all the references to Arbital, it’s a really nice resource.
The AI will hack the system and produce outputs that it’s not theoretically meant to be able to produce at all.
In the first paragraphs following this, you describe this first kind of misalignment as an engineering problem, where you try to guarantee that the instructions that are run on the hardware correspond exactly to the code you are running; being robust from hardware tampering.
I argue that this is actually a subset of the second kind of misalignment. You may have solved the initial engineering problem that at the start the hardware does what the software says, but the agent’s own hardware is part of the world, and so can plausibly be influenced by whatever the agent outputs.
You can attempt to specifically bar the agent from taking actions that target its hardware; that is not a hardware problem, but your second kind of misalignment. For any sufficiently advanced agent, which may find cleverer strategies than the cleverest hacker, no hardware is safe.
Plus, the agent’s hardware may have more parts than you expect as long as it can interact with the outside world. We still have a long way to go before being confident about that part.
Of course the problem with this oracle is that it’s far too inefficient. On every single run we can get at most 1 bit of information, but for that one bit of information we’re running a superhuman artificial intelligence. By the time it becomes practical, ordinary superhuman AIs will have been developed by someone else and destroyed the world.
There are other problems, for instance how can you be sure that the agent hasn’t figured out how to game the automated theorem prover to validate its proofs. You conclusion seems to be that if we manage to make safe enough, it will become impractical enough. But if you try to get more than one bit of information, you run into other issues.
This satisfies our second requirement—we can verify the AIs solution, so we can tell if it’s lying. There’s also some FNP problems which satisfy the first requirement—there’s only one right answer. For example, finding the prime factors of an integer.
Here the verification process is no longer an automated process, it’s us. You correctly point out that most useful problems have various possible solutions, and the more information we feed the agent, the more likely it will be able to find some solution that exploit our flaws and… start a nuclear war, in your example.
I am confused by your setup, which seems to be trying to make it harder for the agent to harm us, when it shouldn’t even be trying to harm us in the first place.
In other words, I’m asking: is there a hidden assumption that, in the process of solving FNP problems, the agent will need to explore dangerous plans?
A superintelligent FNP problem solver would be a huge boon towards building AIs that provably had properties which are useful for alignment. Maybe it’s possible to reduce the question “build an aligned AI” to an FNP problem, and even if not, some sub-parts of that problem definitely should be reducible.
I would say that building a safe superintelligent FNP solver requires solving AI alignment in the first place. A less powerful FNP solver could maybe help with sub-parts of the problem… which ones?
I’m taking the liberty of pointing to Adam’s DBLP page.
All my hopes for this new subscription model! The use of NFTs for posts will, without a doubt, ensure that quality writing remains forever in the Blockchain (it’s like the Cloud, but with better structure). Typos included.
Is there a plan to invest in old posts’ NFTs that will be minted from the archive? I figure Habryka already holds them all, and selling vintage Sequences NFT to the highest bidder could be a nice addition to LessWrong’s finances (imagine the added value of having a complete set of posts!)
Also, in the event that this model doesn’t pan out, will the exclusive posts be released for free? It would be an excruciating loss for the community to have those insights sealed off.
My familiarity with the topic gives me enough confidence to join this challenge!
Write down your own criticism so it no longer feels fresh
Have your criticism read aloud to you by someone else
Argue back to this criticism
Write down your counter-arguments so they stick
Document your own progress
Get testimonials and references even when you don’t “need” them
Praise the competence of other people without adding self-deprecation
Same as above but in their vicinity so they’ll feel compelled to praise you back
Teach the basics of your field to newcomers
Teach the basics of your field to experts from other fields
Write down the basics of your field, for yourself
Ask someone else to make your beverage of choice
Ask them to tell you “you deserve it” when they’re giving it to you
If your instinct is to reply “no I don’t”, consider swapping the roles
Drink your beverage, because it feels nice
Build stuff that cannot possibly be built by chance alone
Stare outside the window, wondering if anybody cares about you
Consider a world where everyone is as insecure as you
Ask friends about their insecurities
Consider you’re too stupid to drink a glass of water, then drink some water
Meditate on the difference between map and territory
Write instructions for the non-impostor version of you
Write instructions for whoever replaces you when people find out you’re an impostor
Validate those instructions with other experts, passing it off as project planning
Follow the instructions to keep the masquerade on
Refine the instructions since they’re “obviously” not perfect
Publish the whole thing here, get loads of karma
Document everything you don’t know for reference
Publish the thing as a list of open problems
Criticize harshly other people’s work to see how they take it
Make amends by letting them criticize you
Use all this bitterness to create a legendary academic rivalry
Consider “impostor” as a cheap rhetorical attack that doesn’t hold up
Become very good at explaining why other people are better than you
Publish the whole thing as in-depth reporting of the life of scientists
Focus on your deadline, time doesn’t care if you’re an impostor or not
Make yourself lunch, balance on one foot, solve a sudoku puzzle
Meditate on the fact you actually can do several complex things well
Consider that competence is not about knowing exactly how one does things
Have motivational pictures near you and argue how they don’t apply to you
Consider the absurdity of arguing with pictures
Do interesting things instead, not because you have to, but to evade the absurdity
Practice the “I have no idea what I’m doing, but no one does” stance
Ask people why they think they know how they do things
If they start experimenting impostor syndrome as well, support them
Join a club of impostors, to learn from better impostors than you
Write an apology letter to everyone you think you’ve duped
Simulate the outrage of anyone reading this letter
Cut ties with everyone who would actually treat you badly after reading
Sleep well, eat well, exercise, brush your teeth, take care of yourself
I hope this makes the case at least somewhat that these events are important, even if you don’t care at all about the specific politics involved.
I would argue that the specific politics inherent in these events are exactly why I don’t want to approach them. From the outside, the mix of corporate politics, reputation management, culture war (even the boring part), all of which belong in the giant near-opaque system that is Google, is a distraction from the underlying (indeed important) AI governance problems.
For that particular series of events, I already got all the governance-relevant information I needed from the paper that apparently made the dominoes fall. I don’t want my attention to get caught in the whirlwind. It’s too messy (and still is after months). It’s too shiny. It’s not tractable for me. It would be an opportunity cost. So I take a deep breath and avert my eyes.
My gratitude for the already posted suggestions (keep them coming!) - I’m looking forward to work on the reviews. My personal motivation resonates a lot with the help people navigate the field part; in-depth reviews are a precious resource for this task.
This is one of the rare times I can in good faith use the prefix “as a parent...”, so thank you for the opportunity.
So, as a parent, lots of good ideas here. Some I couldn’t implement in time, some that are very dependent on living conditions (finding space for the trampoline is a bit difficult at the moment), some that are nice reminders (swamp water, bad indeed), some that are too early (because they can’t read yet)...
… but most importantly, some that genuinely blindsided me, because I found myself agreeing with them, and they were outside my thought process! The one-Brilliant-problem a day one, the let-them-eat-more-cookies, mainly.
I appreciate, in particular, the breadth of the ideas. Thanks for sharing, even if you don’t practice what you preach, you’ll be able to get feedback.
After several nudges (which I’m grateful for, in hindsight), my votes are in.
This is very nice. I subscribed for the upcoming parts (there will be, I suppose?)
I think not mixing up the referents is the hard part. One can properly learn from fictional territory when they can clearly see in which ways it’s a good representation of reality, and where it’s not.
I may learn from an action movie the value of grit and what it feels like to have principles, but I wouldn’t trust them on gun safety or CPR.
It’s not common for fiction to be self-consistent enough and preserve drama. Acceptable breaks from reality will happen, and sure, sometimes you may have a hard SF universe were the alternate reality is very lawful and the plot arises from the logical consequences of these laws (often happens in rationalfic), but more often than not things happen “because it serves the plot”.
My point is, yes, I agree, one should be confused only by lack of self-consistency fiction or not. Yet, given the vast amount of fiction that is set in something close to real Earth, by the time you’re skilled enough to tell apart what’s transferable and what isn’t, you’ve already done most of the learning.
Not counting the meta-skill of detecting inconsistencies, which is indeed extremely useful, for fiction or not, but I’m still unclear where exactly one learns it from.
Thank you for this clear and well-argued piece.
From my reading, I consider three main features of AWSs in order to evaluate the risk they present:
arms race avoidance: I agree that the proliferation of AWSs is a good test bed for international coordination on safety, which extends to the widespread implementation of safe powerful AI systems in general. I’d say this extends to AGI, were we would need all (or at least the first, or only some, depending on takeoff speeds) such deployed systems to conform to safety standards.
leverage: I agree that AWSs would have much greater damage/casualties per cost, or per human operator. I have a question regarding persistent autonomous weapons which, much like landmines, do not require human operators at all once deployed: what, in that case, would be the limiting component of their operation? Ammo, energy supply?
value alignment: the relevance of this AI safety problem to the discussion would depend, in my opinion, on what exactly is included in the OODA loop of AWSs. Would weapon systems have ways to act in ways that enable their continued operation without frequent human input? Would they have other ways than weapons to influence their environment? If they don’t, is the worst-case damage they can do capped at the destruction capabilities they have at launch?
I would be interested by a further investigation on the risk brought by various kinds of autonomy, expected time between human command and impact, etc.
To clarify the question, would a good distiller be one (or more) of:
a good textbook writer? or state-of-the-art review writer?
a good blog post writer on a particular academic topic?
a good science communicator or teacher, through books, videos, tweets, whatever?
Based on the level of articles in Distill I wouldn’t expect producers of introductory material to fit your definition, but if advanced material counts, I’d nominate Adrian Colyer for Computer Science (I’ll put this in a proper answer with extra names based on your reply).
So, assuming an unaligned agent here.
If your agent isn’t aware that its compute cycles are limited (i.e. the compute constraint is part of the math problem), then you have three cases: (1a) the agent doesn’t hit the limit with its standard search, you’re in luck; (1b) the problem is difficult enough that the agent runs its standards search but fails to find a solution in the allocated cycles, so it always fails, but safely. (1c) you tweak the agent to be more compute-efficient, which is very costly and might not work, in practice if you’re in case 1b and it apparently fails safely you have an incentive to just increase the limit.
If your agent is indeed aware of the constraint, then it has an incentive to remove it, or increase the limit by other means. Three cases here again: (2a) identical to 1a, you’re in luck; (2b) the limit is low enough that strategic action to remove the constraint is impossible, the agent fails “safely”; (3b) the agent finds a way to remove the constraint, and you’re in very unsafe territory.
Two observations from there: first, ideally you’d want your agent to operate safely even if given unbounded cycles, that’s the Omni Test. Second, there’s indeed an alignment concept for agents that just try to solve the problem without long-term planning, that’s Myopia (and defining it formally is… hard).