I’m Jérémy Perret. Based in France. PhD in AI (NLP). AI Safety & EA meetup organizer. Information sponge. Mostly lurking since 2014. Seeking more experience, and eventually a position, in AI safety/governance.
Extremely annoyed by the lack of an explorable framework for AI risk/benefits. Working on that.
Ah. Thank you for your attempt to get through to this community anyway, in the face of such incompatibility. Enjoy your freedom then, I hope you’ll do better than us.
Alright, the first chunk of my frowning was from claims about Rationality as a generic concept (and my immediate reaction to it). Second, I am puzzled by a few of your sentences.
Likewise, I consistently see Rationalists have no awareness or care of goals in the first place. Every human acts for a goal. If you don’t set an external one, then your default one becomes the goals motivated by human motivations systems.
What do you make of Goal Factoring, one of the techniques designed to patch that class of behaviors ? If I see a self-identified rationalist not being aware of their own goals, and there are a bunch, goal factoring would be my first suggestion. I would expect them to be curious about it.
If improving your ability to think by going through the uncomfortable process of utilizing a system of the brain that you are unfamiliar with is not something that interests you, then this document is not for you.
Mostly unnecessary caveat; one of the main draws of this website is to study the flaws of our own lenses.
Please be undeterred by the negative karma, it’s only a signal that this particular post may fail at its intended purpose. Namely:
I say all this to bring context to this document’s demand that the. reader does not ask for external justifications of claims. Instead, this document requires that readers test the concepts explored in this document in the real-world. It demands that the readers do not use validity-based reasoning to understand it.
...where is this document? Here I see a warning about the document, a surface clash of concepts, another warning of ignoring advice from other groups, and a bullet point list with too little guidance on how to get those heuristics understood.
Listing the virtues is a starting point, but one does not simply say “go forth and learn for yourself what Good Strategy is” and see that done without a lot of nudging, or else one might stay in the comfort of “validity-based reasoning” all call it a day. Which I would find disappointing.
“Internal betting markets” may be a reference to the Logical Induction paper? Unsure it ties strongly to stop-button/corrigibility.
Hi! Nearly all the statements in your question would benefit from some unpacking. Could you expand on what would count as coddling? My intuition says you’re gesturing at a whole heap of trade-offs, it might help to pick one in particular and study it further. Any proper answer to the question, as stated, is the stuff of entire books.
The post makes clear that two very different models of the world will lead to very different action steps, and the “average” of those steps isn’t what follows the average of probabilities. See how the previous sentence felt awkward and technical, compared to the story? Sure, it’s much longer, but the point gets across better, that’s the value. I have added this story to my collection of useful parables.
Re-reading it, the language remains technical, one needs to understand a bit more probability theory to get the latter parts. I would like to see a retelling of the story, same points, different style, to test if it speaks to a different audience.
I filled out the survey. Thank you so much for running this!
Oh, glad I scrolled to find this comment. Adding a request for France, which does have charity tax deductions… but needs an appropriate receipt.
Could you provide an example of prediction the Γ Framework makes which highlights the divergence between it and the Standard Model? Especially in cases the Standard Model falls short of describing reality well enough?
Best weekend of the year. Been there in 2017, 2018, 2019, 2023, will be delighted to attend again. Consistent source of excellent discussions, assorted activities, fun and snacks. Does indeed feel like home.
Welcome! One gateway for you might be the LW Concepts page about it!
Most of the posts discuss, of course, infohazard policy and properties of information that would be harmful to know, or think about. Directly sharing blatantly harmful information would be irresponsible.
My raw and mostly confused/snarky comments as I was going through the paper can be found here (third section).
Cleaner version: this is not a technical agenda. This is not something that would elicit interesting research questions from a technical alignment researcher. There are however interesting claims:
what a safe system ought to be like; it proposes three scales describing its reliability;
how far up the scales we should aim for at minimum;
how low on the scales currently large deployed models are.
While it positions a variety of technical agendas (mainly those of the co-authors) on the scales, the paper does not advocate for a particular approach, only the broad direction of “here are the properties we would like to have”. Uncharitably, it’s a reformulation of the problem.
The scales can be useful to compare the agenda that belong to the “let’s prove that the system adheres to this specification” family. It makes no claims over what the specification entails, nor failure modes of various (combinations of) levels.
I appreciate this paper as a gateway to the related agendas and relevant literature, but I’m not enthusiastic about it.
Typo: Mira Murati, not Mutari.
Here’s a spreadsheet version you can copy. Fill your answers in the “answers” tab, make your screenshot from the “view” tab.
I plan to add more functionality to this (especially comparison mode, as I collect some answers found on the Internet). You can now compare between recorded answers! Including yours, if you have filled them!
I will attempt to collect existing answers, from X and LW/EA comments.
Survey complete! I enjoyed the new questions, this should bring about some pretty graphs. Thank you for coordinating this.
I am producing videos in French (new batch in progress) on my main channel, Suboptimal.
But I also have a side channel in English, Suboptimal Voice, where I do readings of rationalist-sphere content. Some may appear in the future, I received requests for dramatic readings.
The Mindcrime tag might be relevant here! More specific than both concepts you mentioned, though. Which posts discussing them were you alluding to? Might be an opportunity to create an extra tag.
(also, yes, this in an Open Thread, your comment is in the right place)
Strongly upvoted for the clear write-up, thank you for that, and engagement with a potentially neglected issue.
Following your post I’d distinguish two issues:
(a) Lack of data privacy enabling a powerful future agent to target/manipulate you personally, because your data is just there for the taking, stored in not-so-well-protected databases, cross-reference is easier at higher capability levels, singling you out and fine-tuning a behavioral model on you in particular isn’t hard ;
(b) Lack of data privacy enabling a powerful future agent to build that generic behavioral model of humans from the thousands/millions of well-documented examples from people who aren’t particularly bothered by privacy, from the same databases as above, plus simply (semi-)public social media records.
From your deception examples we already have strong evidence that (b) is possible. LLM capabilities will get better, and it will get worse when [redacted plausible scenario because my infohazard policies are ringing].
In (b) comes to pass, I would argue that the marginal effort needed to prevent (a) would only be useful to prevent certain whole coordinated groups of people (who should already be infosec-aware) to be manipulated. Rephrased: there’s already a ton of epistemic failures all over the place but maybe there can be pockets of sanity linked to critical assets.
I may be missing something as well. Also seconding the Seed webtoon recommendation.
Quick review of the review, this could indeed make a very good top-level post.
Two separate points:
compared to physics, the field of alignment has a slow-changing set of questions (e.g. corrigibility, interpretability, control, goal robustness, etc.) but a fast-evolving subject matter, as capability progresses. I use the analogy of a biologist suddenly working on a place where evolution runs 1000x faster, some insights get stale very fast and it’s hard to know which ones in advance. Keeping up with the frontier is, then, used to know whether one’s work still seems relevant (or where to send newcomers). Agent foundations as a class of research agendas was the answer to this volatility, but progress is slow and the ground keeps shifting.
there is some effort to unify alignment research, or at least provide a textbook to get to the frontier. My prime example is the AI Safety Atlas, I would also consider the BlueDot courses as structure-building, AIsafety.info as giving some initial directions. There’s also a host of papers attempting to categorize the sub-problems but they’re not focused on tentative answers.