Program Coordinator of AI Safety Camp.
Remmelt
Karma: 201
Ah, adding this here:
I personally do not tend to think of AISC as converting money and interested people into useful research. For me, that conjures up the image of a scaleable machine where you can throw in more inputs to spit out more of the output.I view AISC more as designing processes together that tend toward better outcomes (which we can guess at but do not know about beforehand!).
Or as a journey of individual and shared exploration that people – specifically aspiring researchers– go through who are committed to ensuring unsafe AI does not destroy human society and all of biological life.
I am the program coordinator of AI Safety Camp. Let me respond with personal impressions / thoughts:
Apologies, Charlie, that we did not get to call before you wrote this post. Busy months for me, and I had misinterpreted your request as you broadly reaching out to interview organisers of various programs.
First, respect for the thoroughness and consideration of your writing:
It is useful to get an outside perspective of how AI Safety Camp works for participants.
In this sense, I am glad that we as organisers did not get to talk with you yet, which might have ‘anchored’ this post more on our notions of what the camp is about.
Hoping that you and I can still schedule a reverse interview, where I can listen to and learn from your ideas!
Noting that we also welcome honest criticism of AI Safety Camp that could help us rethink or improve the format and/or the way we coordinate editions.
I would personally value if someone could do background research at least half as well as Charlie and play devil’s advocate: come up with arguments against AISC’s current design or ‘set parameters’ being any good for helping certain (potential) participants to contribute to AI existential safety.
Write a quality post and it will get my strong upvote at least!
Glad to have your ideas on parameters to tweak and what to consider focussing on doing well so we can serve new participants better (to come to contribute at the frontiers of preventing the existential risk posed by AI developments).
For example, you made me think that maybe the virtual edition could be adapted to cater for remote ML engineering teams in particular.
Where conceptual research in a group setting may just tend to work better through spontaneous chats and flip-chart scribbles at a physical retreat.
I find myself broadly agreeing with most of your descriptions of whom the camp is for and how we serve our participants.
On ways the camp serves participants looking to contribute to AI x-safety research:35% about testing fit, 30% about signalling, and 15% about object-level work, plus different leftovers.
The relative weighting above matches my impressions, at least for past editions (AISC 1-5).
Having said that, making connections with other aspiring researchers (fellow participants, organisers, speakers, research authors) mattered a lot for some alumni’s career trajectories.
I am not sure how to even introduce a separate weight for ‘networking’ given the overlap with ‘signalling’ and ‘testing fit’ and leftovers like ‘heard about a grant option; started an org’.
BTW your descriptions related to ‘testing fit’ resonated with me!
> What was valuable to them was often what they learned about themselves, rather than about AI....
> Some people attended AISC and decided that alignment research wasn’t for them, which is a success in its own way. On average, I think attending made AI alignment research feel “more real,” and increased peoples’ conviction that they could contribute to it. Several people I talked to came away with ideas only tangentially related to their project that they were excited to work on—but of course it’s hard to separate this from the fact that AISC participants are already selected for being on a trajectory of increasing involvement in AI safety.
Also, in the ‘leftovers’ bucket, there is a lot of potential for tail events – where people’s experiences at the camp either strongly benefit or strongly harm their future collaborations on research for preventing technology-induced existential risks.
For example:Benefits: Research into historically overlooked angles (e.g. alignment impossibility arguments, human biology-based alignment) sparks new insights and reflections that shift the paradigm within which next generation AI x-safety researchers conduct their research.
Harms: We serve alcohol at a fun end-of-camp party, fail at monitoring and checking in with participants, and then someone really ends up ignoring another person’s needs and/or crosses their personal boundaries.
Finally, I would make an ends-means distinction here:
I would agree that at past formats, the value object-level work during the program seemed proportionally smaller than the value of testing fit and networking.
At the same time, I and other past organisers believe that individual participants actually trying to do well-considered rigorous research together helps a lot with them making non-superficial non-fluky traction toward actually contributing at the frontiers of the community’s research efforts.
For future physical editions (in Europe, US, and Asia-Pacific):
I would guess that...
signalling (however we define this and its benefits/downsides) is held about constant.
object-level work (incl. deconfusing meta-level questions) and testing fit (incl. for working on a new research probles, if already decided on a research career) swap weights.
I.e. 35% about object-level work, 30% about signalling, 15% about testing fit, plus different leftovers (leaving aside ‘networking’ and ‘camp tail events’).
Note that edition formats have been changing over time (as you mentioned yourself):
The first camp was a rather grassroots format where participants already quite knowledgeable / connected / experienced in AI safety research could submit their proposals and gather into teams around a chosen research topic.
At later editions, we admitted participants who had spent less time considering what research problems to work on, and we managed to connect at most a few teams per camp with a fitting mentor (mostly, we provided a list of willing mentors a team could reach out to after the team already had decided on a research topic).
At the sixth edition and current edition, we finally rearranged and refined the format to serve our mentors better. The current virtual edition involved people applying to collaborate on an open research problem picked by one of our mentors (progress of the teams so far are mixed, but based on recent feedback about 3 mentors were a little negative, and 3 others were somewhat to very positive about having mentored a team).
The next physical edition in Europe will be about creating ~6 research spaces for some individual past participants and reviewers – who are somewhat experienced at facilitating research – to invite over independent researchers to dig together into an arguably underexplored area of research (on the continuum you mentioned, AISC7 is nearer to the end of a research lab).
On your points re: parameters of AISC that make it good at some things and not others:
Length and length variability: Naturally shorter time mandates easier projects, but you can have easy projects across a wide variety of sub-fields. However, a fixed length (if somewhat short) also mandates lower-variance projects, which discourages the inherent flailing around of conceptual work and is better suited to projects that look more like engineering.
Current camp durations:
For the yearly virtual edition (early Jan to early June), the period is roughly 5 months – from initial onboarding and discussions to research presentations and planning post-camp steps.
For the upcoming physical edition, the period is roughly 2 months (Sep to Oct).
Some/all future editions of AISC may specialise in enabling research in less-explored areas and based on less-explored paradigms (meaning higher variance in the value of the projects’ outputs).
In which case, the length and/or intensity of research (in terms of hours per week, one-on-one interactions) at editions will go up.
Level of mentor involvement: …The more interesting arguments against increasing supervision are that it might not reduce length variability pressure by much (mentors might have ideas that are both variable between-ideas and that require an uncertain amount of time to accomplish, similar to the participants),
This seems more likely than not to me (holding constant how ‘exploratory’ the area of research is).
In part because half or more of the teams at the current virtual edition ended up exploring angles that were different than mentors had planned for.
… and might not increase the total object-level output, relative to the mentor and participants working on different topics on the margin.
This is definitely the case in the short run (i.e. over the 5-month period of the virtual edition).
I feel very unsure when it comes to the long run (i.e. ‘total object-level output’ over the decades following the participants’ collaboration with a mentor at an edition).
Overall, I guess the current average degree of mentor involvement is >10% more likely to increase ‘total output’ than not.
Where the reference of comparison is quick mentor feedback on the team’s initial proposal and any later draft on their research findings.
Where average output on the upside (‘increase in total’) is higher than it is lower on the downside (‘decrease in total’) when working from established alignment research paradigms.
Along with a decrease in the likelihood that team outputs lead to any important paradigmatic changes in AI x-safety research.
Also need to account for that a mentor will occasionally meet a capable independent-minded researcher at the camp and afterwards continue to collaborate with them individually (this seems probably the case for ~2 participants at the current virtual edition).
Evaluation: Should AISC be grading people or giving out limited awards to individuals?
In the early days of AISC, we discussed whether to try to evaluate participant performance during the camp so we could recommend and refer alumni to senior researchers in the community.
We decided against internal evaluations because that could break apart the collaborative culture at the camp.
Basically it could leave people feeling discomfortable about ‘getting watched’, and encourage some individuals to compete with each other to display ‘their work’ (also who am I kidding: for organisers to manage evaluation of performance on this variety of open problems?).
Nuances on a few of your considerations:
AISC doesn’t need to be in the business of educating newbies, because it’s full of people who’ve already spent a year or three considering AI alignment and want to try something more serious.
There are interesting trade-offs between ‘get newcomers up to speed’ vs. ‘foster cognitive diversity’:
There are indeed more aspiring contributors now who spent multiple years considering work in AI existential safety. Also, there are now other programs specialising in bringing students and others up to speed with AI concepts and considerations, like AGI Safety Fundamentals and MLSS (and to a lesser extent, the top-university-named ‘existential risk initiative’ programs).
So agreed there that AISC does not have a comparative advantage in educating newcomers there, and also that this ‘part of the pipeline’ is no longer a key bottleneck.
We never have been in the business of educating people (despite mention of ‘teach’ on our website, which I’ve been thinking of rewriting). Rather, people self-study, apply and do work on their own initiative.
In this sense, AISC can offer people who self-studied or e.g. completed AGISF a way to get their hands dirty on a project at the yearly virtual camp (and from there go on to contribute to a next AISC edition, say, or apply for SERI MATS).
On the other hand, my sense is that most of the people in the crowd you mentioned share roughly similar backgrounds (in terms of STEM disciplines and in being WEIRD: Western, Educated, from Industrialised & Democratic nations).
Many of the aspiring AI x-safety researchers to me appear to broadly share similar inclinations in how they analyse and perceive problems in the world around them (with corresponding blindspots – I tried to summarise some here).
The relatively homogenous reasoning culture of our community is concerning in the sense that where the AI x-safety community collectively shares the same blindspots (reflected in forum ‘Schelling points’ of topics and discussions), individuals participating will tend to overlook any crucial considerations there (in those blindspots) that are relevant for us to help prevent AI developments from destroying human society and all of biological life.
We as organisers look to reach out to and serve individual persons who can bring in their diverse research disciplines, skills and perspectives. We are more accommodating here in terms of how much time such diverse applicants have spent upfront reading about and engaging with AI existential safety research (given that they would have heard less about our community’s work), and try where we can to assist persons individually with getting up to speed.
Here, your suggested ‘program fit’ guideline definitely applies:
> You know at least a bit about the alignment problem—at the very least you are aware that many obvious ways to try to get what we want from AI do not actually work.
This isn’t really a fixed group of people, either—new people enter by getting interested in AI safety and learning about AI, and leave when they no longer get much benefit from the fit-testing or signalling in AISC. I would guess this population leaves room for ~1 exact copy of AISC (on an offset schedule), or ~4 more programs that slightly tweak who they’re appealing to.
Potential participants need to distinguish programs and work out which would serve their needs better. We as organisers need to keep org scopes clear. So I am not excited about the ‘exact copy’ angle (of course, it would also get forced and unrealistic if someone tries to copy over the cultural nuances and the present organisers’ ways of relating with and serving participants).
I would be curious to explore ideas for new formats with anyone who noticed a gap in what AISC and other AIS research training programs do, and who is considering trying out a pilot for a new program that takes a complementary angle on the AI Safety Camp. Do message me!
Ah, found the other page, and I see it is already put under the category ‘Artificial Intelligence’, under the heading ‘Organizations’: https://www.lesswrong.com/tags/all
Thanks for the help getting this sorted, @Plex.
Do you mean as a sub-sub-tag? I think that would be good idea.
I have looked at the LessWrong tag manager, but still do not know how to do it. Any tips?
(if the idea is to merge it with the Organizations tag, I am biased of course, but there are enough posts tagged AI Safety Camp to warrant it being tagged as a distinguishable organisation)
Yeah, that points well to what I meant. I appreciate your generous intellectual effort here to paraphrase back!
Sorry about my initially vague and disagreeable comment (aimed at Adam, who I chat with sometimes as a colleague). I was worried about what looks like a default tendency in the AI existential safety community to start from the assumption that problems in alignment are solvable.
Adam has since clarified with me that although he had not written about it in the post, he is very much open to exploring impossibility arguments (and sent me a classic paper on impossibility proofs in distributed computing).
… making your community and (in this case) the wider world fragile to reality proving you wrong.
We don’t have any proofs that the approaches the referenced researchers are doomed to fail like we have for P!=NP and what you linked.
Besides looking for different angles or ways to solve alignment, or even for strong arguments/proofs why a particular technique will not solve alignment,
… it seems prudent to also look for whether you can prove embedded misalignment by contradiction (in terms of the inconsistency of the inherent logical relations between essential properties that would need to be defined as part of the concept of embedded/implemented/computed alignment).This is analogous to the situation Hilbert and others in the Vienna circle found themselves in trying to ‘solve for’ mathematical models being (presumably) both complete and consistent. Gödel, who was a semi-outsider, instead took the inverse route of proving by contradiction that a model cannot be simultaneously complete and consistent.
If you have an entire community operating under the assumption that a problem is solvable or at least resolving to solve the problem in the hope that it is solvable, it seems epistemically advisable to have at least a few oddballs attempting to prove that the problem is unsolvable.
Otherwise you end up skewing your entire ‘portfolio allocation’ of epistemic bets.
In the end you do want to solve the problem, obviously. But the road from here to there goes through many seemingly weird and insufficient ideas that are corrected, adapted, refined, often discarded except for a small bit. Alignment is no different, including “strong” alignment.
There is an implicit assumption here that is not covering all the possible outcomes of research progress.
With progress on understanding some open problems in mathematics and computer science, they have turned out unsolvable. That is a valuable, decision-relevant conclusion. It means it is better to do something else than keep hacking away at solving that maths problem.
E.g.
Solving for why mathematical models would be both consistent and complete (see https://youtu.be/HeQX2HjkcNo)
Solving for that any distributed data store can guarantee consistency, availability, and partition tolerance (see https://en.m.wikipedia.org/wiki/CAP_theorem)
Solving for 5 degree polynomials with radicals (see http://www.scientificlib.com/en/Mathematics/LX/AbelRuffiniTheorem.html).
We cannot just rely on a can-do attitude, as we can with starting a start-up (where even if there’s something fundamentally wrong about the idea, and it fails, only a few people’s lives are impacted hard).
With ‘solving for’ the alignment of generally intelligent and scalable/replicable machine algorithms, it is different.
This is the extinction of human society and all biological life we are talking about. We need to think this through rationally, and consider all outcomes of our research impartially.
I appreciate the emphasis on diverse conceptual approaches. Please, be careful in what you are looking for.
This seems to presume that a certain literal interpretation of that text is the only one that could be intended or interpreted. I don’t think this is worth discussing this further, so leaving it at that.
> Rationalists who detachedly and coolly model the external world
That’s basically the straw Vulcan accusation. We build pillow forts.I now see how the ‘who’ part of the sentence can come across as me saying that rationalists only detachedly and coolly modelling the external world. I do not think that is the case based on interacting with plenty of self-ascribed rationalists myself (including making a pillow fort and hanging out with them in it myself). I do think rationalists do this mental move a lot more than other people I know.
Instead, I meant this as an action rationalists choose to take more often.
I just edited that sentence to ‘When rationalists detachedly and coolly model...’
Started watching this talk by Bret Victor on representing code for humans. Interesting, thanks for the share
I would be careful here about ascribing some singular definitive personal motivation to why I’m sharing these opinions (in the vain of ‘he and that other guy he talked with are clouded by ideology and that’s why he jumped to this factually incorrect conclusion’ or ‘he didn’t announce the names of the authors of a draft so based on that one can conclude this person doesn’t prioritise transparency’). Particularly when you quickly spot something about my take to disagree with, and might only grasp a small portion of where I’m coming from. Better to first have an actual face-to-face conversation and listen, paraphrase, and check in on each other’s views along with the context needed to interpret them. I’m deliberately not characterising you here based on your comments. I try somewhat awkwardly to stay open to what I’m missing.
That having said, you’ve clearly read a lot more about Dominic Cummings’ work than I have. I appreciate the detailed remarks. They help me break up and reassemble the broad impressions I personally got from reading a few blogposts.
On getting input from focus groups from 70-80% of the population – is the focus here on soliciting and addressing commonly held or majority views that are ignored, or also on aggregating distinct minority views?
Brad Victor setup Dynamicland in a way that’s very intentional about not building technology average early tech adopter but for a wide variety of diverse people.
This sounds cool. Let me watch a video of his.
On setting up organisations that can manage themselves, I agree that this seems a major problem in the US government for instance (and also in e.g. the Dutch Ministry of Justice and Security, and Ministry of Foreign Affairs from the stories I’ve heard of gross misspending and budgets reallocated to political ends from insiders involved there). This EconTalk podcast suggested actually recruiting an experienced chief operating officer in the US government, or at least to oversee an effective central auditing department. I don’t have strong opinions about that one – only that the incentives and power dynamics that you’re enmeshed in within such a big bureaucracy seem really tricky to work with.
The main political fight is whether you want an enviroment where radically new departments (and that includes things like the one that Audrey Tang runs) are possible or not
Did you mean that this is what Dominic Cummings or perhaps you would see as the main political fight here? I wouldn’t define it as such. I guess part of the tension in our conversation comes from talking past each other about what you could see as two separate agendas.
Ah, I was vaguely under the impression indeed that Dominic’s article suggested for-profit tech start-ups would replace government departments. So thanks for correcting that impression.
To be clear though, that’s not core to what I’m disagreeing about here. My disagreement here is not about for-profit entities coming in and running operations more efficiently, it’s about insular monopolistic actors privileging their technical expertise for deciding how to built the systems that will hold sway over the rest of society, while attempting little real dialogue with the diverse persons whose lives they will impact.
IMO Taiwan’s g0v collaborative is an impressive example of technical innovation for amongst others resolving disinformation and tensions between the polar sides of technocratic/technocapatalistic top-down enforcement and populist bottom-up destabilisation (tricky not to get stuck in political idealogy when discussing those issues).
Just read these articles on g0v and vTaiwan. They’re kinda idealistically promotional, but still highly recommend them:
As a case in point, I would try comparing
Cumming’s conception of a seeing room, which from a casual glance seems to be about providing key government decision-makers with a more accurate and more processable overview for making a big decision.
with the way vTaiwan activists compile digestable information from responses to ‘rolling surveys’ amongst stakeholders and relevant experts, before inviting them to deliberate and build a rough consensus on stances online.
Let me google the Emprint Method. The idea of focus on past vs. future in rewarding/encouraging makes intuitive sense to me though.
I haven’t actually heard of Fauci or discussions around him, but appreciate the clarification! Note again, I’m talking about a way you perceive interdependence (not to point to the elements needed for two states to be objectively described as interdependent).
Just scanned through the post. FYI, I broadly disagree with this and a few other takes I read from Dominic Cummings when it comes to enforcing ambitious political changes based on strategies he presents as the rational choice viz a viz predicted scenarios. But don’t want to get into detail here.
Highlighting this point from Dominic’s summary:The Valley is the natural place to build the best model of the electorate and some weird subculture there is more likely than DC to look at the problem with the fresh eyes it needs. It’s also the natural place to think about how to have startups replace parts of the US government and the sort of policies to pursue if you can control the government.
In response, this excerpt encapsulates a more open/less insular approach to collaborating on innovation across society that I’m more excited about:The best we can do—as technologists, financiers, policymakers, and above all citizens—is to find ways to pursue innovation in an experimental and flexible manner, with a conscious focus on improving society as a whole. Without a spirit of collaboration, we may be limited by ideologies like technocapitalism, which promises inevitable progress and epistemic certainty while concentrating economic power, debasing public discourse, and failing to live up to its grand ideals.
Those were the last sentences from The Case Against Naive Technocapitalist Optimism. That article seems rather moderately and reasonably argued, and somewhat interesting (though doesn’t dig much specifically into how technocapitalism debases public discourse or fails to live up to its grand ideals; rather points to lack of fundamental research and selectively motivated applied research delivered by the industry, and sticks to well-travelled economic ideas and issues like income disparity).
re: Processes vs Structure
Your concrete examples made me update somewhat towards process thinking being more common in AI alignment and local rationality communities than I was giving credit for. I also appreciate you highlighting that the rationality community has a diversity of approaches, and that we’re not some homogenous blob (as my post might imply).
A CFAR staff member also pointed me to MIRI’s logical induction paper (with input from Critch) as one of MIRI’s most cited papers (i.e. at least somewhat representative of how outside people might view MIRI’s work) that’s clearly based on an algorithmic process.
Eliezer’s AI Foom post (the one I linked to above) can be read as an explanation of how an individual agent is constructed out of a reinitiating process.Also, there has been some interest in decision and allocation mechanisms like e.g. quadratic voting and funding (both promoted by RxC, latter also by Vitalik Buterin) which seems kinda deliberately process-oriented.
I would guess that within EA you find more people who can only handle the structure-based approach.
This also resonates, and I hadn’t explicitly made that distinction yet! Particularly, how EA researchers have traditionally represented cause areas/problems/interventions to work on (after going through e.g. Importance-Tractability-Neglectedness analysis) seems quite structure-based (80Ks methods for testing personal fit don’t as much however).
IMO CEA-hosted grantmakers also started from a place where they dissected and assessed the possible promising/unpromising traits of a person, the country or professional hub they operate from, and their project idea (based on first principles say, or track record). This was particularly the case with the EA Community Building Grant in early days. But seems to be changing somewhat in how EA Funds grantmakers are offering smaller grants in dialog with possible applicants, and assessing viability for the career aspirant to continue or for a project to expand further as they go.
I made a case before for funding my entrepreneurial endeavour that instead relied on funding processes that relied more directly on eliciting and acting on feedback. And expanded on that in a grant application to the EA Infrastructure Fund:Personally, I think my project work could be more robustly evaluated based on
- feedback on the extent to which my past projects enabled or hindered aspiring EAs.
- the care I take in soliciting feedback from users and in passing that on to strategy coaches and evaluating funders.
- how I use feedback from users and advisors to refine and prioritise next services.
At each stage, I want to improve processes for coordinating on making better decisions. Particularly, to couple feedback with funding better:
deserved recognition
actionable feedback ⇅ - ⇅commensurate funding
improved work
I’m not claiming btw that process-based representations are inherently superior for the social good or something. Just that the value of that kind of thinking is overlooked in some of the work we do.
E.g. In this 80K interview, they made a good case for why adhering to enacting some bureaucratic process can be bad. I also thought they overlooked a point – you can make other, similarly compelling arguments for why rewarding that some previously assessed outcome or end state came into existence or was reached can be bad.Robert Wiblin: You have to focus on outcomes, not process.
Brian Christian: Yeah. One of the examples that I give is my friend and collaborator, Tom Griffiths. When his daughter was really young, she had this toy brush and pan, and she swept up some stuff on the floor and put it in the trash. And he praised her, like “Oh, wow, good job. You swept that really well.” And the daughter was very proud. And then without missing a beat, she dumps the trash back out onto the floor in order to sweep it up a second time and get the same praise a second time. And so Tom—
Robert Wiblin: Pure intelligence.
Brian Christian: Exactly. Yeah. Tom was—
Robert Wiblin: Should be very proud.
Brian Christian: —making the classic blunder of rewarding her actions rather than the state of the kitchen. So he should have praised how clean the floor was, rather than her sweeping itself. So again, there are these surprisingly deep parallels between humans and machines. Increasingly, people like Stuart Russell are making the argument that we just shouldn’t manually design rewards at all.
Robert Wiblin: Because we just have too bad a track record.
Brian Christian: Yes. It’s just that you can’t predict the possible loopholes that will be found. And I think generally that seems right.
Robert Wiblin: Yeah. I guess we’ll talk about some of the alternative architectures later on. With the example of a human child, it’s very visible what’s going wrong, and so you can usually fix it. But I guess the more perverse cases where it really sticks around is when you’re rewarding a process or an activity within an organisation, and it’s sufficiently big that it’s not entirely visible to any one person or within their power to fix the incentives, or you start rewarding people going through the motions of achieving some outcome rather than the outcome itself. Of course rewarding the outcome can be also very difficult if you can’t measure the outcome very well, so you can end up just stuck with not really having any nice solution for giving people the right motivation. But yeah, we see the same phenomenon in AI and in human life yet again.
re: Independent vs. Interdependent
I think generally we do a decent job at modeling how other people are constraint in the choices they make by their enviroment but model ourselves more as independent actors. But then I’m uncertain whether anybody really looks at the way their own decision making depends on other people.
Both resonate for me.
And seeing yourself as more independent than you see others does seem very human (or at least seems like what I’d do :P). Wondering though whether there’s any rigorous research on this in East-Asian cultures, given the different tendency for people living there to construe the personal and agentic self as more interdependent.One is about modeling yourself as independent and the other is about modeling other people as independent actors.
I like your distinction of viewing yourself vs. another as an independent agent.
Some other person-oriented ways of representing that didn’t make it into the post:Identifying an in(ter)dependent personal self with respect to the outside world.
Identifying an in(ter)dependent relation attributable to ‘you’ and/or to an identified social partner.
Identifying a collective of related people as in(ter) dependent with respect to the outside world.
and so on…
Worked those out using the perceptual framework I created, inspired by and roughly matching up with the categories of a paper that seems kinda interesting.
Returning to the simplified two-sided distinction I made above, my sense is still that you’re not capturing it.
There’s a nuanced difference between the way most of your examples are framed, and the framing I’m trying to convey. Struggling to, but here’s another attempt!
Your examples:When discussing for example FDA decisions we don’t see Fauci as a person who’s independent of the system in which he operates.
There’s the moral maze discourse which is also about individuals being strongly influenced by the system in which they are operating
But then I’m uncertain whether anybody really looks at the way their own decision making depends on other people.Each of those examples describes an individual agent as independent vs. ‘not independent’, i.e. dependent.
Dependence is unlike interdependence (in terms of what you subjectively perceive in the moment).
Interdependence involves holding both/all in mind at the same time, and representing how the enduring existence of each is conditional upon both themselves and the other/s. If that sounds wishy-washy and unintuitive, then hope you get my struggle to explain it.
You could sketch out a causal diagram, where one arrow shows the agent affecting the system, and a parallel arrow shows the system affecting the agent back. That translates as “A independently wills a change in S; S depends on A” in framing 1, “S independently causes a change in A; A depends on S” in framing 2.
Then, when you mentally situate framing 2 next to framing 1, that might look like you actually modelled the interdependence of the two parts.
That control loop seems deceptively like interdependence, but it’s not by the richer meaning I’m pointing to with the word. It’s what these speakers are trying to point out when they vaguely talk on about system complexity vs. holistic complexity.
Cartesian frames seem like a mathematically precise way of depicting interdependence. Though this also implicitly imports an assumption of self-centricity: a model in which compute is allocated towards predicting the future as represented within a dualistic ontology (consisting of the embedded self and the environment outside of the self).
I left out nuances to keep the blindspot summary short and readable. But I should have specifically prefaced what fell outside the scope of my writing. Not doing so made claims come across more extreme than I meant for the more literal/explicit readers amongst us :)
So for you who still happens to read this, here’s where I was coming from:
To describe blindspots broadly across the entire rationality and EA community.
In actual fact I see both communities more as loose clusters of interacting and affiliated people. Each gathered group somewhat diverges in how it attracts members who are predisposed towards focussing on (and reinforce each other to express) certain aspects as perceived within certain views.
I pointed out how a few groups diverge in the summary above (e.g. effective animal advocacy vs. LW decision theorists, thriving vs suffering-focussed EAs), but left out many others. Responding to Christian Kl’s earlier comment, I think how the ‘CFAR alumni’ cluster frames aspects meaningfully diverges from the larger/overlapping ‘long-time LessWrong fans’ cluster.
Previously, I suggested that EA staff could coordinate work more through non-EA-branded groups with distinct yet complementary scopes and purposes, so the general overarching tone of this post runs counter to that.
To aggregate common views within which our members seemed to most often frame problems (as expressed to others involved in the community they knew also aimed to work on those problems), and to contrast those with the foci held by other purposeful human communities out there.
Naturally, what an individual human focusses on in any given moment depends on their changing emotional/mental makeup and the context they find themselves (incl. the role they then identify as having) in. I’m not e.g. claiming that when someone who aspires to be a rational researcher at work focusses on brushing their teeth at home while glancing at their romantic partner, they must nevertheless be thinking real abstract and elegant thoughts.
But for me, the exercise of mapping our ingroup’s brightspots onto each listed dimension – relative to the focus of outside groups on – has provided some overview. The dimensions are from a perceptual framework I gradually put together and that is somewhat internally coherent (but predictably overwhelms anyone whom I explain it to, and leaves them wondering how it’s useful; hence this more pragmatic post).
I hope though no reader ends up using this as a personality test – say for identifying their or their friend’s (supposedly stable) character traits to predict their resulting future behaviour (or god forgive, to explain away any confusion or disagreement they sense about what an unfamiliar stranger says).
To keep each blindspot explanation simple and to the point:
If I already mix in a lot of ‘on one hand in this group...but on the other hand in this situation’, the reader will gloss over the core argument. I appreciate people’s comments with nuanced counterexamples though. Keeps me intellectually honest.
Hope that clarifies the post’s argumentation style somewhat.
I had those three starting points at the back of my mind while writing in March. So sorry I didn’t include them.
Thanks for engaging here!
Kahneman’s view of cognitive biases and heuristics was very structure-based. …. Reasoning with CFAR techniques on the other hand is more process-based.
I like this distinction, and actually agree! Have talked with a CFAR (ex-)staff member about this, who confirmed my impression that CFAR has been compensating factor in the community amongst most of the perceptual/cognitive dimensions I listed. Where you and I may disagree though is that I still think the default tendency for many rationalists is to construct problems as structure-based.
I don’t think we do. When discussing for example FDA decisions we don’t see Fauci as a person who’s independent of the system in which he operates.
Good nuances here that we don’t just see individuals as independent of the system they’re operating in. So there is some sense of interconnectivity there. I think you’re only partially capturing what I mean with interdependence though. See other comment for my attempt to convey this.
We certainly care more about the future than many other communities, but we also care about the past. Progress studies and the surrounding debates are very much focused on the past. We do have Petrov day which is about remembering a past event and reminding us that there was a history of the world being at risk.
I agree with all these points. We seem to be on one line here.
Happy to. Glad to hear any follow-up thoughts you have!