Cyber Ambassador đčđŒ Taiwan (2024-) and founding Minister of Digital Affairs.
Senior Accelerator Fellow, Institute for Ethics in AI, University of Oxford.
Audrey Tang
Karma: 444
Indeed, youâve powerfully articulated the classic misaligned singleton failure, which assumes a vertical Singularity.
The alternative is a horizontal Plurality: Not a global emperor, but an ecosystem of local guardians (kami), each bound to a specific community â a river, a city. There is no single âAIâ to seize control. The riverâs kami has no mandate over the forestâs. Power is federated and subsidiary by design.
Far from making humane relationships âinsignificant,â this architecture makes them the only thing that matters. Communitarian values are not trivial decorations; they are the very purpose for which each kami exists.
It would indeed be unfortunate if singular intelligence renders humane relationships insignificant. Letâs free the future â toward a plurality of intelligences that exist to make them possible.
Youâve precisely identified the central crux. This dynamic â whether bounded communities can be stable against unbounded optimization â was a main focus of my recent conversation with Plex: Symbiogenesis vs. Convergent Consequentialism.
Your intuition holds true for offense-dominant environments. The d/âacc move is to actively shape the environment toward defense-dominance. In such a reality, bounded, cooperative strategies (symbiogenesis) become evolutionarily stable, and the most ambitious strategy is robust cooperation, rather than unilateral vampirism.
This is now posted at: âż» Symbiogenesis vs. Convergent Consequentialism
âż» SymÂbioÂgeÂnÂeÂsis vs. ConÂverÂgent Consequentialism
Yes! I did read the revised series with great interest, and I fondly recall my participation in the Foresight Institute 2021 seminar that contributed to the 2022 edition.
On the âspeed-run subsidiarityâ idea, I really resonate with your comment:
This is the first non-destructive non-coercive pivotal act that Iâve considered plausibly sufficient to save us from the current crisis.
In order to rapidly decentralize action without dissolving into chaos or resorting to localized coercion, we need a shared operational ethic for horizontal alignment to scale quickly. This, I believe, is where the concept of âcivic careâ becomes an essential catalyst.
Unlike abstract, âthinâ ideologies, civic care is rooted in normative relational ethics. It emphasizes context, interdependence, and the direct responsibility we have for the relationships we are embedded within. This accelerates the speed-run in several ways:
-
Reduced Coordination Overhead: When civic care becomes the default ethical posture for AI training and deployment, we minimize the friction of cooperative agents by focusing on the immediate, tangible needs of the community.
-
Inherent Locality: The ethic of care naturally aligns with subsidiarity. We are best equipped to care for the relationships and environments closest to us. It intrinsically motivates action at the most appropriate level, strengthening local capacity.
-
Rapid Trust Building: Subsidiarity fails without high trust. By adopting a shared commitment to care, decentralized groups can establish trust much faster. This high-trust environment is a prerequisite for navigating crises effectively.
-
Hi! Thank you for this note and for sharing your work on âWitness.â
Youâre right: AI already underwrites coordination. The lever is not if we use it, but how. Shifting from unbounded strategic agents to bounded, toolâlike âkami of careâ makes safety architectural; scope/âlatency/âbudget are capped, so powerâseeking becomes an antiâpattern.
Witness cleanly instantiates Attentiveness: a push channel that turns lived experience into common knowledge with receipts, so alignment happens by process and at the speed of society. This is d/âacc applied to epistemic security in practice, through groundâup shared reality paired with stronger collective steering.
Great! If there are such areas, in the spirit of d/âacc, Iâd be happy to use a local language model to paraphrase them away and co-edit in an end-to-end-encrypted way to confirm before publishing.
On north star mapping: Does the CIP Global Dialogues and GD Challenge look like something of that shape, or something more like AI Social Readiness Process?
On Raemonâs (very insightful!) piece w.r.t. curing cancer inevitably routing through consequentialism: Earlier this year I visited Bolinas, a birthplace of integrative cancer care, which centers healing for communities, catalyzed by people experiencing cancer. This care ethic prioritizes virtues like attentiveness and responsiveness to relational health over outcome optimization.
Asking a superintelligence to âsolve cancerâ in one fell swoop â regardless of collateral disruptions to human relationships, ecosystems, or agency â directly contravenes this, as it reduces care to a terminal goal rather than an ongoing, interdependent process.
In a d/âacc future, one tends to the research ecosystem so progress emerge through horizontal collaboration â e.g., one kami for proteinâfolding simulation, one for crossâlab knowledge sharing; none has the unbounded objective âcure cancer.â We still pursue cures, but with kamis each having non-fungible purposes. The scope, budget, and latency caps inherent in this configuration means capability gains donât translate into openâended optimization.
Iâd be happy to have an on-the-record conversation, co-edited and published under CC0 to SayIt next Monday 1pm if you agree.
Thank you for the encouragement, recommendations, and for flagging the need for more context on strong ASI models, including the default extremity of the transition!
Youâre spot on; my DeepMind talk emphasized horizontal alignment (defense against coordination failures) as a complement to vertical alignment perils, like those in the orthogonality thesis and instrumental convergence.
Iâve pre-ordered the IABIED book and have now re-read several recommendations: âAGI Ruinâ details lethalities and âA Central AI Alignment Problemâ highlights the sharp left turnâs risk. Just reviewed âFive Theses, Two Lemmas,â which reinforces the intelligence explosion, complexity/âfragility of value, and indirect normativity as paths to safer goals.
These sharpen why 6pack.care prioritizes local kami (bounded, non-maximizing agents) to mitigate unbounded optimization and promote technodiversity over singleton risks.
Topics Iâd love to discuss further:
How might heterarchical ecologies of multipolar AI mitigate instrumental convergence?
How would âthickâ, pluralistic alignment integrate with indirect normativity?
In slower takeoff scenarios, could subsidiarity (as envisioned in d/âacc 2035) help navigate sharp left turns?
Indeed, there are two classes of alignment problems. The first is vertical: making a single agent loyal. The second is horizontal: ensuring a society of loyal agents doesnât devolve into systemic conflict. 6pack.care is a framework for the second challenge as articulated by CAIF.
It posits that long-term alignment is not a static property but a dynamic capability: alignment-by-process. The core mechanism is a tight feedback loop of civic care, turning interactions into a form of coherent blended volition. This is why the flood bot example, though simple, describes this fractal process.
This process-based approach is also our primary defense against power-seeking. Rather than trying to police an agentâs internal motives, we design the architecture for boundedness (kami) and federated trust (e.g. ROOST.tools), making unbounded optimization an anti-pattern. The system selects for pro-sociality.
This bridges two philosophical traditions: EA offers a powerful consequentialist framework. Civic Care provides the necessary process-based virtue ethic for a pluralistic world. The result is a more robust paradigm: EA/âCC (Effective Altruism with Civic Care).
Hi! Great to hear from you. âOptimize for funâ (âOfun) is still very much the spirit of this 6pack.care work.
On practicality (bending the market away from armsârace incentives): Here are some levers that worked, inspired by Taiwanâs tax-filing case, that shift returns from lockâin to civic care:
Interoperability: Make âdata portabilityâ the rule. Mandate fair protocolâlevel interop so users and complements can exit without losing their networks. Platforms must compete on quality of care, not captivity.
Public options: Offer simple public options (and shared research compute) so thereâs always a baseline service that is easy, safe, and nonâextractive. Private vendors must beat it on care, not on lockâin.
Provenance for Paid Reach: For ads and mass reach in political/âfinancial domains, require verifiable sponsorship and durable disclosure. Preserve anonymity for ordinary speech via meronymity.
MissionâLocked Governance â Through procurement rules, ensure stewardâownership/âbenefit structures and boardâlevel safety duties so âcivic careâ is a fiduciary obligation, not a marketing slogan.
Institutionalize Alignment Assemblies and localized evals; preâcommit vendors to adopt outcomes or explain deviations. Federate trust & safety so threat intel flows without central chokepoints.
On symbiosis: the kami view is neither egalitarian sameness nor fixed hierarchy. Itâs a bounded, heterarchical ecology with many stewards with different scopes that coordinate without a permanent apex. (Heterarchy = overlapping centers of competence; authority flows to where the problem lives.)
Egalitarianism would imply interchangeable agents. As capabilities grow, weâll see a range of kami sizes: a steward for continental climate models wonât be the same as one for a local irrigation system. Thatâs diversity of scope, not inequality of standing.
Hierarchy would imply command. Boundedness prevents that: each kami is powerful only within its scope of care and is designed for âenough, not forever.â The river guardian has no mandate nor incentive to run the forest.
When scopes intersect, alignment is defined by civic care: Each kami maintain the relational health of their shared ecosystem at the speed of the garden. Larger systems may act as ephemeral conveners, but they donât own the graph or set permanent policy. Coordination follows subsidiarity and federation: solve issues locally when possible; escalate via shared protocols when necessary. Meanwhile, procedural equality (the right to contest, audit, and exit) keeps the ecology plural rather than feudal.
âż» PluÂralÂity & 6pack.care
I wrote a summary in Business Weekly Taiwan (April 24):
https://ââsayit.archive.tw/ââ2025-04-24-%E5%95%86%E5%91%A8%E5%B0%88%E6%AC%84ai-%E6%9C%AA%E4%BE%86%E5%AD%B8%E5%AE%B6%E7%9A%84-2027-%E5%B9%B4%E9%A0%90%E8%A8%80
https://ââsayit.archive.tw/ââ2025-04-24-bw-column-an-ai-futurists-predictions-f
https://ââwww.businessweekly.com.tw/ââarchive/ââArticle?StrId=7012220
An AI Futuristâs Predictions for 2027
When President Trump declared sweeping reciprocal tariffs, the announcement dominated headlines. Yet inside Silicon Valleyâs tech giants and leading AI labs, an even hotter topic was âAIâ2027.com,â the new report from exâOpenAI researcher Daniel Kokotajlo and his team.At OpenAI, Kokotajlo had two principal responsibilities. First, he was charged with sounding early alarmsâanticipating the moment when AI systems could hack systems or deceive people, and designing defenses in advance. Second, he shaped research priorities so that the companyâs time and talent were focused on work that mattered most.
The trust he earned as OpenAIâs inâhouse futurist dates back to 2021, when he published a set of predictions for 2026, most of which have since come true. He foresaw two pivotal breakthroughs: conversational AIâexemplified by ChatGPTâcaptivating the public and weaving itself into everyday life, and âreasoningâ AI spawning misinformation risks and even outright lies. He also predicted U.S. limits on advancedâchip exports to China and AI beating humans in multiâplayer games.
Conventional wisdom once held that everâlarger models would simply perform better. Kokotajlo challenged that assumption, arguing that future systems would instead pause midâcomputation to âthink,â improving accuracy without lengthy additional training runs. The idea was validated in 2024: dedicating energy to reasoning, rather than only to training, can yield superior results.
Since leaving OpenAI, he has mapped the global chip inventory, density, and distribution to model AI trajectories. His projection: by 2027, AI will possess robust powers of deception, and the newest systems may take their cues not from humans but from earlier generations of AI. If governments and companies race ahead solely to outpace competitors, serious alignment failures could follow, allowing AI to become an independent actor and slip human control by 2030. Continuous investment in safety research, however, can avert catastrophe and keep AI development steerable.
Before the tariff news, many governments were pouring money into AI. Now capital may be diverted to shore up companies hurt by the tariffs, squeezing safety budgets. Yet longâterm progress demands the opposite: sustained funding for safety measures and the disciplined use of highâquality data to build targeted, reliable small modelsâso that AI becomes a help to humanity, not an added burden.
Based on my personal experience in pandemic resilience, additional wakeups can proceed swiftly as soon as a specific society-scale harm is realized.
Specifically, as we are waking up to over-reliance harms and addressing them (esp. within security OODA loops), it would buy time for good enough continuous alignment.
Based on recent conversations with policymakers, labs and journalists, I see increased coordination around societal evaluation & risk mitigation â (cyber)security mindset is now mainstream.
Also, imminent society-scale harm (e.g. contextual integrity harms caused by over-reliance & precision persuasion since ~a decade ago) has shown to be effective in getting governments to consider risk reasonably.
Well, before 2016, I had no idea Iâd serve in the public sector...
(The vTaiwan process was already modeled after CBV in 2015.)
Yes. The basic assumption (of my current day job) is that good-enough contextual integrity and continuous incentive alignment are solvable well within the slow takeoff we are currently in.
Something like a lightweight version of the off-the-shelf Vision Pro will do. Just as nonverbal cues can transmit more effectively with codec avatars, post-symbolic communication can approach telepathy with good enough mental models facilitated by AI (not necessarily ASI.)
Hi! If I understand you correctly, the risk you identify is an AI gardener with its own immutable preferences, pruning humanity into compliance.
Here, a the principle of subsidiarity would entail BYOP: Bring Your Own Policy. A concrete example is our team at ROOST working with OpenAI to release its gpt-oss-safeguard model. Its core feature is not a set of rules, but the ability for any community to evolve its own code of conduct in plain language.
If a community decides its values have changed â that it wants to pivot from its governance to pursue a transformative future â the gardener is immediately responsive. In an ecosystem of communities empowered with kamis, we can preserve the potential to let a garden grow wild if they so choose.