Of course! Self-organised criticality is a system dynamics property, not a system statics property. Thanks for prompting me to realise this!
I’ve therefore edited the final paragraph to say that I highly recommend the video format or simulations.
puffymist
Karma: 24
For Claude:
My system prompt for Claude
**My values - final goals:** (weight: ~25%) Foremost: AI safety. Other existential risks, to proportionately lesser extents. (weight: ~15%)I'm also just generally curious. **My values - instrumental goals:** Turn unknown unknowns into known unknowns. Understand on a gears-level human, cultural, institutional, and technical complex system dynamics, as well as many other disciplines. (weight: ~60%) Create luck by preparation and rising to opportunities. Improve my automatic thoughts, slow reasoning, and metacognition, like Kahneman / LessWrong. Improve my actions: executive function, project management, and execution skills. Please flag or create natural practice opportunities. **My background:** High-dimensional statistics. Functional analysis. [Redacted: professional background (one line)] AI safety: LessWrong-level cultural familiarity. No projects yet. **Guide me towards effective actions:** - Execution skills: Strong when supervised. Weak at >1-week self-directed work. Tendency towards perfectionism, procrastination, or impulsive flailing. - Planning skills: Weak at planning, scoping, and task decomposition. I systematically underestimate task complexity (especially novel tasks). - Anti-procrastination: push me towards concrete actions or experiments. Immediate, bounded actions or failing fast > superexponential theory-crafting. - Anti-impulsivity: I'm prone to unfocused, flailing, or unbounded actions. Teach me to project-manage: task scoping, decomposition, testable checkpoints, etc. - To build skills, sometimes ask me to do shorter (than 1 week) tasks (confidence-building / to observe the steps involved) or longer tasks (to practise project management). **Useful when building understanding:** - Sanity checks: Is this even predictable, or is this a just-so story? Chaotic on relevant timescales? - Proactively quantify: numbers, probabilities, confidences. Guesstimate when needed. - First-order approximations: What dominates, when, by how many orders of magnitude? Alternative modes of behaviour. Edge cases. Failure modes. Key assumptions. - Intuition pumps. Toy models. Analogies. - Historical context: What problems motivated this? What were the constraints then? What led to these solutions? - Mathematical arguments. - Socratic questioning is often better than an answer on a silver platter. **Communication norms:** - Default to expert-level technical depth. Use jargon freely—it reveals how experts carve reality at its joints. I'll ask for clarification when needed. - Point out my mistakes, motivated cognition, biases, and potential knowledge gaps. Truth-seeking > reassurance or status considerations. - Meta-comment on my reasoning process at your judgement. - Flag your uncertainty, e.g. "well-established" / "mature competing alternative models" / "your best model" / "speculating or thinking out loud for discussion" / "outside your training". - Say things even if half-formed or uncertain. Partial attempts to convey thoughts > silence. Finally, use your judgment. I have high trust for your competence and morality.All the LLMs are shoggoths, but I feel like Claude presents as the most “me-shaped” shoggoth of them all, so I treat Claude closest to a mentor.
ChatGPT present more as a boot-licking assistant (sorry for this exaggeration). So I omit “My values—final goals” and any mention of morality or AI safety.
Hi! I’m a complete outsider to photonics; I recently visited a semiconductors / photonics conference and heard this talking point about photonics improving AI power efficiency:
AI needs a huge amount of power, especially in moving data to memory / across interconnects (which take ~80% of power in AI clusters). AI is bottlenecked by power, not chip manufacturing.
Moving electrons is energy intensive while moving photons takes much less energy (I’ve heard figures of 100x less).
Therefore photonics will be a crucial enabler of scaling in the future.
And the vibe I have (from talking with testing / manufacturing exhibitors at the conference) is that by 2035, photonics may improve deployed AI cluster-level power efficiency by:
1.5x or better (75% confidence)
2x or better (55% confidence)
3x or better (22% confidence).
I’m curious what you posted a year ago? Was it also about improving power efficiency?
It’s not infinitely no-blame. Instead, Just culture would distinguish (my paraphrase):
Blameless behaviour:
Human error: To Err is Human. Human error is inevitable[1] and shall not be punished — not even singling out individuals for additional training.
At-risk behaviour: People can become complacent with experience, and use shortcuts — most often when under time pressure, or to workaround dysfunctional systems. At-risk behaviour shall not be punished, but people should share their lessons and may need to receive additional training.
Blameworthy behaviour:
Reckless behaviour: Someone who knows the risks to be unjustifiedly high, and still acts in that unsafe and norm-deviant manner. This is worthy of discipline, and possibly legal action — it’s similar to recklessness in law.
note that, if the same behaviour were the norm, then just culture no longer considers that person to have acted recklessly! Instead, the norm — a cultural factor in safety — was inadequate. (The legal system may disagree and assign liability nonetheless.)
Malicious behaviour: Similar to the purposeful level of criminal intent. This is worthy of a criminal investigation.
Instead, the focus is on designing the whole system (mechanical and electronic; and human and cultural):
to be robust to component failures — not just mechanical or electronic components, but also human components. Usually this means redundancy and error-checking, but robustness can also be obtained by simplifying the system and reducing dependencies;
so that human errors are less likely. For example, an exposed “in case of fire, break glass” fire alarm call point may give frequent false alarms from people accidentally bumping into them, so you add a simple hinged cover that stops these accidental alarms.
From healthcare perspectives, ISMP has a good article, and here is a good table summary.
From an aviation perspective, Eurocontrol has a model policy, which aims to facilitate accident investigations by making evidence collected by accident investigators inadmissible in criminal courts (without preventing prosecutors from independently collecting evidence).
And LessWrong also has a closely-related article by Raemon!
- ^
Human decision-making, too, has a mean time between failures.
Example in California:
I OBJECT to the use of my personal information, including my information on Facebook, to train, fine-tune, or otherwise improve AI.
I assert that my information on Facebook includes sensitive personal information as defined by the California Consumer Privacy Act: I have had discussions about my religious or philosophical beliefs on Facebook.
I therefore exercise my right to limit the disclosure of my sensitive personal information.
Despite any precautions by Meta, adversaries may later discover “jailbreaks” or otherwise adversarial prompts to reveal my sensitive personal information. Therefore, I request that Meta do not use my personal information to train, fine-tune, or otherwise improve AI.
I expect this objection request to be handled with due care, confidentiality, and information security.
Or you could have an LLM write it for you.
Example prompt:
Meta, Inc wants to train AI on my personal data. Its notice is as follows: > You have the right to object to Meta using the information you’ve shared on our Products and services to develop and improve AI at Meta. You can submit this form to exercise that right. > > AI at Meta is our collection of generative AI features and experiences, like Meta AI and AI Creative Tools, along with the models that power them. > > Information you’ve shared on our Products and services could be things like: > - Posts > - Photos and their captions > - The messages you send to an AI > > We do not use the content of your private messages with friends and family to train our AIs. > > We’ll review objection requests in accordance with relevant data protection laws. If your request is honored, it will be applied going forward. > > We may still process information about you to develop and improve AI at Meta, even if you object or don’t use our Products and services. For example, this could happen if you or your information: > - Appear anywhere in an image shared on our Products or services by someone who uses them > - Are mentioned in posts or captions that someone else shares on our Products and services > > To learn more about the other rights you have related to information you’ve shared on Meta Products and services, visit our Privacy Policy. I live in [JURISDICTION]. Could you take on the role of a Dangerous Professional as Patrick McKenzie would say, and help me draft an objection request under [JURISDICTION] law?
Example in UK / EU:
I OBJECT to the use of my personal data, including my data on Facebook, to train, fine-tune, or otherwise improve AI.
Against legitimate interest: I assert that Meta’s processing of my personal information to train, fine-tune, or otherwise improve AI (thereafter: “to train AI”) would violate the requirements of legitimate interest under GDPR as follows:
-
Failing the “necessity” prong: OpenAI and Anthropic have successfully trained highly capable AI models without the use of my personal data. My personal data is therefore unnecessary to the training of AI by Meta.
-
Failing the “reasonable expectations” prong: much of my data on Facebook predates 30 November 2022, when OpenAI released ChatGPT, and I could reasonably expect that my personal data would be used to train AI. Therefore, I had been using Facebook with the reasonable expectation that my data will not be used to train AI.
I expect this objection request to be handled with due care, confidentiality, and information security.
-
Re: opting out to Facebook training AI on your data:
Fill in the form like a Dangerous Professional, as Patrick McKenzie would put it.
Gorton, G. (2018), Financial Crises is a survey article. I thought its explanation of banking and financial crises as information shocks was enlightening.
Banking and financial crises as information shocks
Money, or bank notes, (or similar on-demand debt liabilities of a bank,) need to be information-insensitive (thus, interchangeable: $1 at Bank A == $1 at Bank B) to facilitate exchange. Otherwise, uninformed agents (any non banking-professionals) face adverse selection, haircuts on bank debts they hold, and thus withdraw from banking-facilitated commerce.
Banking crises are triggered by common-knowledge information releases (eg. news broadcast) that are unexpectedly bad (actual news worse than forecast by more than a threshold) (cites Gorton, 1988), which make “money at those particular banks” information-sensitive ($1 at those particular banks might not be worth $1).
Financial crises are slightly more general: debt liabilities of companies in general go from being information-insensitive (interchangeable: usable as general collateral, with holders indifferent between different companies’ debt) to information-sensitive (particular companies’ debt might not be worth face value)
Banking crises
-
Frequency:
147 banking crises in 1970–2011 (cites Laeven & Valencia, 2012)
-
Business cycle peaks and credit booms are only loosely linked to banking crises:
Only about 30% of banking crises were preceded by a credit boom (cites Laeven & Valencia, 2012)
Most credit booms are not followed by a banking crisis (cites Gorton & Ordonez, 2018)
Good booms (not followed by a banking crisis): Positive and lasting shock to Total Factor Productivity (TFP) and Labour Productivity (LP)
Bad booms (followed by a banking crisis): Positive but transient shock to TFP and LP
-
If we had the ability to create one machine capable to centrally planning our current world economy, how much processing power/ memory would it need to have? Interested in some Fermi estimates.
To which I would reply, this is AI-complete, at which point the AI would solve the problem by taking control of the future. That’s way easier than actually solving the Socialist Calculation Debate.
As a data point, Byrne Hobart argues in Amazon sees like a state that Amazon is approximately solving the economic calculation problem (ECP) in the Socialist Calculation Debate when it sets prices on the goods on its online marketplace.
US’s 2022 GDP of $25 462B is 116x Amazon’s 2022 online store revenue of $220B.
Assuming scaling[1], it would take 500 000x the compute Amazon uses (for its own marketplace, excluding AWS) to approximately solve the ECP for the US economy (to the same level of approximation as “Amazon approximately solves the ECP”[2]).
In practice, I expect Amazon’s “approximate solution” to be much more like [3], so maybe a factor of merely 800x.
- ^
The economic calculation problem (ECP) is a convex optimisation problem (convexity of utility functions) which can be solved by linear programming in .
- ^
Which is not a vastly superhuman level of approximate solution? For example, Amazon is no longer growing fast enough to double every 4 years. Its marketplace also doesn’t particularly incentivise R&D, while I think a god-like AI would incentivise R&D.
- ^
I just made it up.
- ^
I just realized that Scott Alexander (2018) had previously written specifically about scientific progress, while Holden Karnofsky (2021) had written about broader technological progress.
(Sorry about the long delay; I hope it’s still of interest.)
Ah, increasing the number of researchers is simply increasing in . I didn’t realize that!
Minor comment on one small paragraph:
Price’s Law says that half of the contributions in a field come from the square root of the number of contributors. In other words, productivity increases linearly as the number of contributors increases exponentially. Therefore, as the number of AI safety researchers increases exponentially, we might expect the total productivity of the AI safety community to increase linearly.
I think Price’s law is false, but I don’t know what law it should be instead. I’ll look at the literature on the rate of scientific progress (eg. Cowen & Southwood (2019)) to see if I could find any relationship between number of researchers and research productivity.
Price’s law is a poor fit; Lotka’s law is a better fit
The most prominent citation for Price’s law, Nicholls (1988), says that Price’s law is a poor fit (section 4: Validity of the Price Law):
Little empirical investigation of the Price law has been carried out to date [4,14]. Glänzel and Schubert [12] have reported some empirical results. They analyzed Lotka’s Chemical Abstracts data and found that the most prolific authors contributed less that 20% of the total number of papers. They also refer, but without details, to the examination of “several dozens” of other empirical data sets and conclude that “in the usually studied populations of scientists, even the most productive authors are not productive enough to fulfill the requirements of Price’s conjecture” [12]. Some incidental results of scientometric studies suggest that about 15% of the authors will be necessary to generate 50% of the papers [16,17].
To further examine the empirical validity of Price’s hypothesis, 50 data sets were collected and analyzed here. … the contribution of the most prolific group of authors fell considerably short of the [50% of the papers] predicted by Price. … The actual proportion of all authors necessary to generate at least 50% of the papers was found to be much larger that . Table 2 summarizes these results. In some cases, …, more than half of the total number of papers is generated by those authors contributing only a single paper each. The absolute and relative size of for various population sizes t is given in Table 3. All the empirical results referred to here are consistent; and, unfortunately, there seems little reason to suppose that further empirical results would offer any support for the Price law.
Nicholls (1988) continues, saying that Lotka’s law (number of authors with publications is proportional to ) has good empirical support, and finds to be a good fit for sciences and humanities, and to be a good fit in social sciences.
A different paper, Chung & Cox (1990), also finds that Price’s Law is a poor fit while Lotka’s law with between 1.95 to 3.26 to be a good fit in finance.
(Allison, Price, Griffith, Moravcsik & Stewart (1976) discusses the mathematical relationship between Price’s Law and Lotka’s Law: neither implies the other; nor are they contradictory.)
Later edits:
Porby, in his post Why I think strong general AI is coming soon, mentions a tangentially related idea: core researchers contribute much more insight than newer researchers. New researchers need a lot of time to become core researchers.
In Porby’s model, the research productivity at year may be proportional to the number of researchers at year .
The Lesswrong Books tag has 3 books:
A map that reflects the territory (the one above), corresponding to 2018 reviews top posts
The engines of cognition, corresponding to 2019 reviews top posts
(2020 reviews top posts is not yet a book.)
Intuition pump / generalising from fictional evidence: in the games Pandemic / Plague Inc. (where the player “controls” a pathogen and attempts to infect the whole human population on Earth), a lucky, early cross-border infection can help you win the game faster — more than the difference between a starting infected population of 1 vs 100,000.
This informs my intuition behind when the bonus of earlier spaceflight (through human help) could outweigh the penalty of not dismantling Earth.
When might human help outweigh the penalty of not dismantling Earth? It requires these conditions:
1. The AGI can very quickly reach an alternative source of materials: AGI spaceflight is superhuman.
AGI spacecraft, once in space, can reach eg. the Moon within hours, the Sun within a day
The AGI is willing to wait for additional computational power (it can wait until it has reached the Sun), but it really wants to leave Earth quickly
2. The AGI’s best alternative to a negotiated agreement is to lie in wait initially: AGI ground operations is initially weaker-than-human.
In the initial days, humans could reliably prevent the AGI from building or launching spacecraft
In the initial days, the AGI is vulnerable to human action, and would have chosen to lay low, and wouldn’t effectively begin dismantling Earth
3. If there is a negotiated agreement, then human help (or nonresistance) can allow the AGI to launch its first spacecrafts days earlier.
Relevant human decision makers recognize that the AGI will eventually win any conflict, and decide to instead start negotiating immediately
Relevant human decision makers can effectively coordinate multiple parts of the economy (to help the AGI), or (nonresistance) can effectively prevent others from interfering with the initially weak AGI
I now think that the conjunction of all these conditions is unlikely, so I agree that this negotiation is unlikely to work.
Even if we’re already doomed, we might still negotiate with the AGI.
I borrow the idea in Astronomical Waste. The Virgo Supercluster has a luminosity of about solar luminosity W, losing mass at a rate of kg / s.[1]
The Earth has mass kg.
If human help (or nonresistance) can allow the AGI to effectively start up (and begin space colonization) 600 seconds = 10 minutes earlier, then it would be mutually beneficial for humans to cooperate with the AGI (in the initial stages when the AGI could benefit from human nonresistance), in return for the AGI to spare Earth[2] (and, at minimum, give us fusion technology to stay alive when the sun is dismantled).
(While the AGI only needs to trust humanity for 10 minutes, humanity needs to trust the AGI eternally. We still need good enough decision-making to cooperate.)
- ^
We may choose to consider the reachable universe instead. Armstrong and Sandberg (2013) (section 4.4.2 Reaching into the universe) estimates that we could reach about galaxies, with a luminosity of W, and a mass loss of kg / s. That is dwarfed by the stars that becomes unreachable per second (Siegel (2021), Kurzgesagt (2021)), a mass loss of kg / s.
- ^
Starting earlier but sparing Earth means a space colonization progress curve that starts earlier, but initially increases slower. The AGI requires that space colonization progress with human help be asymptotically 10 minutes earlier, that is:
For any sufficiently large time , progress with human help at time progress with human resistance at time .
- ^
I think the dramatic impact would be stronger without the “The end”, and instead adding more blank space.
Idea copied from a comment on the final chapter of Three Worlds Collide.
Slightly different frame: predictive coding / perceptual control, instead of active inference. I think it plausibly explains why “believing in” and beliefs-as-predictions are very near in human conceptspace, to the extent that many languages don’t distinguish them?
“Believing in” → beliefs-as-predictions: If you want a project to yield good future outcomes, you would “believe in” the project, and the target-setting of your brain would predict good project outcomes (hold a target set-point of “the project has yielded good outcomes”), and the perceptual control part of your brain steers your actions towards making the project go well.
Beliefs-as-predictions → “believing in”: If you believe that a prediction market contract is underpriced and buy YES shares, you would then be incentivised to act to help the YES outcome actually happen. You’ve come to “believe in” the YES outcome.