Maxime Riché

Karma: 222

Maxime Riché 5 Sep 2024 9:56 UTC
2 points
1
on: The Fragility of Life Hypothesis and the Evolution of Cooperation
For information to the readers and author: I am (independently) working on a project about narrowing down the moral values of alien civilizations on the verge of creating an ASI and becoming space-faring. The goal is to inform the prioritization of longtermist interventions.

I will gladly build on your content, which aggregates and beautifully expands several key mechanisms (individual selection (“Darwinian demon”), kin selection/multilevel selection (“Darwinian angel”), filters (“Fragility of Life Hypothesis)) that I use among others (e.g. sequential races, cultural evolution, accelerating growth stages, etc.).
Thanks for the post!

Maxime Riché 13 Aug 2024 9:51 UTC
8 points
2
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
If the following correlations are true, then the opposite may be true (slave morality being better for improving the world through history):
- Improving the world being strongly correlated with economic growth (this is probably less true when X-risk are significant)
- Economic growth being strongly correlated with Entrepreneurship incentives (property rights, autonomy, fairness, meritocracy, low rents)
- Master morality being strongly correlated with acquiring power and thus decreasing the power of others and decreasing their entrepreneurship incentives

Maxime Riché 5 Jul 2024 15:48 UTC
1 point
0
in reply to: ChristianKl’s comment on: Maxime Riché′s Shortform
Right 👍

So the effects are:

Effects that should increase Anthropic’s salaries relative to OpenAI: A) - The pool of AI safety focused candidates is smaller B) - AI safety focused candidates are more motivated

Effects that should decrease Anthropic’s salaries relative to OpenAI: C) - AI safety focused candidates should be willing to accept significantly lower wages

New notes: (B) and (C) could cancel each other but that would be a bit suspicious. Still a partial cancellation would make a difference between OpenAI and Anthropic lower and harder to properly observe. (B) May have a small effect, given that hires are already world level talents, it would be weird that they could significantly increase even more their performance by simply being more motivated. I.e. non AI safety focused candidates are also very motivated. The difference in motivation between both groups is possibly not large.

Maxime Riché 5 Jul 2024 15:39 UTC
1 point
0
in reply to: Violet Hour’s comment on: Maxime Riché′s Shortform
These forecasts are about the order under which functionalities see a jump in their generalization (how far OOD they work well).

By “Generalisable xxx” I meant the form of the functionality xxx that generalize far.

Maxime Riché 5 Jul 2024 13:47 UTC
2 points
0
on: Maxime Riché′s Shortform
Rambling about Forecasting the order in which functions are learned by NN

Idea:
Using function complexity and their “compoundness” (edit 11 september: these functions seem to be called “composite functions”), we may be able to forecast the order in which algorithms in NN are learned. And we may be able to forecast the temporal ordering of when some functions or behaviours will start generalising strongly.

Rambling:
What happens when training neural networks is similar to the selection of genes in genomes or any reinforcement optimization processes. Compound functions are much harder to learn. You need each part to be independently useful initially to provide enough signal for the compound system to be reinforced.

That means that learning any non-hardcoded algorithms with many variables and multiplicative steps is very difficult.
An important factor in this is the frequency at which an algorithm is useful and to which extent. An algorithm that can be very used in most situations will get much more training signals. The relative strength of the reward signal you get is important because of the noise in the training and because of catastrophic forgetting.

LLMs are not learning complex algorithms yet. They are learning something like a world model because this is used for most tasks and it can be built by first building each part separately and then assembling them.

Regarding building algorithms to exploit this world model, it can be learned later if the algorithm is composed first of very simple algorithms that can be later assembled. An extra difficulty for LLMs to learn algorithms is in situations where heuristics already work very well. In that case, you need to add significant regularisation pushing for simpler circuits. Then you may observe ~~grokking~~ learning and a transition from heuristics to algorithms.

An issue with this reasoning is that heuristics are 1-step algorithms (0 compoundness).

Effects:
- Frequency of reward
- Strength of the additional reward (above the “heuristic baseline”)
- Compoundness

Forecasting game:
(WIP, mostly a failure at that point)

Early to generalize well:
World models can be built from simple parts, and are most of the time valuable.
Generalizable algorithm for simple and frequent tasks on which heuristics fail dramatically: ??? (maybe) generating random numbers, ??

Medium to generalize well:
Generalizable deceptive alignment algorithms: They require several components to work. But they are useful for many tasks. The strength of the additional reward is not especially high or low.
Generalizable instrumental convergence algorithms: Same as deceptive alignment.
Generalizable short horizon algorithms: They, by definition, require fewer sequential steps, as such they should be less “compounded” functions and appear sooner.

Late:
Generalizable long horizon algorithms: They, by definition, require more sequential steps, as such they should be more “compounded” functions and appear later.

The latest:
Generalizable long horizon narrow capabilities: They are not frequently reinforced.

(Time spent on this: 45min)
July 6th update:
Here is a quick experiment trying to observe the effect of increasing “compoundness” on the ordering of ~~grokking~~ learning different functions: https://colab.research.google.com/drive/1B85mfCkqyQZSl1JGbLr0r5BrAS8LYUr5?usp=sharing

Quick results:
The task is predicting the sign of the product of 1 (function 1) to 8 (function 8) standard normal random variables.
Increasing the compoundness by 2 seems to delay the ~~grokking~~ learning by something like 1 OOM.

Maxime Riché 4 Jul 2024 9:02 UTC
2 points
0
on: Maxime Riché′s Shortform
Will we get to GPT-5 and GPT-6 soon?

This is a straightforward “follow the trend” model which tries to forecast when GPT-N-equivalent models will be first trained and deployed up to 2030.

Baseline forecast:
GPT-4.7 GPT-5.3 GPT-5.8 GPT-6.3
Start of training 2024.4 2025.5 2026.5 2028.5
Deployment 2025.2 2026.8 2028 2030.2

Bullish forecast:
GPT-5 GPT-5.5 GPT-6 GPT-6.5
Start of training 2024.4 2025 2026.5 2028.5
Deployment 2025.2 2026.5 2028 2030

FWIW, it predicts roughly similar growth in model size, energy cost and GPU count than described in https://situational-awareness.ai/ while being created the week before this was released.

I spent like 10 hours on this, so I expect to find lingering mistakes in the model.

Maxime Riché 26 Jun 2024 8:10 UTC
3 points
−5
on: Maxime Riché′s Shortform
Could Anthropic face an OpenAI drama 2.0?
I forecast that Anthropic would likely face a similar backlash from its employees than OpenAI in case Anthropic’s executives were to knowingly decrease the value of Anthropic shares significantly. E.g. if they were to switch from “scaling as fast as possible” to “safety-constrained scaling”. In that case, I would not find it surprising that a significant fraction of Anthropic’s staff threatened to leave or leave the company.
The reasoning is simple, given that we don’t observe significant differences in the wages of OpenAI and Anthropic employees and assuming that they are overall of the same distribution of skill and skill level. Then it seems that Anthropic is not able to use the argument of its AI safety focus as a bargaining argument to reduce the wages significantly. If true this would mean that safety is of relatively little importance to most of Anthropic’s employees.
Counter argument: Anthropic is hiring from a much more restricted pool of candidates. From only the safety-concerned candidates. In that case, Anthropic would have to pay a premium to hire these people. And it happens that this premium is roughly equivalent to the discount that these employees are willing to give to Anthropic because of its safety focus.

Maxime Riché 10 Jun 2024 10:16 UTC
2 points
1
on: Maxime Riché′s Shortform
What is the difference between Evaluation, Characterization, Experiments, and Observations?
The words evaluations, experiments, characterizations, and observations are somewhat confused or confusingly used in discussions about model evaluations (e.g., ref, ref).
Let’s define them more clearly:
- Observations provide information about an object (including systems).
  - This information can be informative (allowing the observer to update its beliefs significantly), or not.
- Characterizations describe distinctive features of an object (including properties).
  - Characterizations are observations that are actively designed and controlled to study an object.
- Evaluations evaluate the quality of distinctive features based on normative criteria.
  - Evaluations are composed of both characterizations and normative criteria.
  - Evaluations are normative, they inform about what is good or bad, desirable or undesirable.
  - Normative criteria (or “evaluation criterion”) are the element bringing the normativity. They are most of the time directional or simple thresholds.
  - Evaluations include both characterizations of the object studied and characterization of the characterization technique used (e.g., accuracy of measurement).
- Scientific experiments test hypotheses through controlled manipulation of variables.
  - Scientific experiments are composed of: characterizations, and hypothesis
In summary:
- Observations
- Characterizations = Designed and controlled Observations
- Evaluations = Characterization of object + Characterization of the characterization method + Normative criteria
- Scientific experiments = Characterizations + Hypothesis
Examples:
- An observation is an event in which the observer receives information about the AI system.
  - E.g., you read a completion returned by a model.
- A characterization is a tool or process used to describe an AI system.
  - E.g., you can characterize the latency of an AI system by measuring it. You can characterize how often a model is correct (without specifying that correctness is the goal).
- An AI system evaluation will associate characterizations and normative criteria to conclude about the quality of the AI system on the dimensions evaluated.
  - E.g., alignment evaluations use characterizations of models and the normative criteria of the alignment with X (e.g., humanity) to conclude on how well the model is aligned with X.
- An experiment will associate hypotheses, interventions, and finally characterizations to conclude on the veracity of the hypotheses about the AI system.
  - E.g., you can change the training algorithm and measure the impact using characterization techniques.
Clash of usage and definition:
These definitions slightly clash with the usage of the term evals or evaluations in the AI community. Regularly the normative criteria associated with an evaluation are not explicitly defined, and the focus is solely put on the characterizations included in the evaluation.

(Produced as part of the AI Safety Camp, within the project: Evaluating alignment evaluations)
What links here?
- A Taxonomy Of AI System Evaluations [WIP] by Maxime Riché (19 Aug 2024 9:07 UTC; 5 points)
- Thinking About What Are Propensity Evaluations [WIP] by Maxime Riché (19 Aug 2024 9:23 UTC; 5 points)

Maxime Riché 15 May 2024 10:29 UTC
2 points
2
in reply to: ektimo’s comment on: How to do conceptual research: Case study interview with Caspar Oesterheld
Likely: Path To Impact

Maxime Riché 29 Apr 2024 8:49 UTC
2 points
0
on: Refusal in LLMs is mediated by a single direction
Interestingly, after a certain layer, the first principle component becomes identical to the mean difference between harmful and harmless activations.
Do you think this can be interpreted as the model having its focus entirely on “refusing to answer” from layer 15 onwards? And if it can be interpreted as the model not evaluating other potential moves/choices coherently over these layers. The idea is that it could be evaluating other moves in a single layer (after layer 15) but not over several layers since the residual stream is not updated significantly.
Especially can we interpret that as the model not thinking coherently over several layers about other policies, it could choose (e.g., deceptive policies like defecting from the policy of “refusing to answer”)? I wonder if we would observe something different if the model was trained to defect from this policy conditional on some hard-to-predict trigger (e.g. whether the model is in training or deployment).

Maxime Riché 26 Apr 2024 20:35 UTC
1 point
0
in reply to: Vladimir_Nesov’s comment on: Scaling of AI training runs will slow down after GPT-5
Thank for the great comment!

Do we know if distributed training is expected to scale well to GPT-6 size models (100 trillions parameters) trained over like 20 data centers? How does the communication cost scale with the size of the model and the number of data centers? Linearly on both?
After reading for 3 min this:
Google Cloud demonstrates the world’s largest distributed training job for large language models across 50000+ TPU v5e chips (Google November 2023). It seems that scaling is working efficiently at least up to 50k GPUs (GPT-6 would be like 2.5M GPUs). There are also some surprising linear increases in start time with the number of GPUs, 13min for 32k GPUs. What is the SOTA?

Maxime Riché 26 Apr 2024 19:36 UTC
7 points
0
in reply to: Chris_Leong’s comment on: Scaling of AI training runs will slow down after GPT-5
The title is clearly an overstatement. It expresses more that I updated in that direction, than that I am confident in it.
Also, since learning from other comments that decentralized learning is likely solved, I am now even less confident in the claim, like only 15% chance that it will happen in the strong form stated in the post.

Maybe I should edit the post to make it even more clear that the claim is retracted.

Maxime Riché 26 Apr 2024 12:39 UTC
1 point
0
in reply to: Maxime Riché’s comment on: The longest training run
This is actually corrected on the Epoch website but not here (https://epochai.org/blog/the-longest-training-run)

Maxime Riché 26 Apr 2024 12:38 UTC
1 point
0
on: The longest training run
We could also combine this with the rate of growth of investments. In that case we would end up with a total rate of growth of effective compute equal to $g_{H} + g_{I} + g_{S} \approx 0.28 + 3.84 + 0.54 = 4.66$ . This results in an optimal training run length of $L = 1 / (g_{H} + g_{I} + g_{S}) \approx 0.21$ years, ie $2.52$ months.
Why is g_I here 3.84, while above it is 1.03?

Maxime Riché 23 Mar 2024 23:49 UTC
3 points
0
on: Dangers of Closed-Loop AI
Are memoryless LLMs with a limited context window, significantly open loop? (Can’t use summarization between calls nor get access to previous prompts)

Maxime Riché 23 Jan 2024 11:57 UTC
LW: 9 AF: 3
0
AF
on: We need a science of evals
FYI, the “Evaluating Alignment Evaluations” project of the current AI Safety Camp is working on studying and characterizing alignment(propensity) evaluations. We hope to contribute to the science of evals, and we will contact you next month. (Somewhat deprecated project proposal)

Maxime Riché 4 Dec 2023 3:28 UTC
1 point
0
in reply to: Tom Davidson’s comment on: An illustrative model of backfire risks from pausing AI research
Interesting! I will see if I can correct that easily.

Maxime Riché 10 Nov 2023 14:57 UTC
7 points
9
on: AI Timelines
Thanks a lot for the summary at the start!

Maxime Riché 9 Oct 2023 9:31 UTC
3 points
0
in reply to: Adele Lopez’s comment on: AI Alignment Breakthroughs this week (10/08/23)
I wonder if the result is dependent on the type of OOD.

If you are OOD by having less extractable information, then the results are intuitive.
If you are OOD by having extreme extractable information or misleading information, then the results are unexpected.

Oh, I just read their Appendix A: “Instances Where “Reversion to the OCS” Does Not Hold”
Outputting the average prediction is indeed not the only behavior OOD. It seems that there are different types of OOD regimes.

Maxime Riché 2 Oct 2023 16:35 UTC
1 point
0
in reply to: Kaj_Sotala’s comment on: Expectations for Gemini: hopefully not a big deal
This comes from OpenAI saying they didn’t expect ChatGPT to be a big commercial success. It was not a top-priority project.

	`GPT-4.7`	`GPT-5.3`	`GPT-5.8`	`GPT-6.3`
Start of training	`2024.4`	`2025.5`	`2026.5`	`2028.5`
Deployment	`2025.2`	`2026.8`	`2028`	`2030.2`

	`GPT-5`	`GPT-5.5`	`GPT-6`	`GPT-6.5`
Start of training	`2024.4`	`2025`	`2026.5`	`2028.5`
Deployment	`2025.2`	`2026.5`	`2028`	`2030`

Maxime Riché

What is the difference between Evaluation, Characterization, Experiments, and Observations?