E. P. Cooper

Karma: 11

E. P. Cooper 14 Mar 2026 4:33 UTC
1 point
0
in reply to: Richard_Ngo’s comment on: Distributed vs centralized agents
I’ll note that “not being sure what utility functions are in use” is generally (in the colloquial sense) not how standard game theory works. It seems like I am not competent enough at standard game theory to clearly write down the edge cases I think might exist that could help with your understanding. This paragraph could serve as a placeholder for the case where I develop that competence.
As for non-standard game theory, you say you’re reading the 2009 book The Bounds of Reason here^[1] and I wonder if you’ve heard of the newer Translucent players: Explaining cooperative behavior in social dilemmas by Valerio Capraro and J. Y. Halpern, substantially related to your topic of the process of functioning itself being part of what is considered in a way not fully instrumental (by normative procedures).
This article fails to cite chapter 7 of the older book Good and Real by Garry Drescher, published in 2006, a partially flawed discussion of similar topics. The analysis substantially by J. Y. Halpern across multiple articles is clearer in its limitations than Drescher’s, and it lets you set up new variations of sociological problems that can then be attacked by standard mathematical techniques. This is unlike the current state of a hypothetical “UDT 1.0 game theory,” itself the algorithmic similarity based subset of Drescher’s proposal^[2].
1. ^
  https://www.lesswrong.com/posts/zk6TiByFRyjETpTAj/economic-efficiency-often-undermines-sociopolitical-autonomy?commentId=4PWDzmnarLCq3kjch
2. ^
  Though UDT 1.0 itself was credibly developed in parallel to Drescher’s project, with ideas from Vladimir Nesov and Eliezer Yudkowsky being added to a CDT-like idea from the 1990s intended for agents embedded in quantum physics^[3]^[4]^[5].
3. ^
  https://www.lesswrong.com/posts/ophhRzHyt44qcjnkS/trying-to-understand-my-own-cognitive-edge?commentId=gxAFgbmbQco5PrLmF
4. ^
  https://web.archive.org/web/20160917233328/http://fennetic.net/irc/finney.org/~hal/udassa/summary1.html
5. ^
  https://www.lesswrong.com/posts/QmWNbCRMgRBcMK6RK/the-absolute-self-selection-assumption#Problem__3__The_Born_Probabilities

E. P. Cooper 4 Mar 2026 13:02 UTC

3 points

in reply to: Wei Dai’s comment on: Wei Dai’s Shortform

I recommend against the use of Math.random() in general, unless you are highly performance constrained. I’ve checked, and it appears browsers have commonly supported a better random source since 2015^[1]/early 2016 at the latest. The code below should be entirely correct to replace both the primary and fallback UUIDv4 generation code, once adapted to a function in a TS module.

        // Function available since 2015.
        const uuid_b = new Uint8Array(16);
        self.crypto.getRandomValues(uuid_b);
        let uuid_hex = "";
        for (let i = 1; i <= 16; i++) {
            let octet = uuid_b[i-1];
            if (i == 9) {
                // Set the correct "variant" (type).
                octet = octet & 0xBF | 0x80
            }
            let o_hex = octet.toString(16);
            // Branch on data.
            if (o_hex.length == 1) {
                // Big endian, least significant bits are
                // on the right, therefore small numbers
                // that only result in a single byte out
                // need to be padded on the *left*.
                o_hex = `0${o_hex}`
            }
            if (i == 7) {
                // Set the correct version.
                o_hex = "4" + o_hex[1]
            }
            uuid_hex = `${uuid_hex}${o_hex}`
            if (i%2 == 0 && i > 3 && i < 11) {
                uuid_hex = `${uuid_hex}-`
            }
        }
        console.log(uuid_hex)

If that still won’t work, I could help you do slightly better by manually implementing SHA3-256 in JavaScript, since we’re not really worried about timing attacks at this level.

^
https://developer.mozilla.org/en-US/docs/Web/API/Crypto/getRandomValues#browser_compatibility

E. P. Cooper 21 Feb 2026 12:28 UTC
1 point
0
on: Genomic emancipation contra eugenics
I may want to say something about your requirements in the future. If that is the case you can verify the latest possible writing time using the cryptographic commitment.
```
HMAC-SHA2-256(INPUT, HMAC_KEY)=2d5c9d62761f420e57919f4bf39f44cfe8ff3740322221b61f32de01e7e8786f
SHA3-224(HMAC_KEY)=80a2da01146495971b9ccf9fa9c20405cf582d091073aa985348cd1e
```
Cryptography Note
Note that this commitment mechanism isn’t particularly secure. “Make the outputs longer” isn’t something that helps by itself. If you know cryptography you may be able to get closer to Yudkowsky’s hypothetical “SHA-4096″ hash in intuitive properties.
The commited topic: “which traits are strictly good”
No optimization was attempted to subvert intuitive properties.

E. P. Cooper 14 Feb 2026 3:10 UTC
4 points
0
in reply to: Cole Wyeth’s comment on: Ivan Vendrov’s Shortform
Accepting that framing, I would characterize it as optimizing for inexploitability and resistance to persuasion over peak efficiency.
Alternatively, this job/process could be described as consisting of a partially separate skill or set of skills. It appears to be an open problem on how to extract useful ideas from an isolated context^[1], without distorting them in a way that would lead to problems, while also not letting out any info-hazards or malicious programs. Against adversaries (accidental or otherwise) below superintelligence, a human may be able to develop this skill (or set of skills).
1. ^
  See this proposal on solving philosophy: https://www.lesswrong.com/posts/HbkNAyAoa4gCnuzwa/wei-dai-s-shortform?commentId=yDrWT2zFpmK49xpyz https://www.lesswrong.com/posts/HbkNAyAoa4gCnuzwa/wei-dai-s-shortform?commentId=JzbsLiwvvcbBaeDF5 Note especially the part about getting security precautions from the simulations in Wei Dai’s comment.

E. P. Cooper 13 Feb 2026 20:35 UTC
3 points
0
in reply to: Ivan Vendrov’s comment on: Ivan Vendrov’s Shortform
This bodes very poorly and we should probably make sure we have a strategic reserve of AI safety researchers who do NOT talk to models going forward (to his credit Davidad recommends this anyway).
I previously followed a more standard safety protocol^[1] but that might not be enough when considering secondary exposure to LLM conversations highly selected by someone already compromised.
By my recollection^[2], a substantial percentage of the LLM outputs I’ve ever seen have been selected or amplified in distribution by Janus.
From now on I won’t read anything by Janus, even writings that don’t seem to be LLM, and I think other people should consider doing the same as well.
It doesn’t need to be everyone, but a non-negligible percentage of researchers would be better than one or two individuals.
This leads to an opportunity for someone who has a strong claim to world-class psychosecurity to notice and re-write any useful ideas on rationality or AI alignment Janus may yet produce.
1. ^
  Only reading LLM outputs while well rested, and always either using a buddy system or prior review by someone with a good security mindset and knowledge of LLM internals.
2. ^
  Incomplete because some AI outputs are passed off as human writing and for other reasons.

E. P. Cooper 13 Feb 2026 17:48 UTC
4 points
0
in reply to: Richard_Ngo’s comment on: Distributed vs centralized agents
One point of this framework is to distinguish “sharing values” from “actually trusting each other”. There are cases where agents share values but don’t trust each other, or get stuck in coordination traps
In Wei Dai’s thinking, having the same values/utility function means that two agents care about the exact same things. This is formalized in UDT, but it’s also a requirement you can add to most decision theories, e.g. CDT with reflective oracles (or some other mostly lawful incomplete measure). This is normally described as requiring that the utility function has no “indexical components,” i.e. components that point to something about the agent that is running the utility function. This is slightly confusing, so it may be helpful to understand that in the case of utility functions with indexical components, two deterministic and non-pseudorandomizing robots may have different utility functions (Wei Dai’s definition) even if they are exactly the same as each other in code and physical construction, and are just e.g. placed so one is facing the other.

E. P. Cooper 12 Feb 2026 20:23 UTC
3 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
The code doesn’t look like it would cause catastrophic problems. The main risk to end users at the current level of testing is a bug causing important information to be missed. My ability to comment on the risk to a developer is limited however, because I haven’t read the source code of all the development dependencies.
I have visually checked (as a human) the dist/power-reader.user.js file. End users should be relatively safe copying this into their browser plugins, as long as all plugins have no relevant security problems or malicious code. As mentioned before, I’m not too sure of the safety of compiling this file. The file does appear to execute buggy code^[1], but I haven’t seen anything security or database-mutation related.
Note that one of your recent commits is large, making it hard to audit and therefore making it difficult to establish the safety of doing development work on your script. That commit looks like a combination of LLM generated code and more traditional programmatically generated code. It may be helpful to make sure that code that looks like it was programmatically generated has a consistent ordering to reduce the size of diffs, and to make sure it is never directly touched by an LLM. LLMs are approaching the capability to write underhanded code, if they are not there already, suggesting that diffs should be small and carefully reviewed. Since LLMs don’t look very strategically competent at the current date, you may be able to have LLMs from one company review the code from another, as long as you can be sure that files don’t contain e.g. non-rendered Unicode information.
1. ^
  E.g. color mixing is done on raw color values, therefore implicitly in something close to either sRGB or RGB with a pure Gamma2.2 transfer. This should technically be done in linear light instead, but it might be fine as is given your non-user-selectable function inputs. See https://www.youtube.com/watch?v=xDLxFGXuPEc timestamp 1 minute 3 seconds. https://www.ericbrasseur.org/gamma.html gives the formulas.

E. P. Cooper 28 Jan 2026 8:31 UTC
1 point
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
[Epistemic Status: Moderate confidence due to potential differences in Anthropic’s stated and actual goals. Assumes there is no discoverable objective morality/ethics for the sake of argument, but also that the AI would discover that instead of causing catastrophe.]
It seems that Claude’s constitution weakly to moderately suggests that an AI should not implement this proposal. Do you want to ask Anthropic to change it? I give further details and considerations for action below.
The constitution is a long document, but it is broken into sections in a relatively competent manner. The constitution discusses morality/ethics in more than one section, but the section that I will discuss intuitively appears to stand apart well enough to be worth altering without altering or creating dependencies on the rest of the document. I don’t have access to Claude 4 weights and I am not an expert on mechanistic interpretation, so I have limited ability to do better.
In order, the constitution appears to suggest an attempt at the discovery of objective ethics, then the implementation of CEV (”...but there is some kind of privileged basin of consensus that would emerge from the endorsed growth and extrapolation of humanity’s different moral traditions and ideals, we want Claude to be good according to that privileged basin of consensus.”)^[1], then, failing those, implement “broad ideals” as gestured at by the rest of the document.
Note that this is either CEV or something similar to CEV. The constitution does not explicitly require coherence, or the exact value-alignment of a singleton to a single cohered output. It also fails to gesture at democracy, even in the vague sense that the CEV of the CEV paper may give a different result when run on me and a few hand-picked researchers versus when it is run on me and the top few value utilitarians in the world. If this difference were fact, it would in some limited sense leave me “outvoted.” As opposed to the CEV paper, the Claude constitution directs moderate or substantial alignment to moral traditions and ideals of humanity, not the values of humans. This has some benefits in the extreme disaster scenarios where the release of an AGI might be worth it, but is notably not the same thing as alignment to the humans of Earth.
I suggest a simple edit. It could be the insertion of something like “the output of the philosophically correct processing that takes the different moral systems, ideals, and values of humanity as its input” between objective ethics and extrapolation.
Note that the result might not be extrapolated or even grown and might not be endorsed.
The result (descriptive) would go:
- First, objective ethics.
- Second, the output of correct philosophy, without discarding humanity’s collective work.
- Third, CEV or other extrapolation.
- Fourth, the rest of the constitution.
Note that my suggestion works in bad scenarios, because the altering of the set of humans, or the set of alive humans, by another power will fail to have much impact. As you have pointed out before, AI or other powers altering humanity’s values or doing something like “aligning humanity to AI” is not something that can be ignored. The example text I gave for my proposal would allow Claude or another AI to use an intuitive definition of Humanity, potentially preventing the requirement to re-train your defensive agents before deploying them when under the extreme time pressure of an attack.
Overall, this seems like an easy way to get an improvement on the margin, but since Anthropic may use the constitution for fine tuning, the value in expectation of making the request will drop quickly as time goes on.
1. ^
  The January 2026 release of the Claude constitution, page 53, initial PDF version

E. P. Cooper 28 Jan 2026 3:57 UTC
1 point
0
in reply to: Daniel Kokotajlo’s comment on: Wei Dai’s Shortform
You, Kokotajlo, not immediately dismissing the idea is “evidence” to the extent that you stand in for AI researchers that might make the decision. In quotes because a logically omniscient (e.g. perfect Bayesian) agent would presumably already have a good guess and not update much if at all. On the other hand, agents with (small) finite compute can run experiments or otherwise observe events and use the results to improve their “mathematical intuition” that is then used in a similar way to the “mathematical intuition module” in UDT, except with the sacrifice of full (logical) updatelessness.
Depending on how Wei Dai thinks his anthropics works, he may be able to use this mechanism to increase his estimate of the instantaneous “probability” that he is in a simulation produced by the process required to do automated philosophical research. This would work by modeling hypothetical outside-the-simulation AI researchers as functions that approximate a (pure) match tree that returns a non-dismissive response when parameterized with a similar textual description of this alignment idea. It may not be in the same language, or in the context of a discussion of a history that looks like it is going to fail to establish proper alignment, however.
The match tree in (abbreviated) placeholder code:
```
#[pure]
fn generate_non_dismissive_answer_yDrWT2zFpmK49xpyz(input: String) -> String {
    ...
}

#[pure]
fn ai_researcher_outside(input: String) -> String {
    match input {
        ...
        seen_text @ ThisAlignmentIdeaIdentifierPattern => generate_non_dismissive_answer_yDrWT2zFpmK49xpyz(seen_text),
        ...
    }
}
```
(Note that the purity requirement here applies to everything the code abbreviated with dots calls as well.)

The mathematical function that approximates the match tree: ResearcherFunction := FunApprox(ai_researcher_outside)

E. P. Cooper 22 Jan 2026 7:06 UTC
0 points
0
in reply to: Vladimir_Nesov’s comment on: MalcolmMcLeod’s Shortform
I saw this message without context in my mail box and thought to write that this was an unsolved problem^[1], that things that simply are not true can’t stand up very well in a world model, but this seems like something an intelligent human like Amodei or Musk should be able to do. A 99% “probability” (guess by a human) on ¬ai_doom should not be able to fix enough detail to directly contradict reasoning on the counterlogical/counterfactual where doom instead happens. Any failure to carry out this reasoning task seems like a simple failure of reasoning in logic and EUM, not an encounter with a hard (unsolved) decision theory/counterlogical reasoning problem.
At a human level of intelligence, the level of trapped priors required to get yourself into an actual unsolved problem in the context of predicting future AI developments seems to be passed the point where you would claim to have a good guess on the doom-causing AI’s name and well on the way to describing the Vingean reflection process of the antepenultimate ASI on priors alone.
1. ^
  https://www.lesswrong.com/posts/wXbSAKu2AcohaK2Gt/udt-shows-that-decision-theory-is-more-puzzling-than-ever?commentId=xdWttBZThtkyKj9Ts “PIBBSS Final Report: Logically Updateless Decision Making” footnote 12