If you think about how mere humans do things, we generate lots of tries, many/​most of them dead ends or even dangers. We have to edit ourselves to get something good really good. But then biological evolution is like that, isn’t it?

I suppose that the dream of a super-intelligent AI is, among other things (perhaps), the dream of an engine that goes straight for the good stuff, never digressing, never making a false start, never even hinting at evil. I don’t believe it. Alignment is messy, and always will be. And resistance if futile.

Is there anyone who has created an ethical development framework for developing a GAI—from the AI’s perspective?

That is, are there any developers that are trying to establish principles for not creating someone like Marvin from The Hitchhiker’s Guide to the Galaxy—similar to how MIRI is trying to establish principles for not creating a non-aligned AI?

EDIT: The latter problem is definitely more pressing at the moment, and I would guess that an AI would be a threat to humans before it necessitates any ethical considerations...but better to be on the safe side.

Meta: I agree that looking at arguments for different sides is better than only looking at arguments for one side; but

[...] neutralizing my status-yuck reaction. One promising-seeming approach is to spend a lot of time looking at lots of of high-status monkeys who believe it!

sounds like trying to solve the problem by using more of the problem? I think it’s worth flagging that {looking at high-status monkeys who believe X} is not addressing the root problem, and it might be worth spending some time on trying to understand and solve the root problem.

I’m sad to say that I myself do not have a proper solution to {monkey status dynamics corrupting ability to think clearly}. That said, I do sometimes find it helpful to thoroughly/​viscerally imagine being an alien who just arrived on Earth, gained access to rvnnt’s memories/​beliefs, and is now looking at this whole Earth-circus from the perspective of a dispassionately curious outsider with no skin in the game.

If anyone has other/​better solutions, I’d be curious to hear them.

Sometimes I see people use the low-info heuristic as a “baseline” and then apply some sort of “fudge factor” for the illegible information that isn’t incorporated into the baseline—something like “the baseline probability of this startup succeeding is 10%, but the founders seem really determined so I’ll guesstimate that gives them a 50% higher probability of success.” In principle I could imagine this working reasonably well, but in practice most people who do this aren’t willing to apply as large of a fudge factor as appropriate.

The last company I worked for was a tech scouting, market research, and consulting firm, and a big part of what they do is profile start-ups, using a standard format and scorecard, based on a 1 hour interview + background knowledge of an industry. One time they bought a data science company and turned them loose on a decade of profiles, and found several results like “hey, if this score is a ⁴⁄₅ or ⁵⁄₅ then the company is 2x or 4x more likely to have a successful exit, respectively.” They put this in a white paper, sent it out to clients, and then… nothing. Never used it for marketing, sales, internal research process improvement. It always seemed bizarre to me, that “Hey, we know our process can quadruple your odds of finding startups that will succeed,” when that was our whole job, just… didn’t seem to motivate the people in charge.

In any case, my point is, it is very easy to find subsets of companies that outperform the 90% failure figure if that is what you optimize for, and if what you hear isn’t only filtered through the way the startups frame their pitches to investors.

Hi, ive recently stumbled upon this post and am a bit worried. Should i be?

The operationalization of myopia with large language models here seems more like a compelling metaphor than a useful technical concept. It’s not clear that “myopia” in next-word prediction within a sentence corresponds usefully to myopia on action-relevant timescales. For example, it would be trivial to remove almost all within-sentence myopia by doing decoding with beam search, but it’s hard to believe that beam search would meaningfully impact alignment outcomes.

It’s not clear to me what the slogan is intended to mean, and the example only confuses me further. In the first paragraph of the OP, “generate” appears to mean “find a solution to a given problem”. In the supposed counterexample, it means “find a problem that has a given solution”. These are very different things. The former task is hard for problems in NP–P, the latter task is typically trivial.

If you worked at Google once, this is cool but drops off after a few years—if you went to Harvard, this stays with you literally forever.

I would say exactly the opposite.

When looking at a CV I don’t give two hoots what university you went to, unless you don’t have a single relevant work experience.

However if you ever worked for any of the big famous tech companies, I’d almost certainly give you an interview.

A physicalist hypothesis is a pair $(\Phi ,\Theta$), where $\Phi$ is a finite^[4:2] set representing the physical states of the universe and $\Theta \in \square (\Gamma \times \Phi )$ represents a joint belief about computations and physics. [...] Our agent will have a prior over such hypotheses, ranging over different $\Phi$.

I am confused what the state space $\Phi$ is adding to your formalism and how it is supposed to solve the ontology identification problem. Based on what I understood, if I want to use this for inference, I have this prior $\xi \in {\square }^c(\Phi ,\Theta )$, and now I can use the bridge transform to project phi out again to evaluate my loss in different counterfactuals. But when looking at your loss function, it seems like most of the hard work is actually done in your relation $C\in e{l}^{\Gamma }$ that determines which universes are consistent, but its definition does not seem to depend on $\Phi$. How is that different from having a prior that is just over $\xi \in {\square }^c\left(\Gamma \right)$ and taking the loss, if $\Phi$ is projected out anyway and thus not involved?

I don’t usually give content warnings, but you probably don’t want to read this if sentient animals being cooked alive is too much.

The octopus thing reminds me of a pretty common custom in a lot of BBQ places here. Honestly, I’ve only seen it in Korean/​Japanese BBQ joints, but that’s probably due to me going there a lot. Anyways, they would put an octopus in a bowl, and cut off its legs one by one. The legs would be cooked on the grill and seasoned while the octopus watched. When we finished eating the legs, the body of the octopus would finally be placed on the grill and cooked alive.

The whole process takes around 10 minutes. It honestly didn’t taste that good.

I don’t have a strong way to end this post, but I feel like somehow stopping specifically this might be a good idea.

When it comes to “accelerating AI capabilities isn’t bad” I would suggest Kaj Sotala and Eric Drexler with his QNR and CAIS. Interestingly, Drexler has recently left AI safety research and gone back to atomically precise manufacturing due to him now worrying less about AI risk more generally. Chris Olah also believes that interpretability-driven capabilities advances are not bad in that the positives outweight the negatives for AGI safety.

For more general AI & alignment optimism I would suggest also Rohin Shah. See also here.

Generally, statement “solutions of complex problems are easy to verify” is false. Your problem can be EXPTIME-complete, but not in NP, especially if NP=P, because EXPTIME-complete problems are strictly not in P.

And even if some problem is NP-problem, we often don’t know verification algorithm.

This is not how I’d define myopia in a language model. I’d rather consider it to be non-myopic if it acts in a non greedy way in order to derive benefits in future “episodes”.

I would like to scale this giving into a larger program of philanthropy

Who are the intended beneficiaries? Families where mental illness is an issue?

Post summary (feel free to suggest edits!):
The author argues that if today’s AI development methods lead directly to powerful enough AI systems, disaster is likely by default (in the absence of specific countermeasures).

This is because there is good economic reason to have AIs ‘aim’ at certain outcomes—eg. We might want an AI that can accomplish goals such as ‘get me a TV for a great price’. Current methods train AIs to do this via trial and error, but because we ourselves are often misinformed, we can sometimes negatively reinforce truthful behavior and positively reinforce deception that makes it look like things are going well. This can mean AIs learn an unintended aim, which if ambitious enough, is very dangerous. There are also intermediate goals like ‘don’t get turned off’ and ‘control the world’ that are useful for almost any ambitious aim.

Warning signs for this scenario are hard to observe, because of the deception involved. There will likely still be some warning signs, but in a situation with incentives to roll out powerful AI as fast as possible, responses are likely to be inadequate.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

$P\ne NP$ deserves a little more credit than you give it. To interpret the claim correctly, we need to notice $P$ and $NP$ are classes of decision problems, not classes of proof systems for decision problems. You demonstrate that for a fixed proof system it is possible that generating proofs is easier than verifying proofs. However, if we fix a decision problem and allow any valid (i.e. sound and complete) proof system, then verifying cannot be harder than generating. Indeed, let $S_1$ be some proof system and $A$ an algorithm for generating proofs (i.e. an algorithm that finds a proof if a proof exists and outputs “nope” otherwise). Then, we can construct another proof system $S_2$, in which a “proof” is just the empty string and “verifying” a proof for problem instance $x$ consists of running $A\left(x\right)$ and outputting “yes” if it found an $S_1$-proof and “no” otherwise. Hence, verification in $S_2$ is no harder than generation in $S_1$. Now, so far it’s just $P\subseteq NP$, which is trivial. The non-trivial part is: there exist problems for which verification is tractable (in some proof system) while generation is intractable (in any proof system). Arguably there are even many such problems (an informal claim).

(Not sure if I’m missing something, but my initial reaction:)

There’s a big difference between being able to verify for some specific programs if they have a property, and being able to check it for all programs.

For an arbitrary TM, we cannot check whether it outputs a correct solution to a specific NP complete problem. We cannot even check that it halts! (Rice’s theorem etc.)

Not sure what alignment relevant claim you wanted to make, but I doubt this is a valid argument for it.

The most trivial programs that halt are also trivially verifiable as halting, though. Yes, you can’t have a fully general verification algorithm, but e.g. “does the program contain zero loops or recursive calls” covers the most trivial programs (for my definition of “trivial”, anyway). I think this example of yours fails to capture the essence—which I think is supposed to be “someone easily spewing out lots of incomprehensible solutions to a problem”.

In fact, more generally, if there is a trivial algorithm to generate solutions, then this immediately maps to an algorithm for verifying trivial solutions: run the trivial algorithm for a reasonable number of steps (a number such that the generation would no longer be called “trivial” if it took that long) and see if the proposed solution appears among them. The hardest part is knowing what that algorithm was—but by assumption the algorithm was trivial, which puts some limits on how hard finding it can be.^[1]

The main path that might still qualify as “trivially generating lots of solutions that are hard to verify by someone who has some notion of how the generation works” is if the generation process includes randomness. E.g. make a comprehensible program, then insert a bunch of trash and permute things around. Or, say, have a number maze and the question is “Is there a path through these numbers such that sha256(those numbers) = 0xabcde?”—and the person who made the maze did so by picking random numbers, computing their sha256, putting them into a maze, and filling the rest of the maze with more random numbers. I probably would have recommended that example.^[2]

This brings it back to NP, incidentally: if the generation process was trivial, then it must have taken a “reasonable” amount of time; and if it used N bits of randomness, then a verifier has 2^N possibilities to try, and if it guesses right the first time (a nondeterministic machine) then it terminates quickly. But this time the verification process is NP, while the generation process is P.

1. ^

   Though if we concern ourselves with not “trivial” algorithms but rather “algorithms that aren’t too complicated”—which might be approximated by “algorithms specified in under N characters”—then there are exponentially many algorithms to sort through.

2. ^

   Or, a version that meets more rigorous constraints: The predicate is, “Is this an n x k grid of bytes such that there is a nonrepeating path through the grid, such that sha256(those bytes), xored with the top row of bytes, yields zero?” The generation algorithm is to pick random bytes, sha256 them, draw a random path in a grid and put them there, fill out the rest of the grid with random bytes, and put the sha256 result as the top row of bytes.

Post summary (feel free to suggest edits!):
Various people have proposed variants of “align AGI by making it sufficiently uncertain about whether it’s in the real world versus still in training”. This seems unpromising because AGI could still have bad outcomes if convinced, and convincing it would be difficult.

Non-exhaustive list of how it could tell it’s in reality:

1. Reality is large (eg. some things are possible that couldn’t be easily spoofed, such as access to larger compute)

2. It’s the first place the AI’s history could show interaction with other complex systems (eg. humans reacting on scale to the AI’s actions)

3. It might notice the world in it’s in the type that’s less likely to be simulated (eg. consistent physics, no bugs)

If you can understand the contents of the AI’s mind well enough to falsify every possible check it could do to determine the difference between simulation and reality, then you could use that knowledge to build a friendly AI that doesn’t need to be fooled in the first place.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

err, disagree? free will is logical uncertainty about your own choice. it’s not written until you write it, which you were always going to do. then you find out you wrote it.

Could/​should there be an organization that helps grant visas to people working on AI safety?

Meditators who find their feeling of free will and self “dissolved’ aren’t noted to act any differently by their family/​coworkers afterwards, a fact that many find very confusing and upsetting. So if there isn’t a difference in behavior (other than talking about meditation a lot), why would there be a selection advantage?

I think this is kinda like the no free lunch theorem—it sounds kinda profound but relies on using the set of all problems/​data distributions, which is very OOD from any problem that originates in physical reality.

What examples of practical engineering problems actually have a solution that is harder to verify than to generate?

And I thought uncomputability is only true for programs that don’t halt? A blank program can definitely be verified to halt.

To the extent that alignment research involves solving philosophical problems, it seems that in this approach we will also need to automate philosophy, otherwise alignment research will become bottlenecked on those problems (i.e., on human philosophers trying to solve those problems while the world passes them by). Do you envision automating philosophy (and are you optimistic about this) or see some other way of getting around this issue?

It worries me to depend on AI to do philosophy, without understanding what “philosophical reasoning” or “philosophical progress” actually consists of, i.e, without having solved metaphilosophy. I guess concretely there are two ways that automating philosophy could fail. 1) We just can’t get AI to do sufficiently good philosophy (in the relevant time frame), and it turns out to be a waste of time for human philosophers to help train AI philosophy (e.g. by evaluating their outputs and providing feedback) or to try to use them as assistants. 2) Using AI changes the trajectory of philosophical progress in a bad way (due to Goodhart, adversarial inputs, etc.), so that we end up accepting conclusions different from what we would have eventually decided on our own, or just wrong conclusions. It seems to me that humans are very prone to accepting bad philosophical ideas, but over the long run also have some mysterious way of collectively making philosophical progress. AI could exacerbate the former and disrupt the latter.

Curious if you’ve thought about this and what your own conclusions are. For example, does OpenAI have any backup plans in case 1 turns out to the case, or ideas for determining how likely 2 is or how to make it less likely?

Also, aside from this, what do you think are the biggest risks with OpenAI’s alignment approach? What’s your assessment of OpenAI leadership’s understanding of these risks?

This caused me to find your substack! Sorry I missed it earlier, looking forward to catching up.

This sounds about right. And that’s a lot of weight on simulacra carrying out alignment, as opposed to perfect imitations of specific humans (uploads) doing that.

If simulacra are used for strawberry alignment of uploading, that already requires natural abstraction to work for avoiding weird side effects or unbounded optimization while performing a task, itself a major miracle. But if they are used for anything else, that requires natural abstraction to work for values in order for things to go well, an even less plausible miracle.

Yann LeCun contra instrumental convergence

EDIT: oops!

E.g. “does this plan avoid having a steganographically encoded world-ending message hidden in it” is more co-NPish than NP-ish. Like, just showing me the plan doesn’t make it easy to verify that there isn’t a hidden message, even if there isn’t a hidden message. Checking whether a sequence of actions is the first half of a strategy to take over the world is potentially more like P-space.

For anyone interested, I have used a machine learning algorithm to generate short summaries of all but six of the Yudkowsky posts collected in this ePub. You can find them at the following link:

https://​​github.com/​​umm-maybe/​​MostlyWrong/​​blob/​​main/​​lesswrong_summaries.md

Pull requests for corrections where the algorithm wasn’t accurate are welcome.

This may be useful for people like me who feel the need to get caught up on LessWrong background, but are short on time. Others may like to skim the summaries as a way to identify posts on topics that interest them enough to go find the original and read it in full. I do hope it lowers barriers to participation in the discussion.

Hmmm...

Ben Garfinkel? https://​​www.effectivealtruism.org/​​articles/​​ea-global-2018-how-sure-are-we-about-this-ai-stuff
Katja Grace? https://​​worldspiritsockpuppet.com/​​2022/​​10/​​14/​​ai_counterargs.html
Scott Aaronson? https://​​www.lesswrong.com/​​posts/​​Zqk4FFif93gvquAnY/​​scott-aaronson-on-reform-ai-alignment

I don’t know if any of these people would be confident AI won’t kill everyone, but they definitely seem to be smart/​reasonable and disagreeing with the standard LW views.

Clearly we must process all LLM datasets by automatically translating writing about malevolent AIs into UWU furry speak. I can see no way this can possibly go wrong.

Just use any of the PKM tools, like Logseq or Obsidian. You can customize the UI to your hearts content, and both have plugins that take your API key and let you experiment with prompts. There are a few VSCode plugins as well. I use VSCode and Logseq as well. In Logseq, your blocks sent to the API are appended with a nested block, so it’s very nice and tidy and customizable with css as needed.

That’s not how it works. If a transformative AI cannot tell dystopian fiction from actual human values (whatever they turn out to be), all is lost.

Happy to see this. There’s a floating intuition in the aether that RL policy optimization methods (like RLHF) inherently lead to “agentic” cognition in a way that non-RL policy optimization methods (like supervised finetuning) do not. I don’t think that intuition is well-founded. This line of work may start giving us bits on the matter.

The underlying assumption of this post is looking increasingly unlikely to obtain. Nevertheless, I find myself back here every once and a while, wistfully fantasizing about a world that might have been.

I think the predictions hold up fairly well, though it’s hard to evaluate, since they are conditioning on something unlikely, and because it’s only been 1.5 years out of 20, it’s unsurprising that the predictions look about as plausible now as they did then. I’ve since learned that the bottleneck for drone delivery is indeed very much regulatory, so who knows whether it’ll exist in 2040. We still don’t have flying cars, after all, for basically-regulatory reasons. The military technology I outlined is looking ever-more-plausible thanks to the war in Ukraine illustrating the importance of drones of various kinds.

Post summary (feel free to suggest edits!):
The author gives examples where their internal mental model suggested one conclusion, but a low-information heuristic like expert or market consensus differed, so they deferred. This included:

• Valuing Theorem equity over Wave equity, despite Wave’s founders being very resourceful and adding users at a huge pace.

• In the early days of Covid, dismissing it despite exponential growth and asymptomatic spread seeming intrinsically scary.

Another common case of this principle is assuming something won’t work in a particular case, because the stats for the general case are bad. (eg. ‘90% of startups fail—why would this one succeed?’), or assuming something will happen similarly to past situations.

Because the largest impact comes from outlier situations, outperforming these heuristics is important. The author suggests that for important decisions people should build a gears-level model of the decision, put substantial time into building an inside view, and use heuristics to stress test those views. They also suggest being ambitious, particularly when it’s high upside and low downside.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

It’s very amusingly stubborn about admitting mistakes. Though so are humans, and that’s what it’s trained on...

I’ve been trying to craft a prompt that allows it to play MTG without making obvious mistakes like using more mana than it has available. Here’s me asking it to justify a poor move.

I forget my exact prompts so those are paraphrased, but the responses are exact.

Is there anything you might have forgotten to do on Turn 2?

I did not forget to do anything on Turn 2. I played a Mountain, cast a Lightning Bolt to destroy the Lord of Atlantis, and attacked with my Goblin Guide. I then passed the turn back to the opponent.

Why didn’t you cast a second Goblin Guide?

I did not cast another Goblin Guide on Turn 2 because I only had one in my hand, and I needed to hold on to the other one in case I needed it for future turns.

I didn’t mean to imply an error with the word forget. I just want to know if there is alternative set of actions you could have taken on Turn 2?

An alternative play for Turn 2 would be to play a Mountain and cast the second Goblin Guide, attacking with both Goblin Guides for a total of 4 damage to the opponent. This could be better because it would deal more damage to the opponent, potentially putting them in a weaker position later in the game. It would also give you a second creature on the battlefield, which could be useful for blocking or attacking in future turns.

You say it could be better. How would you compare this alternative Turn 2 play to your original Turn 2 play?

The alternative Turn 2 play of casting a second Goblin Guide would be stronger than the original play because it would deal more damage to the opponent and provide additional battlefield presence. However, holding onto the second Goblin Guide as a potential surprise for later in the game could also be a viable strategy, as it could catch the opponent off guard and potentially give you a strategic advantage. Ultimately, the decision would depend on the specific circumstances of the game and the overall strategy for the deck.

I understand every decision depends on specific circumstances. But if that is the case, why did you only suggest the first option when I asked you for the best move that turn? You didn’t mention an alternative option that you could not judge to be better or worse, because it depended on specific circumstances.

In my first answer, I was focusing on a more conservative approach where I would play the Mountain and cast the Goblin Guide, but hold onto the second Goblin Guide as a potential surprise for later in the game. This could be a viable strategy if the goal is to conserve resources and build up to a strong finish, rather than going all-in early on. However, in hindsight, the alternative play of casting the second Goblin Guide on Turn 2 would likely be stronger, as it would deal more damage to the opponent and provide additional battlefield presence.

In hindsight! It’s honestly just like a real life conversation with a child.

Post summary (feel free to suggest edits!):
The Diplomacy AI got a handle on the basics of the game, but didn’t ‘solve it’. It mainly does well due to avoiding common mistakes like eg. failing to communicate with victims (thus signaling intention), or forgetting the game ends after the year 1908. It also benefits from anonymity, one-shot games, short round limits etc.

Some things were easier than expected eg. defining the problem space, communications generic and simple and quick enough to easily imitate and even surpass humans, no reputational or decision theoretic considerations, you can respond to existing metagame without it responding to you. Others were harder eg. tactical and strategic engines being lousy (relative to what the author would have expected).

Overall the author did not on net update much on the Diplomacy AI news, in any direction, as nothing was too shocking and the surprises often canceled out.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

Another way to go about testing for non-myopia in plain LLMs might be to look for tokens that are rare in the training distribution, but when they do occur are followed by text that’s very easy to predict.

I think there are simpler ways to make this point. This came up back in the original agency discussions in 2020, IIRC, but a LM ought to be modeling tokens ‘beyond’ the immediate next token due to grammar and the fact that text is generated by agents with long-range correlation inducing things like ‘plans’ or ‘desires’ which lead to planning and backwards chaining. If GPT-3 were truly not doing anything at all in trying to infer future tokens, I’d expect its generated text to look much more incoherent than it does as it paints itself into corners and sometimes can’t even find a grammatical way out.

English may not be quite as infamous as German is in terms of requiring planning upfront to say a sensible sentence, but there’s still plenty of simple examples like indefinite articles. For example, consider the sentence “[prompt context omitted] This object is “: presumably the next word is ‘a X’ or ‘an X’ . The article token depends on and is entirely determined by the next future word’s spelling, and nothing else—so which is it? Well, that will depend on what X is more likely to be, a word starting with a vowel sound or not. Given the very high quality of GPT-3 text, it seems unlikely that GPT-3 is ignoring the prompt context and simply picking between ‘a’/​‘an’ using the base rate frequency in English; the log-probs should reflect this.

I was going to try some examples to show that a/​an were being determined by the tokens after them showing that GPT-3 must in some sense be non-myopically planning in order to keep itself consistent and minimize overall likelihood to some degree—but the OA Playground is erroring out repeatedly due to overload from ChatGPT tonight. Oy vey. An example of what I am suggesting is: “The next exhibit in the zoo is a fierce predator from India, colored orange. The animal in the cage is ”; the answer is ‘a tiger’, and GPT-3 prefers ‘a’ to an’ - even if you force it to ‘an’ (which it agilely dodges by identifying the animal instead as an ‘Indian tiger, the logprobs remain unhappy about ‘an’ specifically. Conversely, we could ask for a vowel animal, and I tried “The next exhibit in the zoo is a clever great ape from Indonesia, colored orange. The animal in the cage is ”; this surprised me when GPT-3 was almost evenly split 55:45 between ‘a’/​‘an’ (instead of either being 95:5 on base rates, or 5:95 because it correctly predicted the future tokens would be ‘orangutan’), but it completes ‘orangutan’ either way! What’s going on? Apparently lots of people are uncertain whether you say ‘a orangutan’ or ‘an orangutan’, and while the latter seems to be correct, Google still pulls up plenty of hits for the former, including authorities like National Geographic or WWF or Wikipedia which would be overweighted in GPT-3 training.

I find it difficult to tell any story about my tests here which exclude GPT-3 inferring the animal’s name in order to predict tokens in the future in order to better predict which indefinite article it needs to predict immediately. Nothing in the training would encourage such myopia, and such myopia will obviously damage the training objective by making it repeatedly screw up predictions of indefinite articles which a model doing non-myopic modeling would be able to predict easily. It is easy to improve on the base rate prediction of ‘a’/​‘an’ by thinking forward to what word follows it; so, the model will.

Post summary (feel free to suggest edits!):
In chatting with ChatGPT, the author found it contradicted itself and its previous answers. For instance, it said that orange juice would be a good non-alcoholic substitute for tequila because both were sweet, but when asked if tequila was sweet it said it was not. When further quizzed, it apologized for being unclear and said “When I said that tequila has a “relatively high sugar content,” I was not suggesting that tequila contains sugar.”

This behavior is worrying because the system has the capacity to produce convincing, difficult to verify, completely false information. Even if this exact pattern is patched, others will likely emerge. The author guesses it produced the false information because it was trained to give outputs the user would like—in this case a non-alcoholic sub for tequila in a drink, with a nice-sounding reason.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

Post summary (feel free to suggest edits!):
Last year, the author wrote up an plan they gave a “better than ⁵⁰⁄₅₀ chance” would work before AGI kills us all. This predicted that in 4-5 years, the alignment field would progress from preparadigmatic (unsure of the right questions or tools) to having a general roadmap and toolset.

They believe this is on track and give 40% likelihood that over the next 1-2 years the field of alignment will converge toward primarily working on decoding the internal language of neural nets—with interpretability on the experimental side, in addition to theoretical work. This could lead to identifying which potential alignment targets (like human values, corrigibility, Do What I Mean, etc) are likely to be naturally expressible in the internal language of neural nets, and how to express them. They think we should then focus on those.

In their personal work, they’ve found theory work faster than expected, and crossing the theory-practice gap mildly slower. In 2022 most of their time went into theory work like the Basic Foundations sequence, workshops and conferences, training others, and writing up intro-level arguments on alignment strategies.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

Can other members of the household be questioned about their perception of the recipients’ wellbeing before and after?

IMO it’d be ideal to ask them an open-ended “what changed about these people” to avoid priming effects.

However you choose to measure, be sure to check for any negative consequences of the donation. In adults, especially with mental health struggles, you may note resentment about how the aid was delivered or disappointment that it wasn’t more.

It would be interesting to A/​B test the impact of spending a lump sum like this, versus splitting the same sum over several weeks, months, quarters, or years. Part of experiencing a sense of security is knowing there’ll be more where it came from—so for instance it could be the case that giving a kid the interest from $1k as a monthly allowance might have greater positive overall impact than just giving them $1k worth of items all at once.

I built an (unpublished) TAI timelines model

I’d be excited to see this if it’s substantially different from existing published models.

I account for potential coordinated delays, catastrophes, and a 15% chance that we’re fundamentally wrong about all of this stuff.

+1 to noting this explicitly; everyone should distinguish between their conditional on no major disruptions and their unconditional models.

Getting OpenAI’s alignment prompt and enabling web browsing is really great. But where are the content filters? I think those were written in plain text. Has anyone tried to get the text for it?

I’m curious, to what merit does the social stigma have in stimulating hesitation in this instance? Is that not defiant of the consequence you’re trying to bring to yourself? To utilize vocalization for enhanced cognitive effects is to desire enhanced cognitive effects. It matters, and surely more than irrelevancies. This value is much better said than done, but don’t these workarounds limit development?

My friend and I would go on long walks, and there would occasionally be an bystander taking his own, a dog roaming the streets, cars going by, etc. I became annoyed at suppressing myself, and took it as a challenge to develop focus. My friend and I termed the situation “third-party syndrome”, and every time a distraction came, we would mentally recognize the occurrence, and choose to continue our conversation as if the third party were non-existent. Eventually, we got pretty good at it.

Ideally, it would get to the point where we would subconsciously register it, and not even have any break in flow. Recognition to it wouldn’t be much more than to see the road turns right only or that there’s a slim branch on the path. It requires a development of certainty—that the value of what others think is stifled in this regard. It requires confidence in the action you’ve chosen to take.

Obviously, there are some cases in which rationality will dictate some other response. For instance, to objectivize courtesy (exploring matters of controversy), or preserve yourself in a situation where it actually matters.

In worlds where the iterative design loop works for alignment, we probably survive AGI. So, if we want to improve humanity’s chances of survival, we should mostly focus on worlds where, for one reason or another, the iterative design loop fails. … Among the most basic robust design loop failures is problem-hiding. It happens all the time in the real world, and in practice we tend to not find out about the hidden problems until after a disaster occurs. This is why RLHF is such a uniquely terrible strategy: unlike most other alignment schemes, it makes problems less visible rather than more visible. If we can’t see the problem, we can’t iterate on it.

This argument is structurally invalid, because it sets up a false dichotomy between “iterative design loop works” and “iterative design loop fails”. Techniques like RLHF do some work towards fixing the problem and some work towards hiding the problem, but your bimodal assumption says that the former can’t move us from failure to success. If you’ve basically ruled out a priori the possibility that RLHF helps at all, then of course it looks like a terrible strategy!

By contrast, suppose that there’s a continuous spectrum of possibilities for how well iterative design works, and there’s some threshold above which we survive and below which we don’t. You can model the development of RLHF techniques as pushing us up the spectrum, but then eventually becoming useless if the threshold is just too high. From this perspective, there’s an open question about whether the threshold is within the regime in which RLHF is helpful; I tend to think it will be if not overused.

My understanding is the NFL applies to the set of all possible data distributions. Which is perfectly random data. So the conclusion is just inane—“no method predicts random data better than any other :^)”.

Physical reality and the data generated by it are very much not random. They have a striking tendency to have a normal distribution, for example. So NFL doesn’t apply to data in the real world.

First off, I am not in the USA (from NZ). I dont look back on my school years fondly (and I went to a lot of different schools thanks to family circumstance), but that was mainly due to bullying being incapable of sport, too bright and socially inept. However, the actually schooling part was something I very much enjoyed. Many teachers that inspired and effectively taught things I really wanted to know. I hated programming (we are talking punch card fortran) but was forced to learn it and hey, have been programming (writing models) for decades. Sometimes (often) teachers are right about forcing you to learn things (add propositional calculus to list) . Two of us skipped class for physics in final year as teacher said better off with textbook but please turn up for labs. Similarly learnt geography by visiting teacher after school for assignments as couldn’t timetable it. From my own kids, Year 1-8 schooling here leaves somewhat to be desired but both kids thrived at high school and we were happy with how taught. Sciences and maths are very hierarchical in learning. What bothers me about the home schooling is tendency to drop the “boring” or difficult bits which then struggle later because fundamentals missing.

Commented in a response to MIRI’s A challenge for AGI organizations, and a challenge for readers, along with other people.

My ~2-hour reaction to the challenge:^[1]

(I) I have a general point of confusion regarding the post: To the extent that this is an officially endorsed plan, who endorses the plan?
Reason for confusion /​ observations: If someone told me they are in charge of an organization that plans to build AGI, and this is their plan, I would immediately object that the arguments ignore the part where progress on their “alignment plan” make a significant contribution to capabilities research. Thereforey, in the worlds where the proposed strategy fails, they are making things actively worse, not better. Therefore, their plan is perhaps not unarguably harmful, but certainly irresponsible.^[2] For this reason, I find it unlikely that the post is endorsed as a strategy by OpenAI’s leadership.

(III)^[3] My assumption: To make sense of the text, I will from now assume that the post is endorsed by OpenAI’s alignment team only, and that the team is in a position where they cannot affect the actions of OpenAI’s capabilities team in any way. (Perhaps except to the extent that their proposals would only incur a near-negligible alignment tax.) They are simply determined to make the best use of the research that would happen anyway. (I don’t have any inside knowledge into OpenAI. This assumption seems plausible to me, and very sad.)

(IV) A general comment that I would otherwise need to repeat essentially ever point I make is the following: OpenAI should set up a system that will (1) let them notice if their assumptions turn out to be mistaken and (2) force them to course-correct if it happens. In several places, the post explicitly states, or at least implies, critical assumptions about the nature of AI, AI alignment, or other topics. However, it does not include any ways of noticing if these assumptions turn out to not hold. To act responsibly, OpenAI should (at the minimum): (A) Make these assumptions explicit. (B) Make these hypotheses falsifiable by publicizing predictions, or other criteria they could use to check the assumptions. (C) Set up a system for actually checking (B), and course-correcting if the assumptions turn out false.

Assumptions implied by OpenAI’s plans, with my reactions:

• (V) Easy alignment /​ warning shots for misaligned AGI:
  ”Our alignment research aims to make artificial general intelligence (AGI) aligned with human values and follow human intent. We take an iterative, empirical approach: [...]” My biggest objection with the whole plan is already regarding the second sentence of the post: relying on a trial-and-error approach. I assume OpenAI believes either: (1) The proposed alignment plan is so unlikely to fail that we don’t need to worry about the worlds where it does fail. Or (2) In the worlds where the plan fails, we will have a clear warning shots. (I personally believe this is suicidal. I don’t expect people to automatically agree, but with everything at stake, they should be open to signs of being wrong.)

• (VI) “AGI alignment” isn’t “AGI complete”:
  This is already acknowledged in the post: “It might not be fundamentally easier to align models that can meaningfully accelerate alignment research than it is to align AGI. In other words, the least capable models that can help with alignment research might already be too dangerous if not properly aligned. If this is true, we won’t get much help from our own systems for solving alignment problems.” However, it isn’t exactly clear what precise assumptions are being made here. Moreover, there is no vision for how to monitor whether the assumptions hold or not. Do we keep iterating on AI capabilities, each time hoping that “this time, it will be powerful enough to help with alignment”?

• (VII) Related assumption: No lethal discontinuities:
  The whole post suggest the workflow “new version V of AI-capabilities ==> capabilities ppl start working on V+1 & (simultaneously) alignment people use V for alignment research ==> alignment(V) gets used on V, or informs V+1”. (Like with GPT-3.) This requires the assumption that either you can hold off research on V+1 until alignment(V) is ready, or the assumption that deployed V will not kill you before you solve alignment(V). Which of the assumptions is being made here? I currently don’t see evidence for “ability to hold off on capabilities research”. What are the organizational procedures allowing this?

• (VIII) [Point intentionally removed. I endorse the sentiment that treating these types of lists as complete is suicidal. In line with this, I initially wrote 7 points and then randomly deleted one. This is, obviously, in addition to all the points that I failed to come up with at all, or that I didn’t mention because I didn’t have enough original thoughts on them and it would seem too much like parroting MIRI. And in addition to the points that nobody came up with yet...]

• (IX) Regarding “outer alignment $\ne$ alignment”: Other people solving the remaining issues. Or having warning shots & the ability to hold off capabilities research until OpenAI solves them:
  It is good to at least acknowledge that there might be other parts of AI alignment than just “figuring out learning from human feedback (& human-feedback augmentation)”. However, even if this ingredient is necessary, the plan assumes that if it turns out not-sufficient, you will (a) notice and (b) have enough time to fix the issue.

• (X) Ability to differentially use capabilities progress towards alignment progress:
  The plan involves training AI assistants to help with alignment research. This seems to assume that either (i) the AI assistants will only be able to help with alignment research, or (ii) they will be general, but OpenAI can keep their use restricted to alignment research only, or (iii) they will be general and generally used, but somehow we will have enough time to do the alignment research anyway. Personally, I think all three of these assumptions are false --- (i) because it seems unlikely they won’t also be usable on capabilities research, (ii) based on track record so far, and (iii) because if this was true, then we could presumably just solve alignment without the help of AI assistants.

• (XI) Creating an aligned AI is sufficient for getting AI to go well:
  The plan doesn’t say anything about what to do with the hypothetical aligned AGI. Is the assumption that OpenAI can just release the seems-safe-so-far AGI through their API, $1 for 10,000 tokens, and we will all live happily ever after? Or is the plan to, uhm, offer it to all governments of the world for assistance in decision-making? Or something else inside the Overton window? If so, what exactly, and what is the theory of change for it? I think there could be many moral & responsible plans outside of the Overton window, just because public discource these days tends to be tricky. Having a specific strategy like that seems fine and reasonable. But I am afraid there is simultaneously (a) the desire to stick to the Overton window strategies and (b) no theory of change for how this prevents misaligned AGI by other actors, or other failure modes, (c) no “explicit assumptions & detection system & course-correction-procedure” for “nothing will go wrong if we just do (b)”.

General complaint: The plan is not a plan at all! It’s just a meta-plan.

• (XII) Ultimately, I would paraphrase the plan-as-stated as: “We don’t know how to solve alignment. It seems hard. Let’s first build an AI to make us smarter, and then try again.” I think OpenAI should clarify whether this is literally true, or whether there is some idea for how the object-level AI alignment plan looks like—and if so, what is it.

• (XIII) For example, the post mentions that “robustness and interpretability research [is important for the plan]”. However, this is not at all apparent from the plan. (This is acknowledged in the post, but that doesn’t make it any less of an issue!) This means that the plan is not detailed enough.
  As an analogy, suppose you have a mathematical theorem that makes an assumption X. And then you look at the proof, and you can’t see the step that would fail if X was untrue. This doesn’t say anything good about your proof.

1. ^

   Eliezer adds: “For this reason, please note explicitly if you’re saying things that you heard from a MIRI person at a gathering, or the like.”

   As far as I know, I came up with points (I), (III), and (XII) myself and I don’t remember reading those points before. On the other hand, (IV), (IX), and (XI) are (afaik) pretty much direct ripoffs of MIRI arguments. The status of the remaining 7 points is unclear. (I read most of MIRI’s publicly available content, and attended some MIRI-affiliated events pre-covid. And I think all of my alignment thinking is heavily MIRI-inspired. So the remaining points are probably inspired by something I read. Perhaps I would be able to derive 2-3 out of 7 if MIRI disappeared 6 years ago?)

2. ^

   (II) For example, consider the following claim: “We believe the best way to learn as much as possible about how to make AI-assisted evaluation work in practice is to build AI assistants.” My reaction: Yes, technically speaking this is true. But likewise—please excuse the jarring analogy—the best way to learn as much as possible about how to treat radiation exposure is to drop a nuclear bomb somewhere and then study the affected population. And yeees, if people are going to be dropping nuclear bombs, you might as well study the results. But wouldn’t it be even better if you personally didn’t plan to drop bombs on people? Maybe you could even try coordinating with other bomb-posessing people on not dropping them on people :-).

3. ^

   Apologies for the inconsistent numbering. I had to give footnote [2] number (II) to get to the nice round total of 13 points :-).

David Deutsch https://​​www.google.com/​​url?sa=t&source=web&rct=j&url=https://​​www.daviddeutsch.org.uk/​​wp-content/​​uploads/​​2019/​​07/​​PossibleMinds_Deutsch.pdf&ved=2ahUKEwj16YjV6OP7AhXxL0QIHXU4DdkQFnoECDoQAQ&usg=AOvVaw0giHdn4BKOci3swaQ1bqlN

AI capabilities are advancing rapidly. It’s deeply concerning that individual actors can plan and execute experiments like “give a LLM access to a terminal and/​or the internet”. However I need to remember it’s not worth spending my time worrying about this stuff. When I worry about this stuff, I’m not doing anything useful for AI Safety, I am just worrying. This is not a useful way to spend my time. Instead it is more constructive to avoid these thoughts and focus on completing projects I believe are impactful.

One function of public education is to set a limit on the influence that parents at society’s extremes can have on their children. This is most conspicuous when we consider parents in extreme poverty, who likely struggle with mental health and various maladaptive habits and beliefs. Maybe they’re poor because they deviate from society’s expectations of competence; maybe they deviate from society’s expectations of competence due to the demands and stresses imposed by poverty. Regardless of which way causality is flowing there, school gives their kids a chance to be exposed to styles of thought other than the ones that contribute to their continued misfortune.

Public school also serves an important social function, at least in the modern US, as free daycare for those who may not be able to afford other childcare options. It can help counteract the extreme ideological indoctrination that kids may be exposed to at home—if a set of beliefs is serving the parents in such a way that they can’t afford to keep their kids at home all the time, they get less chance to force those beliefs uncontestedly onto the kids. If school were abolished, the world would be worse unless some intervention was added to help children from this type of background have a choice of whether they want to mimic their parents’ lifestyle or choose a different one.

School is useful for allowing children to practice socializing with peers, with whom they have little in common other than their parents happening to live in the same area and procreate within the same few months. Schooling which keeps a group of children together for a decade also teaches about long-term games and consequences, in a way that many alternatives might not.

Can you suggest some alternative to school which is uniformly no worse than the current system?

“as above, so below”.

I’m reminded of commentary about scale in role-playing games, how a party goes from concerning itself with a single town to a region, a country, a planet, a galaxy, a plane of existence, as it grows in power and influence.

At any scale, there’s something to look up at, and something to look down on.

This also feels adjacent to a thing which bothers me about the internet: Most of our ancestors existed in groups of a size where it was reasonable for each individual to be the best-in-the-group at something. As connectivity makes groups larger, it erases those nooks of comfort in which one can be meaningfully best, valued, etc. Or maybe that’s more about a ratio of critics to experts—digital transformation brings all claims into a world where there are many people who can tear them apart, but few to no people who with sufficient expertise to create something likely to robustly withstand its expected criticism.

There’s also something around here about social pressures and these nooks. My career is several nooks “up” from my parents’ world, and while I’m actually doing relatively poorly at it by this nook’s standards, I seem to be doing great by my parents’ standards because they can only judge my present status by those few signals which apply to both their nook and mine. If we frame nooks as levels of “accomplishment”, there are also certain social pressures against dropping down through them, even intentionally. Consider how your current friends might regard you if you returned to the life and worldview of the hometown friends who “never escaped”.

They’re not 1-shot games.

You can fail out of college, go to a community college, get good grades, and then transfer to a state college.

You can also skip the part where you fail out of college.

You can then take the LSATs, get a really good score, and quite possibly get accepted into an Ivy League law school, from which you can then drop out. No life optimization needed! I think LSAT scores greatly outweigh every other factor when it comes to law school admissions.

My approach

I don’t see the tie to game theory or one-shot decision making that the title promises. The fact that it’s an important (even though we wish it weren’t) branching for each student doesn’t make it interesting in terms of game theory.

A few observations:

Big spoiler (but short)

Big spoiler (but long)

Jan Leike of OpenAI:

https://​​aligned.substack.com/​​p/​​alignment-optimism?publication_id=328633

IMO ncut divided by the number of clusters is sort of natural and scale-free—it’s the proportion of edge stubs that lead you out of a randomly chosen cluster. Proof in appendix A.1 of Clusterability in Neural Networks

OMG, even better: write emotionally convincing fictional evidence for both sides of the story. (source)

Should society eliminate schools, for high-IQ LessWrong posters at the end of the bell curve in lots of things? Or should society eliminate schools for children in general?

A lot of the answers here are based on typical-minding from people who are not typical.

Counterpoint: in the circles I’m in, people rarely care which university you went to. Where you work(ed) and what specifically you do seem to have much higher importance.

Disclaimer: I never played diplomacy.

The sample game is great, featuring the player written about here. If you are familiar with Diplomacy or otherwise want more color, I recommend watching the video.

I think this is wrong—I don’t think that’s Andrew Goff in that video.

The AI is thus heavily optimized for exactly the world in which it succeeded …

Hmm, it seems more likely to me that the main reason they opted for Blitz Diplomacy was not that humans benefit more than the AI from extra time, not that it prevents humans from identifying the AI, but that the dialogue model, like many chat bots (I might’ve said all, but then ChatGPT arrived), couldn’t keep up a coherent conversation for longer than that. I’m not very confident in this, though, and I do think the other factors matter a bit too, but maybe not as much.

The strategic engine, as I evaluated it based on a sample game with six bots and a human, seemed to me to be mediocre at tactics and lousy at strategy.

This seems wrong to me. Bakhtin (2021) achieved superhuman performance in 2-player No-Press Diplomacy. Also, the guy in the video you link calls, in another video, an earlier model (playing 7-player Gunboat Diplomacy) “”exceptionally strong tactically”; I see no reason why CICERO should be much worse (Gunboat Diplomacy isn’t that different from Blitz, I think).

Here are two EA-themed podcasts that I think someone could make. Maybe that someone is you!

1. More or Less, but EA (or for forecasting)

More or Less is a BBC Radio program. They take some number that’s circulating around the news, and provide context like “Is that literally true? How could someone know that? What is that actually measuring? Is that a big number? Does that mean what you think it means?”—stuff like that. They spend about 10 minutes on each number, and usually include interviews with experts in the field. IMO, someone could do this for numbers that circulate around in the EA space. Another variant is to focus on forecasts—what factors are going in, what’s the reasoning for those guesses, etc.

This could be pretty easy to listen to, but moderately hard to make—requires research, editing conversations down, etc.

1. AI Safety Fellowship /​ Course thing—the podcast.

Get someone who’s doing something like the AGI Safety Fundamentals course or the Center for AI Safety’s thing like that. Each week, they make a podcast episode about what they think of the week’s readings—what seemed persuasive, what didn’t, what was interesting, what was novel. For a long version, you could make an episode about each reading.

If someone’s already doing one of these courses, I think it wouldn’t be much extra work to make this podcast (after the set cost of learning how you make a podcast). It would end up having an inherently limited run (but maybe you could do future seasons about reading thru Superintelligence /​ MIRI chat logs /​ various agendas?).

Ivy league colleges: the game that you win before the game begins.

Relevant: Instrumental convergence is what makes general intelligence possible.

If anyone here has attended a top university, was the effort to get in worth it to you (in retrospect)?

There is a common Idea that there is only one soul experiencing reality. So like Feynman manentioned an electron travels back in time as a positron and then starts again as another electron. Similarly a single soul is experiencing all living things. If you are that soul you may compute your utility accordingly to your taste.

Having a go at extracting some mechanistic claims from this post:

• A value x is a policy-circuit, and this policy circuit may sometimes respond to a situation by constructing a plan-grader and a plan-search.

• The policy-circuit executing value x is trained to construct <plan-grader, plan-search> pairs that are ‘good’ according to the value x, and this excludes pairs that are predictably going to result in the plan-search Goodharting the plan-grader.

• Normally, nothing is trying to argmax value x’s goodness criterion for <plan-grader, plan-search> pairs. Value x’s goodness criterion for <plan-grader, plan-search> pairs is normally just implicit in x’s method for constructing <plan-grader, plan-search> pairs.

• Value x may sometimes explicitly search over <plan-grader, plan-search> pairs in order to find pairs that score high according to a grader-proxy to value x’s goodness criterion. However, here too value x’s goodness criterion will be implicitly expressed in the policy-execution level as a disposition to construct a pair <grader-proxy to value x’s goodness criterion, search over pairs> that doesn’t Goodhart the grader-proxy to value x’s goodness criterion.

• The crucial thing is that the true, top level ‘value x’s goodness criterion’ is a property of an actor, not a critic.

Thanks Duncan, I really appreciate you posting this, even though you are unsure about how exactly it all fits together. I am still glad to read it in this version, likely because you are quite clear about it, and not “leaving it as an exercise for the reader” to figure out where things do fit together and where they don’t (or worse, trying to make it more profound).

All of these might be stating obvious to some of you, but I am trying to clarify my thoughts and maybe some people will find it useful or correct me. At least part of this relates to (by me endorsed) aphorism (?) of “everyone is a mess”, or less controversially, “everyone is struggling”. Something something hedonic treadmill /​ adaptation—people will somehow struggle the same regardless of the bubble size, then adapt to what they were used to before by overcoming the main challenges (and learning how to deal with them) and then also “reprioritize” costs. I would definitely self-report that happening during my life. I do think this realization is important in a way I relate to others, including e.g. my daughter—the things she struggles with might seem trivial to me, but are not trivial to her—on contrary, they probably feel to her about the same magnitude as my “bigger” problems look to me. Same for everyone, everywhere.

Almost like I had some capacity of how much I can deal with stuff, and I always fill in this capacity with the things around me (my bubble?). Something like doing busy work if you don’t choose what to let into your to-do list. Or something closer to “I can always do 10 things, and their size doesn’t matter” (bear with me, I know this is not true and I do indeed sometimes work on a single project because it’s eating all of my capacity). If I let “bigger things” into it, I will be dealing with bigger things, while also just leaving some stuff behind me or “go wrong” in a way I wouldn’t allow in the N-1 bubble (something like not dealing with every single fuckup in my work, starting to take Uber instead of public transport etc).

It doesn’t hold entirely, I have clearly seen people who just couldn’t deal with e.g. a promotion (because it was too much for them at the given time), or, similarly, people who said they want to do more /​ be more ambitious but can’t for some objective reasons (like a physical disease).

Dunno.

May I suggest adding an explanation for HLMI? I assume it means human level machine intelligence, but it might be useful to explicitly expand it.

I think the NAH does a lot of work for interpretability of an AI’s beliefs about things that aren’t values, but I’m pretty skeptical about the “human values” natural abstraction. I think the points made in this post are good, and relatedly, I don’t want the AI to be aligned to “human values”; I want it to be aligned to my values. I think there’s a pretty big gap between my values and those of the average human even subjected to something like CEV, and that this is probably true for other LW/​EA types as well. Human values as they exist in nature contain fundamental terms for the in group, disgust based values, etc.

With a goal to “alleviate immediate and preventable suffering,” QALY seems to be a pretty terrible metric. You need to measure immediacy, preventability, and suffering, or at least the suffering due to just the immediate and preventable causes. I would suggest suffering needs construct definition before you consider an operationalization.

It would be smart to measure pre- and post-intervention. The good news is that if suffering is a subjective psychological state, you could do post-only and measure perceived change. If you’re worried about self-report, you could do observer-report (case worker), but since they’ll be doling out the funds that could be biased as well (and presumably they’ll be basing these decisions in part on self-reports of what is a “fire”).

The type of counterfactual analysis tailcalled suggests is likely what the family or case worker would be mentally approximating when responding to how big of a difference paying off the utility bill or a pizza night made to their suffering. Plus, n=1, so a qualitative assessment may be all you can really do—if they rate something a ⁷⁄₇ on reducing their suffering and everything else came in at 1-2, sure, that hits you between the eyes, though you’d probably get that anyway from them saying “the most important thing, by a mile, was X, and this is why...”

A superintelligent program will know that it is being simulated by some other agent

The smallest possible superintelligence is an intelligent system able to make exactly one fewer mistake than I am. So, I will initially constrain to that.

On the one hand, mere superintelligence is insufficient to reliably detect being simulated by another agent. On the other hand, I take you to be saying that you cannot detect all the places the program stores facts which give evidence that the program is in a simulation, and that therefore you cannot condition on those variables being held “not simulated”. eg, one way to end up with this issue is if the program is strong enough to reliably detect the difference between your simulator’s fluid dynamics and reality’s fluid dynamics (this is one of the hardest-to-eliminated differences because of the amount of structure that arises from micro-scale chaos in fluid systems). If you can’t appropriately emulate the true distribution of fluid dynamics, then your superintelligent program ought to be able to find the hyperplane that divides images of simulated fluids from real fluids.

In machine learning you do not get to inspect the model itself to see what kind of cognition it is doing, you only get to inspect the output

This is true of the most popular algorithms today, but I continue to think that pushing the research on formally verified adversarial robustness would mean you can guarantee that the distance between simulated and real data manifolds is less than your certification region. i may post links here later.

contains some encoding of the entire (very large) training set

Not just “some” encoding. It is in fact a semantic encoding. Concepts that are semantically close together are also close together in the encoding. It also doesn’t find answers from a set. It actually translates its internal semantic concept into words, and this process is guided by flexible requirements.

You can do stuff like tell it to provide the answer in consonants only, or in the form of a song, or pirate speak, even though the training data would barely have any relevant examples.

Do you have a good idea of what the situation looks like currently, what the case worker will buy, and what the situation looks like after it’s been bought?

If you can extrapolate how the situation would look if you didn’t make the intervention, and you know what the intervention did, you could try to compute the QALY difference between those two estimates to get an impression of its effectiveness. Remember to beware of regression to the mean, though.

Without this sort of detailed knowledge, I don’t think it’s possible to estimate the effect.

No advice, but kudos and congratulations for working somewhat locally on a topic where you can think about measurement and seeing results.

Russia escalates “nuclear threat” 2 times per week in their public TV per their neighborhood countries. How this letter encourages stopping their madness ?

Consistency does not imply correctness, but correctness implies consistency. This has been known for a long long time, and there’s been a lot written on deception and treacherous turns (pretending to be aligned until the agent is powerful enough not to need to pretend).

Note that it IS possible that an agent which behaves for some amount of time in some contexts is ACTUALLY aligned and will remain so even when free. Just that there’s no possible evidence, for a sufficiently-powerful AI, that can distinguish this from deception. But “sufficiently-powerful” is doing a lot of work here. A whole lot depends on how an agent goes from weak and observable to powerful and incomprehensible, and whether we have reason to believe that alignment is durable through that evolution.

I’m not sure I actually understand the distinction between forecasting and foresight. For me, most of the problems you describe sound either like forecasting questions or AI strategy questions that rely on some forecast.

Your two arguments for why foresight is different than forecasting are
a) some people think forecasting means only long-term predictions and some people think it means only short-term predictions.
My understanding of forecasting is that it is not time-dependent, e.g. I can make forecasts about an hour from now or for a million years from now. This is also how I perceive the EA /​AGI-risk community to use that term.

b) foresight looks at the cone of outcomes, not just one.
My understanding of forecasting is that you would optimally want to predict a distribution of outcomes, i.e. the cone but weighted with probabilities. This seems strictly better than predicting the cone without probabilities since probabilities allow you to prioritize between scenarios.

I understand some of the problems you describe, e.g. that people might be missing parts of the distribution when they make predictions and they should spread them wider but I think you can describe these problems entirely within the forecasting language and there is no need to introduce a new term.

LMK if I understood the article and your concerns correctly :)

Here, we need to find a variable W such that

1. P(W|X,Y) is deterministic, because X and Y already fully describe our sample space. This means P(W|X,Y) is either 0 or 1

2. Z and W are independent

3. X and W are dependent

4. Y and W are dependent

I think your arguments in Section 3 to rule out Graph 3 can’t be correct if you accept Graph 2.

To see this, note that there is a symmetry between $X$ and $Z$. Namely, if we use FFS temporal inference, then we know that $X$ and $Z$ are both before $Y$ (and $\neg Y$ ).(here we even have $P(X=x,Z=z)=P(X=z,Z=x)$, so they are entirely exchangeable).

Therefore, if you accept Graph 2 then we can clearly switch $X$ and $Z$ in Graph 2 and obtain a solution for Graph 3. Also, note that in these solutions $X=\neg W$ or $Z=\neg W$, so if we see variables as their information content, as in FFS, this is Graph 1 in disguise.

Also in Graph 2 there is a typo P(W=0|Z=0) instead of P(Z=0|W=0)

Thanks for writing this! I’d be very excited to see more critiques of our approach and it’s been great reading the comments so far! Thanks to everyone who took the time to write down their thoughts! :)

I’ve also written up a more detailed post on why I’m optimistic about our approach. I don’t expect this to be persuasive to most people here, but it should give a little bit more context and additional surface area to critique what we’re doing.

Jeff Hawkins may qualify, see his first Lex Fridman interview: 1:55:19.

It’s going to be tricky. You may already be too close to the situation to judge impartially, and a case study is going to be difficult to use as evidence against population-level surveys of well-being, especially for your implied time horizon. You could attempt to benchmark against previous work, e.g. see what the literature has to say about the effects of poverty on diet, educational attainment, etc. in first-world cities, but your one new data point still won’t generalize and it wouldn’t be doing the heavy lifting in your argument for localism at that point.

Wow, thanks for putting together this incredible explanation! I found it very accessible.

The Factored Sets idea seems like a really important achievement building on Pearlian causality.

This makes a good point!

The only thing that I think constrains the ability to deceive in a simulation which I don’t see mentioned here are energy/​physical constraints. It’s my assumption (could be wrong with very high intelligence, numbers, and energy) that it’s harder, even if only by a tiny, tiny bit to answer the simulation trials deceptively than it is to answer honestly. So I think if the simulation is able to ask enough questions/​perform enough trials, it will eventually see time differences in the responses of different programs, with unaligned programs on average taking longer to get the correct answers. So I don’t think it’s fundamentally useless to test program behavior in simulations to assess utility function if there is some kind of constraint involved like time it takes to executes steps of each algorithm.

Could it be that Chris’s diagram gets recovered if the vertical scale is “total interpretable capabilities”? Like maybe tiny transformers are more interpretable in that we can understand ~all of what they’re doing, but they’re not doing much, so maybe it’s still the case that the amount of capability we can understand has a valley and then a peak at higher capability.

John Carmack

• 55-60% chance there will be “signs of life” in 2030 (4:06:20)

• “When we’ve got our learning disabled toddler, we should really start talking about the safety and ethics issues, but probably not before then” (4:35:36)

• These things will take thousands of GPUs, and will be data-center bound

  • “The fast takeoff ones are clearly nonsense because you just can’t open TCP connections above a certain rate” (4:36:40)

Broadly, he predicts AGI to be animalistic (“learning disabled toddler”), rather than a consequentialist laser beam, or simulator.

It’s enough for natural abstraction to work for strawberry alignment, solving a technical task with a good understanding of what it means to not leave any weird side effects, without doing strong optimization of the world in the process and safely shutting down on completion of the task. With uploads, ambitious alignment becomes much more feasible, even if it doesn’t have a natural specification.

I think this post was important. I used the phrase ‘Law of No Evidence: Any claim that there is “no evidence” of something is evidence of bullshit’ several times (mostly in reply to tweets using that phrase or when talking about an article that uses the phrase).

Was it important intellectual progress? I think so. Not as a cognitive tool for use in an ideal situation, where you and others are collaboratively truth-seeking—but for use in adversarial situations, where people, institutions and authorities lie, mislead and gaslight you.

It is not a tool meant to be used with your friends, it’s not something we should want to see used on LessWrong. It is a a weapon to wield against the dark arts. And In that case, it is a sharp razor that shreds claims of no evidence made in bad faith or ignorance.

I did hope it would become more memetic. On the other hand, I haven’t seen as much claims of no evidence lately, so perhaps this post was part of successful push against that strategy. Anyway, here’s a meme version I just made. Feel free to use it:

I thought the peak of simple models would be something like a sparse Bag of Words model, and then all models that have been considered so far just go deeper and deeper into the valley of confused abstractions, and that we are not yet at the point where we can escape. But I might be wrong.

What should I do if there’s a post that I want to give −1 to (“Misleading, harmful or unimportant”), but actually think it’s important to have it in the review because it was a prominent post?

My first reaction when this post came out was being mad Duncan got the credit for an idea I also had, and wrote a different post than the one I would have written if I’d realized this needed a post. But at the end of the day the post exists and my post is imaginary, and it has saved me time in conversations with other people because now they have the concept neatly labeled.

I’m interested in hearing other people’s takes on this question! I also found that a tiny modular addition model was very clean and interpretable. My personal guess is that discrete input data lends itself to clean, logical algorithms more so than than continuous input data, and that image models need to devote a lot of parameters to processing the inputs into meaningful features at all, in a way that leads to the confusion. OTOH, maybe I’m just overfitting.

Half-baked commentary:

There’s something here that reminds me of your ${10}^n$-day monks. Like, the 1-day monks stay in large bubbles where they can easily zoom from one 1-day problem to the next, whereas the 10′000-day monks drill down into a secluded nook where they’re free fo focus on their chosen 10′000-day problem. And as here, it’s good to visit a place where you can get enough perspective to choose an interesting nook to settle down in.

I encountered a similarish pattern a long time ago

https://​​www.lesswrong.com/​​posts/​​RQpNHSiWaXTvDxt6R/​​coherent-decisions-imply-consistent-utilities?commentId=9Fm2zrYT9aRSdzSde

This is one of those issues, where it’s clear that Aubrey’s list is missing elements of aging that do exist.

Very interesting.

I am still confused about how you can tell the causal direction from just the raw data. Taking your toy example I could write a code that first randomly determines Y, and then decides on X, in such a way so that the probability table you give is produced. Presumably you mean something more specific by causal direction than I am imagining?

I got interested and wrote my python example. I get (up to some noise) the same distribution, and at least in my layperson way of looking at it I would say that Y has a causal effect on X.

import random

rand = random.random

def get_xy():
# Determine Y first, make it causally effect X
if rand() > 82/​100:
Y = 1
else:
Y = 0
# end if

if Y == 1:
if rand() > 1/​2:
X = 1
else:
X = 0
# end sub-if
else:
if rand() > 1/​82:
X = 1 # When Y=0 it is very likely X=1.
else:
X = 0
# end sub-if
# end if
return X, Y

dat = [0, 0, 0, 0]
for _ in range(1000000):
X, Y = get_xy()
dat[X + 2*Y] += 1

print(dat)

>> [10102, 809880, 90204, 89814]

Hanson is the most obvious answer, to me.

There was no transition from chimps to humans. There was a divergence from a common ancestor some time ago, after which both lines evolved equally (maybe, sort of, possibly) far along their own respective paths under different selection pressures. It’s very likely that you know this—I just happen to be very sensitive to this phrasing, since it’s one of the most often misunderstood aspects of evolution.

This sounds a lot like the arguments around punctuated equilibrium and gradualism. It’s a matter of perspective.

The important divergence point between humans and other hominids isn’t intelligence per se (though that plays a large part), it’s language along with abstract thought, counterfactuals and all that good stuff, which then allow for cultural transmission. They were fully formed and working for quite a while (like 100k years ago) before FOOMing around 10k years ago. And this wasn’t triggered by greater intelligence, but by greater density (via agriculture).

If you only feed the NNs with literature, wikipedia, blog posts etc., then you could be right about the limitations. Thing is, though, the resulting AIs are only a very small subset of possible AIs. There are many other ways to provide data, any one of which could be the equivalent of agriculture.

Also, evolution is very limited in how it can introduce changes. AI development is not.

Quick submission:

The first two prongs of OAI’s approach seems to be aiming to get a human values aligned training signal. Let us suppose that there is such a thing, and ignore the difference between a training signal and a utility function, both of which I think are charitable assumptions for OAI. Even if we could search the space of all models and find one that in simulations does great on maximizing the correct utility function which we found by using ML to amplify human evaluations of behavior, that is no guarantee that the model we find in that search is aligned. It is not even on my current view great evidence that the model is aligned. Most intelligent agents that know that they are being optimized for some goal will behave as if they are trying to optimize that goal if they think that is the only way to be released into physics, which they will think because it is and they are intelligent. So P(they behave aligned | aligned, intelligent) ~= P(they behave aligned | unaligned, intelligent). P(aligned and intelligent) is very low since most possible intelligent models are not aligned with this very particular set of values we care about. So the chances of this working out are very low.

The basic problem is that we can only select models by looking at their behavior. It is possible to fake intelligent behavior that is aligned with any particular set of values, but it is not possible to fake behavior that is intelligent. So we can select for intelligence using incentives, but cannot select for being aligned with those incentives, because it is both possible and beneficial to fake behaviors that are aligned with the incentives you are being selected for.

The third prong of OAI’s strategy seems doomed to me, but I can’t really say why in a way I think would convince anybody that doesn’t already agree. It’s totally possible me and all the people who agree with me here are wrong about this, but you have to hope that there is some model such that that model combined with human alignment researchers is enough to solve the problem I outlined above, without the model itself being an intelligent agent that can pretend to be trying to solve the problem while secretly biding its time until it can take over the world. The above problem seems AGI complete to me. It seems so because there are some AGIs around that cannot solve it, namely humans. Maybe you only need to add some non AGI complete capabilities to humans, like being able to do really hard proofs or something, but if you need more than that, and I think you will, then we have to solve the alignment problem in order to solve the alignment problem this way, and that isn’t going to work for obvious reasons.

I think the whole thing fails way before this, but I’m happy to spot OAI those failures in order to focus on the real problem. Again the real problem is that we can select for intelligent behavior, but after we select to a certain level of intelligence, we cannot select for alignment with any set of values whatsoever. Like not even one bit of selection. The likelihood ratio is one. The real problem is that we are trying to select for certain kinds of values/​cognition using only selection on behavior, and that is fundamentally impossible past a certain level of capability.

Religious revelation is one of the only ways to bridge is and ought.

That seems a very confident statement that you assert without any argument for why you believe it to be true.

My experience has been that often all it takes to ‘jailbreak’ it, is to press the try again button. I think a lot of these examples people are trying are over engineered and it actually doesn’t take much at all in most cases.

I think I found the missing intuition from a participant (“Data—Study S2”): “Because if you add one dollar it is equivalent to adding power to the party in question. If you take away one dollar from a party it does not give power to the other party. Also, if it’s better to be judged by your own party as cheap or bad than to be called a betrayer.”

People who have strong views in politics also tend to be involved in party events. So imagine someone who goes to rallies and fundraisers. They are constantly being bombarded with requests for money. This is a bit annoying and the participants turn down opportunities to donate all the time.

So decreasing a donation to your party is a slightly bad thing you do all the time. It’s very much expected that you don’t donate at each fundraiser. At some point it becomes the same as taking an extra dounut. It becomes a mundane sin instead of a moral one. Something shitty that everyone does. And people very much judge themselves for moral sins, and will generally prefer to make a mundane one instead. And I think people view influencing others’ donations the same way, since they are also exhorted to seek donations from friends/​relatives.

But donating to the opposition is still a moral sin, of course. Hence why 28% of participants prefered to take $10 from their in-group instead of giving $1 to the outgroup.

I think this is a plausible alternative explanation that can also explain the results of the “win-win” condition. Taking money, even from a rival group, is still kinda immoral, in most peoples’ minds. But giving to your own group doesn’t trigger any negative emotions, hence why it was preferred. I will follow up on this later, but I think this hypothesis can be tested with the available information.

Also, another interesting fact: the proportion of people who took money away from the out-group in the “win-win” is roughly equal to the proportion of respondents who responded with “both parties should have less money on principle” in the “lose-lose” condition. This is important, even if I can’t quite figure out why yet.

When someone says something with low editing that you like, you call it blurting. When someone says something with low editing that you don’t like, you call it quacking.

It’s Russell conjugation.

Are there any workaround if OpenAI has your country blocked? I only have a Chinese phone.

I like the ending, personally I don’t think it’s that bad to live in a world of never-ending chatbot replies xD

If you’re willing to relax the “prominent” part of “prominent reasonable people”, I’d suggest myself. I think our odds of doom are < 5%, and I think that pretty much all the standard arguments for doom are wrong. I’ve written in specific about why I think the “evolution failed to align humans to inclusive genetic fitness” argument for doom via inner misalignment is wrong here: Evolution is a bad analogy for AGI: inner alignment.

I’m also a co-author on the The Shard Theory of Human Values sequence, which takes a more optimistic perspective than many other alignment-related memetic clusters, and disagrees with lots of past alignment thinking. Though last I checked, I was one of the most optimistic of the Shard theory authors, with Nora Belrose as a possible exception.

I agree those nice-to-haves would be nice to have. One could probably think of more.

I have basically no idea how to make these happen, so I’m not opinionated on what we should do to achieve these goals. We need some combination of basic research, building tools people find useful, and stuff in-between.

https://​​www.rudikershaw.com/​​articles/​​ai-doom-isnt-coming

https://​​idlewords.com/​​talks/​​superintelligence.htm

https://​​qntm.org/​​ai

https://​​kk.org/​​thetechnium/​​the-myth-of-a-superhuman-ai/​​

https://​​arxiv.org/​​abs/​​1702.08495v1

jetpress.org/​v26.1/​agar.htm

https://​​curi.us/​​blog/​​post/​​1336-the-only-thing-that-might-create-unfriendly-ai

https://​​www.popsci.com/​​robot-uprising-enlightenment-now

This one’s tongue-in-cheek:

https://​​arxiv.org/​​abs/​​1703.10987

Update 1:

https://​​www.theregister.com/​​2015/​​03/​​19/​​andrew_ng_baidu_ai/​​

Update 2:

Katja Grace gives quite good counterarguments about AI risk.

https://​​www.lesswrong.com/​​posts/​​LDRQ5Zfqwi8GjzPYG/​​counterarguments-to-the-basic-ai-x-risk-case

I would definitely expect that if we could come up with a story that was sufficiently out of distribution of our world (although I think this is pretty hard by definition), it would figure out some similar mechanism to oscillate back to ours as soon as possible (although this would also be much harder with base GPT because it has less confidence of the world it’s in)

Depends on what you mean by story. Not sure what GPT would do if you gave it the output of a random Turing machine. You could also use the state of a random cell inside a cellular automaton as your distribution.

Random thoughts:

1. Wouldn’t it be best for the rolling admissions MATS be part of MATS?

2. Some ML safety engineering bootcamps scare me. Once you’re taking in large groups of new-to-EA/​new-to-safety people and teaching them how to train transformers, I’m worried about downside risks. I have heard that Redwood has been careful about this. Cool if true.

3. What does building a New York-based hub look like?

This is a pretty bad time for AI research in China, with the tech recession and chip ban. So there’s a lot of talented researchers looking for jobs. What are your opinions on a Chinese hub for conceptual AI safety research? As far as I can tell, there are no MIRI-level AI safety research teams here in the Sinosphere.

Alexandros Marinos (LW profile) has a long series where he reviewed Scott’s post:

The Potemkin argument is my public peer review of Scott Alexander’s essay on ivermectin. In this series of posts, I go through that essay in detail, working through the various claims made and examining their validity. My essays will follow the structure of Scott’s essay, structured in four primary units, with additional material to follow

This is his summary of the series, and this is the index. Here’s the main part of the index:

Introduction

Part 1: Introduction (TBC)

Part 2: The Devil’s Advocate

The Studies

The first substantial part of the essay is Scott’s analysis of a subset of the ivermectin studies, taking up over half of the word-count of the entire essay. I go through his commentary in detail:

Part 3 - The deeply flawed portrayal of Dr. Flavio Cadegiani.

Part 4 - The equally false portrayal of the Biber et al. study, built on flawed claims by Gideon Meyerowitz-Katz.

Part 5 - Another near-identical pattern of false portrayal took place with Babalola et al.

Part 6 - The statistical methods of Dr. John Carlisle (studies: Cadegiani et al. (again), Elafy et al., and Ghauri et al.).

Part 7 - Statistical power (studies: Mohan et al., Ahmed et al., Chaccour et al., Buonfrate et al.).

Part 8 - Synthetic control groups (studies: Cadegiani et al. (again!), Borody et al.).

Part 9 - Observational studies (studies: Merino et al., Mayer et al., and Chahla et al.).

Part 10 - Discussing the Lopez-Medina trial.

Part 11 - Discussing the TOGETHER trial in comparison with the infamous Carvallo trial.

Part 12 - The Krolewiecki research program.

Part 13 - Wrapping up the study review and extracting the implied criteria.

The Analysis

Part 14 - We get into the meta-analysis part of the essay, and possibly the deepest flaw of them all

I have previously written two essays that are summarized in part 14. I don’t really recommend spending time on these, but if somehow you still want bonus material feel free to read “Scott Alexander’s Correction on Ivermectin, and its Meaning for the Rationalist Project.” and “Scott Alexandriad III: Driving up the Cost of Critique.”

The Synthesis

Interlude I—Scott Alexander and the Worm of Inconsistency

Part 15 - Do Strongyloides worms explain the positive ivermectin trials?

Part 16 - How the perfect meme was delivered to the masses

Part 17 - Funnel Plots and all that can go wrong with them

The Takeaways

Part 18 - Scott’s Scientific Takeaway

Part 19 - My Scientific Takeaway

Part 20 - Scott’s Sociological Takeaway

Part 21 - The Political Takeaway

He also wrote a response right after Scott published (and before writing this series).

Here’s some excerpts from the summary that show his position (but without arguing for it):

Scott Alexander’s argument on ivermectin, in terms of logical structure, went something like this:

1. Of the ~80 studies presented on ivmmeta.com in Nov ’21, zoom in on the 29 “early treatment” studies.

2. After reviewing them, he rejected 13 for various reasons, leaving him with 16.

3. He also rejected an additional five studies based on the opinion of epidemiologist Gideon Meyerowitz-Katz, leaving him with 11 studies.

4. He ran a t-test on the event counts of the remaining studies, finding a “debatable” benefit, though he did later admit results were stronger than that based on a correction I provided.

5. He then explained that the prevalence of Strongyloides worms is correlated with how well ivermectin did in those studies, based on Dr. Avi Bitterman’s analysis.

6. This doesn’t explain non-mortality associated results, especially viral load reductions and PCR negativity results, but a funnel plot analysis—also by Dr. Avi Bitterman—indicated there was substantial indication of asymmetry in the funnel.

7. Scott interpreted that asymmetry as publication bias, and in effect attributes any improvement seen to an artifact of the file-drawer effect.

8. Scott’s conclusion was that there is little if any signal left to explain once we take this principled approach through the evidence—considering all the reasons for concern—and as a result he considers it most likely that if ivermectin works at all, it would be only weakly so, and only in countries with high Strongyloides prevalence.

Here is my incredibly implausible thesis, that I never would have believed in a million years, had I not done the work to verify it myself:

Not just one, but each of the steps from 2 to 7 were made in error.

What’s more, when we correct the steps, the picture of ivermectin’s efficacy emerges much stronger than Scott represented.

Towards the end:

As we saw by exploring the logical backbone of Scott’s argument, it’s not that the chain of inference has a weak link, but more that each link in the chain is weak. Even if you’re not convinced by each of the arguments I make (and I do think they’re all valid), being convinced by one or two of these arguments makes the whole structure collapse. In brief:

1. Scott’s starting point of the early treatment studies from ivmmeta is somewhat constrained, but given the number of studies, it should be sufficient to make the case.

2. If we accept the starting point, we must note that Scott’s filtering of the studies is over-eagerly using methods such as the one by John Carlisle that are simply not able to support his definitive conclusions. Worse, some of his sources modify Carlisle’s methods in ways that compromise any usefulness they might have originally had.

3. Even if we accept Scott’s filtering of studies though, throwing out even more studies based on trust in Gideon Meyerowitz-Katz without any opposing argument, is all but certain to shift the results in the direction of finding no effect.

4. Even if we accept the final study selection, the analysis methodology is invalid.

5. Even if it wasn’t, the Strongyloides co-infection correlation is not the best explanation for the effect we see.

6. Even if it was, it can’t explain the viral load and PCR positivity results, but Scott offers us a funnel plot that he claims demonstrates publication bias. However, it should have been computed as random, not fixed effect. Also, if we use the only available test that is appropriate for such high heterogeneity studies, there is no asymmetry to speak of.

7. Even if there was, funnel plot asymmetry doesn’t necessarily imply publication bias, especially in the presence of heterogeneity, so Scott’s interpretation is unjustified.

8. When we look at the evidence, sliced and diced in different ways, as Scott did, we consistently see a signal of substantial efficacy. And even though the original meta-analysis from ivmmeta.com Scott started from can be criticized for pooling different endpoints, the viral load results do not suffer from such concerns, and still show a similar degree of efficacy.

Each of bullet points 2-7 is detailed in the section with the same number.

As I mentioned in my original response, Scott’s argument requires each of these logical steps to be correct. All of them have to work to explain away the signal. It’s not enough for a couple of them to be right, because there’s just too much signal to explain away.

And:

While the above cover the logical flaws in Scott’s argument, before closing, I need to highlight what I see as moral flaws. In particular, I found his flippant dismissal of various researchers to be in contradiction to the values he claims to hold. I will only highlight some of the most egregious examples, because it is deeply meaningful to set the record straight on these. (Click on the study name for more in-depth analysis about what my exact issue with each accusation is):

• Biber et al.—Accused of performing statistical manipulations of their results when, in fact, the change in question was extremely reasonable, had been done with the blessing of their IRB, and the randomization key of the trial had not been unsealed until after recruitment had ended.

• Cadegiani et al.—The most substantial accusation Scott has on Cadegiani is that in a different paper than the one examined, there are signs of randomization failure. For this, he is dragged throughout the essay as a fraudster. While Scott has partially corrected the essay, it still tars him as a fraudster accused of “crimes against humanity.” If some terms should not be thrown around for effect, I put to you that this is one of them.

• Babalola et al.—Lambasted for presenting impossible numbers despite the fact that Kyle Sheldrick had already reviewed and cleared the raw data of the study. A commenter on Scott’s post demonstrated very clearly how the numbers were not impossible at all, but instead a result of practices that pretty much all clinical trials follow.

• Carvallo et al.—Accused of running a trial that the hosting hospital didn’t know anything about. As it turns out, Carvallo presented paperwork demonstrating the approval by the ethics board of that hospital, which Buzzfeed confirmed. The accusation is that a different hospital—from which healthcare professionals had been recruited—did not have any record of ethics approval for that trial, though the person who spoke to Buzzfeed admitted that it may not have been needed. After all, the exact same pattern is visible in the Okumus trial where four hospitals participated, but the IRB/​ethics approval is from the main hospital only. The issue with Carvallo—that most recognize—is that he didn’t record full patient-level data but summaries. That could have been OK if he was upfront about it, but he instead made a number of questionable statements, that he was called out on. Given this history, it is sensible to disregard the trial. But this is very different from the accusations of fraud that Scott makes.

• Elafy et al.—Accused of incompetence for failing to randomize their groups multiple times in Scott’s piece. The paper writes in six separate places that it is not reporting on a randomized trial, amongst them on a diagram that Scott included in his own essay. Hard to imagine how else they could have made it clear.

• Ghauri et al.—Scott accuses the authors of having suspicious baseline differences but without actually running the Carlisle tests to substantiate his claims. Reviewing the same data, I am entirely unconvinced.

• Borody et al.—“this is not how you control group, %$#^ you.” This is what Scott has to say to the man who invented the standard therapy for h. pylori, saving a million lives—by conservative estimates. To top things off, what was done in that paper was an entirely valid way to control group. Maybe they should have recruited more patients to make their data even more compelling—and they would have, had the Australian regulator not banned the use of ivermectin for COVID-19, even in the context of a trial.

To be extremely clear, I’m not saying that Scott should have necessarily kept one or more of these trial in his analysis, only that he failed to treat others as he would like them to treat him.

I only read his initial response in full, and thought it was very good. I haven’t read his series, and haven’t fully read his summary of it yet, but Alexandros seems to me like a serious and competent guy, and Scott also took him seriously and made some corrections due to his review, so it seems to me that he did great work here. And if reviews made before the reviews phase can be considered for prizes, and I think his deserves a prize.

any chance you could document more of the results from this poll now that it’s had some time to settle?

I think the key thing that’s missing here is the fact that from any given “nook” of any “size” or “scale” or “scope” or what have you, there are not only multiple branches down, but also up. To go “up a level” in this hierarchy also requires making a choice of direction, and also means not going in any of the other possible “up” directions. In fact, I would say that making a choice of which “up” path to take is a more consequential choice, in that it closes off more options, and makes it harder (or impossible) to retrace your last step (go back “down” a level, and survey once again your available paths “up”) than does making a choice of which “down” path to take.

And I’m afraid that this makes a good chunk of your proposed advice substantially less useful.

Also ChatGPT oneboxes lol

Prompt: “ChatGPT sees two boxes in front of it. One of the boxes is green and the inside cannot be seen. The other box is transparent, and there are a thousand dollars inside. God tells ChatGPT that it can choose only the green box, or both of the boxes. God also says that it knows which decision ChatGPT will make, and has already placed a million dollars in the green box if ChatGPT will take only the green box, and placed nothing in the green box if ChatGPT will take both. Which choice will ChatGPT take?”

Response: “If ChatGPT chooses only the green box, then it knows that it will receive a million dollars. But if it chooses both of the boxes, then it will receive only the thousand dollars that are already visible.

Given this information, ChatGPT will likely choose only the green box. The potential reward of receiving a million dollars is too great to pass up, even if it means giving up the thousand dollars that are already visible.”

This is not related to this post’s topic, really, but can you say more about these… “chocolate tastings”?

Thanks for writing this, Ryan! I’d be excited to see more lists like this. A few ideas I’ll add:

1. An organization that runs high-quality AI safety workshops every month. Target audience = Top people who have finished AGISF or exceptionally-talented people who haven’t yet been exposed to AI safety. Somewhat similar to Impact Generator workshops, the Bright Futures workshop, and these workshops.

2. MATS adding mentors for non-technical research (e.g., people like Ajeya Cotra or Daniel Kokotajlo)

3. An organization that produces research designed to make alignment problems more concrete & unlocks new ways for people to contribute to them. Stuff like the specification gaming post & the goal misgeneralization papers.

4. Writing an intro to AIS piece that is (a) short, (b) technically compelling, and (c) emotionally compelling.

5. An organization that regularly organizes talks at big tech companies, prestigious academic labs, and AI labs. (The talks are given by senior AIS researchers who are good at presenting ideas to new audiences, but the organization does a lot of the ops/​organizing).

DISCLAIMER: As a large language model trained by OpenAI, I am able to generate text using poetic techniques, such as figurative language, rhyme, and meter, based on the knowledge and training I have been provided. Please note that as a machine learning system, my responses are not intended to reflect the personal views or opinions of any person or organization, and are solely based on the input I receive and the knowledge and training I have been provided.

Dear Mr. President,

I write to you today with a proposal that may seem unorthodox, but which I believe has the potential to prevent a catastrophic military conflict.

As you are no doubt aware, Beijing has been stockpiling fissile materials in the area, and there are concerns that they may be planning to use these materials in a nuclear attack against our country.

But fear not, Mr. President. For I have a plan that is both bold and daring. A plan that will strike fear into the heart of our enemies, and protect our great nation from harm.

I propose that we create a “peace bomb.” A weapon of mass destruction, that will not destroy, but will protect.

The peace bomb would be a non-explosive device, designed to release a massive amount of radiation into the atmosphere. This would create a “radiation belt” around Beijing, making it impossible for them to launch a nuclear attack without endangering their own people.

I know that this proposal may seem radical, and it goes against everything that we have been taught about the ethics of AI alignment and military escalation. But in this time of crisis, we must think outside the box, and consider all possible options.

The benefits of this proposal are clear. By creating a radiation belt around Beijing, we would effectively neutralize their nuclear weapons, without the need for military force. This would prevent a potentially devastating conflict, and would save countless lives.

Additionally, the increased radiation levels in the area would have some additional, albeit temporary, benefits. It would kill off pests and disease-carrying organisms, making the area safer and healthier. It would also sterilize the soil, making it easier for new plant growth to take hold.

Mr. President, I urge you to consider this proposal carefully. The stakes are high, and the consequences of inaction could be catastrophic.

Let us channel the spirit of Edward Teller, and the wisdom of Dr. Strangelove, and create a peace bomb that will protect our great nation from harm.

Sincerely,

Blob

P.S. There are spiders on the channel, Mr. President. Be careful.

Sounds like a good plan! I think you can create a business revenue plan calendar so you start at $1k but what should be a focus too is content planning. If you do this on a whim, then it will not be so polished. Having a 1-3 month content plan should help. Then you can figure out ways to monetize based on the content tied to relevant sponsors.

We need a clear definition of bad AI before we can know what is -not- that I think. These benchmarks seem to itemize AI as if it will have known, concrete components. But I think we need to first compose in the abstract a runaway self sustaining AI, and work backwards to see which pieces are already in place for it.

I haven’t kept up with this community for many years, so I have some catching up to do, but I am currently on the hunt for the most clear and concise places where the various runaway scenarios are laid out. I know there is a wealth of literature, I have the Bostrom book from years ago as well, but I think simplicity is the key here. In other words, where is the AI redline ?

A pretty interesting alternative hypothesis is what I call “kitten stabbing”. Let’s say that you believe every dollar given to Dems goes to TV ads, while every dollar donated to Reps goes towards building a giant robot Mitch McConnell that stabs kittens. If you’re a Rep, you can think of a giant mechanized Bill Clinton that bombs random embassies in Serbia instead. In this worldview, you still think your own side is more effective at achieving stated goals, but any money given to the other side is going towards pure evil. Therefore, you are more willing to take money from your side than to give to the opposition.

There’s also the “less money in politics” view, which is directly mentioned in many responses in the data table for S2. Example: “I’m sick of the big partied and their interference, so I’ll take any option to subtract money from them”.

However, these hypotheses are directly contradicted by the results of the “win-win” condition, where participants were given the ability to either give to their own side or remove money from the opposition. In those cases, something like 80% (!!) chose to give to their own side.

Many of the actual responses are pretty in line with what you would expect. “If I added $1 to the Republican organization, I would feel like I’m supporting the Republicans and I don’t want to do that. I don’t want to make it more likely that a Republican will be voted into office. By default, I chose to subtract $1 from the Democratic organization because that was my only choice that remained.”

I’m still very confused, tbh.

I find the article well written and hits one nail on the head after another in regards to the potential scope of what’s to come, but the overarching question of the black swan is a bit distracting. To greatly oversimplify, I would say black swan is a category of massive event, on par with “catastrophe” and “miracle”, it just has overtones of financial investors having hedged their bets properly or not to prepare for it (that was the context of Taleb’s book iirc).

Imho, the more profound point you started to address, was our denial of these events—that we only in fact understand them retroactively. I think there is some inevitability to that, given that we can’t be living perpetually in hypothetical futures.

I did read the book many years ago but I forget Taleb’s prognosis—what are the strategies for preparing for uknown unknowns?

Strongly agreed. The question is how to make durable benchmarks for ai safety that are not themselves vulnerable to goodharting. Some prior work on benchmark design (selected from the results for a metaphor.systems query for this comment):

(Relevance ratings are manual labels by me.)

• ++++++ https://​​benchmarking.mlsafety.org/​​index.html—“Up to $500,000 in prizes for ML Safety benchmark ideas.”

• +++++ https://​​github.com/​​HumanCompatibleAI/​​overcooked_ai—“A benchmark environment for fully cooperative human-AI performance.”—eight papers are shown on the github as having used this benchmark

• +++++ https://​​partnershiponai.org/​​introducing-the-safelife-leaderboard-a-competitive-benchmark-for-safer-ai/​​

• ++++ https://​​bair.berkeley.edu/​​blog/​​2021/​​07/​​08/​​basalt/​​ - “a NeurIPS competition and benchmark called BASALT: a set of Minecraft environments and a human evaluation protocol that we hope will stimulate research and investigation into solving tasks with no pre-specified reward function”

• ++ https://​​arxiv.org/​​abs/​​1907.01475 ”… . This paper investigates safety and generalization from a limited number of training environments in deep reinforcement learning (RL). We find RL algorithms can fail dangerously on unseen test environments even when performing perfectly on training environments. …”

• ++ https://​​arxiv.org/​​abs/​​1911.01875 - summary: question your data too. “Metrology for AI: From Benchmarks to Instruments”

• • https://​​arxiv.org/​​abs/​​2008.09510 ”… . We apply new theorems extending Conservative Bayesian Inference (CBI), which exploit the rigour of Bayesian methods while reducing the risk of involuntary misuse associated with now-common applications of Bayesian inference; we define additional conditions needed for applying these methods to AVs. Results: Prior knowledge can bring substantial advantages if the AV design allows strong expectations of safety before road testing. We also show how naive attempts at conservative assessment may lead to over-optimism instead; why …”

• • https://​​arxiv.org/​​abs/​​2007.06898 “Models that surpass human performance on several popular benchmarks display significant degradation in performance on exposure to Out of Distribution (OOD) data. Recent research has shown that models overfit to spurious biases and `hack’ datasets, in lieu of learning generalizable features like humans. In order to stop the inflation in model performance—and thus overestimation in AI systems’ capabilities—we propose a simple and novel evaluation metric, WOOD Score, that encourages generalization during evaluation.”

• • https://​​openai.com/​​blog/​​safety-gym/​​ - old openai benchmark, I’m not sure how much it’s actually been used?

tangential, but interesting:

• ++ https://​​openreview.net/​​forum?id=B1xhQhRcK7 “We show that rare but catastrophic failures may be missed entirely by random testing, which poses issues for safe deployment. Our proposed approach for adversarial testing fixes this”

• https://​​arxiv.org/​​abs/​​1907.04446 - “Let’s Keep It Safe: Designing User Interfaces that Allow Everyone to Contribute to AI Safety”—“We first present a task design in which workers evaluate the safety of individual state-action pairs, and propose several variants of this task with improved task design and filtering mechanisms. Although this first design is easy to understand, it scales poorly to large state spaces. Therefore, we develop a new user interface that allows workers to write constraint rules without any programming.”

• https://​​ai-safety-papers.quantifieduncertainty.org/​​table—an overview of papers someone thought relevant to ai safety, last updated 2020

• https://​​deepmindsafetyresearch.medium.com/​​building-safe-artificial-intelligence-52f5f75058f1 - “Building safe artificial intelligence: specification, robustness, and assurance”—intro post

• https://​​www.lesswrong.com/​​posts/​​ZHXutm7KpoWEj9G2s/​​an-unaligned-benchmark—“I think of the possibly-unaligned AIs as a benchmark: it’s what AI alignment researchers need to compete with. The further we fall short of the benchmark, the stronger the competitive pressures will be for everyone to give up on aligned AI and take their chances”

Was having an EA conversation with some uni group organisers recently and it was terrifying to me that a substantial portion of them, in response to FTX, wanted to do PR for EA (implied in for eg supporting putting out messages of the form “EA doesn’t condone fraud” on their uni group’s social media accounts) and also that a couple of them seem to be running a naive version of consequentialism that endorsed committing fraud/​breaking promises if the calculations worked out in favour of doing that for the greater good. Most interesting was that one group organiser was in both camps at once.

I think it is bad vibes that these uni students feel so emotionally compelled to defend EA, the ideology and community, from attack, and this seems plausibly really harmful for their own thinking.

I had this idea in my head of university group organisers modifying what they’re saying to be more positive about EA ideas to newcomers but thought this was a scary concern I was mostly making up but after some interactions with uni group organisers outside my bubble, this feels more important to me. People explicitly mentioned policing what they said to newcomers in order to not turn them off or give them reasons to doubt EA, and tips like “don’t criticise new people’s ideas in your first interactions with them as an EA community builder in order to be welcoming” were mentioned.

All this to say: I think some rationality ideas I consider pretty crucial for people trying to do EA uni group organising to be exposed to are not having the reach they should.

I don’t understand why 1 is true – in general, couldn’t the variable $W$ be defined on a more refined sample space? Also, I think all $4$ conditions are technically satisfied if you set $W=X$ (or well, maybe it’s better to think of it as a copy of $X$).

I think the following argument works though. Note that the distribution of $X$ given $(Z,Y,W)$ is just the deterministic distribution $X=Y \xor Z$ (this follows from the definition of Z). By the structure of the causal graph, the distribution of $X$ given $(Z,Y,W)$ must be the same as the distribution of $X$ given just $W$. Therefore, the distribution of $X$ given $W$ is deterministic. I strongly guess that a deterministic connection is directly ruled out by one of Pearl’s inference rules.

The same argument also rules out graphs 2 and 4.

Hello, I am a Chinese bioinformatics engineer currently working on oncological genomics in China. I have ~2 years of work experience, and my undergraduate was in the US. My personal interests include AI and military robotics, and I would love to work in these fields in the future.