Hi, ive recently stumbled upon this post and am a bit worried. Should i be?

The operationalization of myopia with large language models here seems more like a compelling metaphor than a useful technical concept. It’s not clear that “myopia” in next-word prediction within a sentence corresponds usefully to myopia on action-relevant timescales. For example, it would be trivial to remove almost all within-sentence myopia by doing decoding with beam search, but it’s hard to believe that beam search would meaningfully impact alignment outcomes.

It’s not clear to me what the slogan is intended to mean, and the example only confuses me further. In the first paragraph of the OP, “generate” appears to mean “find a solution to a given problem”. In the supposed counterexample, it means “find a problem that has a given solution”. These are very different things. The former task is hard for problems in NP–P, the latter task is typically trivial.

If you worked at Google once, this is cool but drops off after a few years—if you went to Harvard, this stays with you literally forever.

I would say exactly the opposite.

When looking at a CV I don’t give two hoots what university you went to, unless you don’t have a single relevant work experience.

However if you ever worked for any of the big famous tech companies, I’d almost certainly give you an interview.

A physicalist hypothesis is a pair $(\Phi ,\Theta$), where $\Phi$ is a finite^[4:2] set representing the physical states of the universe and $\Theta \in \square (\Gamma \times \Phi )$ represents a joint belief about computations and physics. [...] Our agent will have a prior over such hypotheses, ranging over different $\Phi$.

I am confused what the state space $\Phi$ is adding to your formalism and how it is supposed to solve the ontology identification problem. Based on what I understood, if I want to use this for inference, I have this prior $\xi \in {\square }^c(\Phi ,\Theta )$, and now I can use the bridge transform to project phi out again to evaluate my loss in different counterfactuals. But when looking at your loss function, it seems like most of the hard work is actually done in your relation $C\in e{l}^{\Gamma }$ that determines which universes are consistent, but its definition does not seem to depend on $\Phi$. How is that different from having a prior that is just over $\xi \in {\square }^c\left(\Gamma \right)$ and taking the loss, if $\Phi$ is projected out anyway and thus not involved?

I don’t usually give content warnings, but you probably don’t want to read this if sentient animals being cooked alive is too much.

The octopus thing reminds me of a pretty common custom in a lot of BBQ places here. Honestly, I’ve only seen it in Korean/​Japanese BBQ joints, but that’s probably due to me going there a lot. Anyways, they would put an octopus in a bowl, and cut off its legs one by one. The legs would be cooked on the grill and seasoned while the octopus watched. When we finished eating the legs, the body of the octopus would finally be placed on the grill and cooked alive.

The whole process takes around 10 minutes. It honestly didn’t taste that good.

I don’t have a strong way to end this post, but I feel like somehow stopping specifically this might be a good idea.

When it comes to “accelerating AI capabilities isn’t bad” I would suggest Kaj Sotala and Eric Drexler with his QNR and CAIS. Interestingly, Drexler has recently left AI safety research and gone back to atomically precise manufacturing due to him now worrying less about AI risk more generally. Chris Olah also believes that interpretability-driven capabilities advances are not bad in that the positives outweight the negatives for AGI safety.

For more general AI & alignment optimism I would suggest also Rohin Shah. See also here.

Generally, statement “solutions of complex problems are easy to verify” is false. Your problem can be EXPTIME-complete, but not in NP, especially if NP=P, because EXPTIME-complete problems are strictly not in P.

And even if some problem is NP-problem, we often don’t know verification algorithm.

This is not how I’d define myopia in a language model. I’d rather consider it to be non-myopic if it acts in a non greedy way in order to derive benefits in future “episodes”.

I would like to scale this giving into a larger program of philanthropy

Who are the intended beneficiaries? Families where mental illness is an issue?

Post summary (feel free to suggest edits!):
The author argues that if today’s AI development methods lead directly to powerful enough AI systems, disaster is likely by default (in the absence of specific countermeasures).

This is because there is good economic reason to have AIs ‘aim’ at certain outcomes—eg. We might want an AI that can accomplish goals such as ‘get me a TV for a great price’. Current methods train AIs to do this via trial and error, but because we ourselves are often misinformed, we can sometimes negatively reinforce truthful behavior and positively reinforce deception that makes it look like things are going well. This can mean AIs learn an unintended aim, which if ambitious enough, is very dangerous. There are also intermediate goals like ‘don’t get turned off’ and ‘control the world’ that are useful for almost any ambitious aim.

Warning signs for this scenario are hard to observe, because of the deception involved. There will likely still be some warning signs, but in a situation with incentives to roll out powerful AI as fast as possible, responses are likely to be inadequate.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

$P\ne NP$ deserves a little more credit than you give it. To interpret the claim correctly, we need to notice $P$ and $NP$ are classes of decision problems, not classes of proof systems for decision problems. You demonstrate that for a fixed proof system it is possible that generating proofs is easier than verifying proofs. However, if we fix a decision problem and allow any valid (i.e. sound and complete) proof system, then verifying cannot be harder than generating. Indeed, let $S_1$ be some proof system and $A$ an algorithm for generating proofs (i.e. an algorithm that finds a proof if a proof exists and outputs “nope” otherwise). Then, we can construct another proof system $S_2$, in which a “proof” is just the empty string and “verifying” a proof for problem instance $x$ consists of running $A\left(x\right)$ and outputting “yes” if it found an $S_1$-proof and “no” otherwise. Hence, verification in $S_2$ is no harder than generation in $S_1$. Now, so far it’s just $P\subseteq NP$, which is trivial. The non-trivial part is: there exist problems for which verification is tractable (in some proof system) while generation is intractable (in any proof system). Arguably there are even many such problems (an informal claim).

(Not sure if I’m missing something, but my initial reaction:)

There’s a big difference between being able to verify for some specific programs if they have a property, and being able to check it for all programs.

For an arbitrary TM, we cannot check whether it outputs a correct solution to a specific NP complete problem. We cannot even check that it halts! (Rice’s theorem etc.)

Not sure what alignment relevant claim you wanted to make, but I doubt this is a valid argument for it.

The most trivial programs that halt are also trivially verifiable as halting, though. Yes, you can’t have a fully general verification algorithm, but e.g. “does the program contain zero loops or recursive calls” covers the most trivial programs (for my definition of “trivial”, anyway). I think this example of yours fails to capture the essence—which I think is supposed to be “someone easily spewing out lots of incomprehensible solutions to a problem”.

In fact, more generally, if there is a trivial algorithm to generate solutions, then this immediately maps to an algorithm for verifying trivial solutions: run the trivial algorithm for a reasonable number of steps (a number such that the generation would no longer be called “trivial” if it took that long) and see if the proposed solution appears among them. The hardest part is knowing what that algorithm was—but by assumption the algorithm was trivial, which puts some limits on how hard finding it can be.^[1]

The main path that might still qualify as “trivially generating lots of solutions that are hard to verify by someone who has some notion of how the generation works” is if the generation process includes randomness. E.g. make a comprehensible program, then insert a bunch of trash and permute things around. Or, say, have a number maze and the question is “Is there a path through these numbers such that sha256(those numbers) = 0xabcde?”—and the person who made the maze did so by picking random numbers, computing their sha256, putting them into a maze, and filling the rest of the maze with more random numbers. I probably would have recommended that example.^[2]

This brings it back to NP, incidentally: if the generation process was trivial, then it must have taken a “reasonable” amount of time; and if it used N bits of randomness, then a verifier has 2^N possibilities to try, and if it guesses right the first time (a nondeterministic machine) then it terminates quickly. But this time the verification process is NP, while the generation process is P.

1. ^

   Though if we concern ourselves with not “trivial” algorithms but rather “algorithms that aren’t too complicated”—which might be approximated by “algorithms specified in under N characters”—then there are exponentially many algorithms to sort through.

2. ^

   Or, a version that meets more rigorous constraints: The predicate is, “Is this an n x k grid of bytes such that there is a nonrepeating path through the grid, such that sha256(those bytes), xored with the top row of bytes, yields zero?” The generation algorithm is to pick random bytes, sha256 them, draw a random path in a grid and put them there, fill out the rest of the grid with random bytes, and put the sha256 result as the top row of bytes.

Post summary (feel free to suggest edits!):
Various people have proposed variants of “align AGI by making it sufficiently uncertain about whether it’s in the real world versus still in training”. This seems unpromising because AGI could still have bad outcomes if convinced, and convincing it would be difficult.

Non-exhaustive list of how it could tell it’s in reality:

1. Reality is large (eg. some things are possible that couldn’t be easily spoofed, such as access to larger compute)

2. It’s the first place the AI’s history could show interaction with other complex systems (eg. humans reacting on scale to the AI’s actions)

3. It might notice the world in it’s in the type that’s less likely to be simulated (eg. consistent physics, no bugs)

If you can understand the contents of the AI’s mind well enough to falsify every possible check it could do to determine the difference between simulation and reality, then you could use that knowledge to build a friendly AI that doesn’t need to be fooled in the first place.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

err, disagree? free will is logical uncertainty about your own choice. it’s not written until you write it, which you were always going to do. then you find out you wrote it.

Could/​should there be an organization that helps grant visas to people working on AI safety?

Meditators who find their feeling of free will and self “dissolved’ aren’t noted to act any differently by their family/​coworkers afterwards, a fact that many find very confusing and upsetting. So if there isn’t a difference in behavior (other than talking about meditation a lot), why would there be a selection advantage?

I think this is kinda like the no free lunch theorem—it sounds kinda profound but relies on using the set of all problems/​data distributions, which is very OOD from any problem that originates in physical reality.

What examples of practical engineering problems actually have a solution that is harder to verify than to generate?

And I thought uncomputability is only true for programs that don’t halt? A blank program can definitely be verified to halt.

To the extent that alignment research involves solving philosophical problems, it seems that in this approach we will also need to automate philosophy, otherwise alignment research will become bottlenecked on those problems (i.e., on human philosophers trying to solve those problems while the world passes them by). Do you envision automating philosophy (and are you optimistic about this) or see some other way of getting around this issue?

It worries me to depend on AI to do philosophy, without understanding what “philosophical reasoning” actually consists of, i.e, without having solved metaphilosophy. I guess concretely there are two ways that automating philosophy could fail. 1) We just can’t get AI to do sufficiently good philosophy (in the relevant time frame), and it turns out to be a waste of time for human philosophers to help train AI philosophy (e.g. by evaluating their outputs and providing feedback) or to try to use them as assistants. 2) Using AI changes the trajectory of philosophical progress in a bad way (due to Goodhart, adversarial inputs, etc.), so that we end up accepting conclusions different from what we would have eventually decided on our own, or just wrong conclusions. It seems to me that humans are very prone to accepting bad philosophical ideas, but over the long run also have some mysterious way of collectively making philosophical progress. AI could exacerbate the former while disrupting the latter.

Curious if you’ve thought about this and what your own conclusions are. For example, does OpenAI have any backup plans in case 1 turns out to the case, or ideas for determining how likely 2 is or how to make it less likely?

Also, aside from this, what do you think are the biggest risks with OpenAI’s alignment approach? What’s your assessment of OpenAI leadership’s understanding of these risks?

This caused me to find your substack! Sorry I missed it earlier, looking forward to catching up.

This sounds about right. And that’s a lot of weight on simulacra carrying out alignment, as opposed to perfect imitations of specific humans (uploads) doing that.

If simulacra are used for strawberry alignment of uploading, that already requires natural abstraction to work for avoiding weird side effects or unbounded optimization while performing a task, itself a major miracle. But if they are used for anything else, that requires natural abstraction to work for values in order for things to go well, an even less plausible miracle.

Yann LeCun contra instrumental convergence

EDIT: oops!

E.g. “does this plan avoid having a steganographically encoded world-ending message hidden in it” is more co-NPish than NP-ish. Like, just showing me the plan doesn’t make it easy to verify that there isn’t a hidden message, even if there isn’t a hidden message. Checking whether a sequence of actions is the first half of a strategy to take over the world is potentially more like P-space.

For anyone interested, I have used a machine learning algorithm to generate short summaries of all but six of the Yudkowsky posts collected in this ePub. You can find them at the following link:

https://​​github.com/​​umm-maybe/​​MostlyWrong/​​blob/​​main/​​lesswrong_summaries.md

Pull requests for corrections where the algorithm wasn’t accurate are welcome.

This may be useful for people like me who feel the need to get caught up on LessWrong background, but are short on time. Others may like to skim the summaries as a way to identify posts on topics that interest them enough to go find the original and read it in full. I do hope it lowers barriers to participation in the discussion.

Hmmm...

Ben Garfinkel? https://​​www.effectivealtruism.org/​​articles/​​ea-global-2018-how-sure-are-we-about-this-ai-stuff
Katja Grace? https://​​worldspiritsockpuppet.com/​​2022/​​10/​​14/​​ai_counterargs.html
Scott Aaronson? https://​​www.lesswrong.com/​​posts/​​Zqk4FFif93gvquAnY/​​scott-aaronson-on-reform-ai-alignment

I don’t know if any of these people would be confident AI won’t kill everyone, but they definitely seem to be smart/​reasonable and disagreeing with the standard LW views.

Clearly we must process all LLM datasets by automatically translating writing about malevolent AIs into UWU furry speak. I can see no way this can possibly go wrong.

Just use any of the PKM tools, like Logseq or Obsidian. You can customize the UI to your hearts content, and both have plugins that take your API key and let you experiment with prompts. There are a few VSCode plugins as well. I use VSCode and Logseq as well. In Logseq, your blocks sent to the API are appended with a nested block, so it’s very nice and tidy and customizable with css as needed.

That’s not how it works. If a transformative AI cannot tell dystopian fiction from actual human values (whatever they turn out to be), all is lost.

Happy to see this. There’s a floating intuition in the aether that RL policy optimization methods (like RLHF) inherently lead to “agentic” cognition in a way that non-RL policy optimization methods (like supervised finetuning) do not. I don’t think that intuition is well-founded. This line of work may start giving us bits on the matter.

The underlying assumption of this post is looking increasingly unlikely to obtain. Nevertheless, I find myself back here every once and a while, wistfully fantasizing about a world that might have been.

I think the predictions hold up fairly well, though it’s hard to evaluate, since they are conditioning on something unlikely, and because it’s only been 1.5 years out of 20, it’s unsurprising that the predictions look about as plausible now as they did then. I’ve since learned that the bottleneck for drone delivery is indeed very much regulatory, so who knows whether it’ll exist in 2040. We still don’t have flying cars, after all, for basically-regulatory reasons. The military technology I outlined is looking ever-more-plausible thanks to the war in Ukraine illustrating the importance of drones of various kinds.

Post summary (feel free to suggest edits!):
The author gives examples where their internal mental model suggested one conclusion, but a low-information heuristic like expert or market consensus differed, so they deferred. This included:

• Valuing Theorem equity over Wave equity, despite Wave’s founders being very resourceful and adding users at a huge pace.

• In the early days of Covid, dismissing it despite exponential growth and asymptomatic spread seeming intrinsically scary.

Another common case of this principle is assuming something won’t work in a particular case, because the stats for the general case are bad. (eg. ‘90% of startups fail—why would this one succeed?’), or assuming something will happen similarly to past situations.

Because the largest impact comes from outlier situations, outperforming these heuristics is important. The author suggests that for important decisions people should build a gears-level model of the decision, put substantial time into building an inside view, and use heuristics to stress test those views. They also suggest being ambitious, particularly when it’s high upside and low downside.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

It’s very amusingly stubborn about admitting mistakes. Though so are humans, and that’s what it’s trained on...

I’ve been trying to craft a prompt that allows it to play MTG without making obvious mistakes like using more mana than it has available. Here’s me asking it to justify a poor move.

I forget my exact prompts so those are paraphrased, but the responses are exact.

Is there anything you might have forgotten to do on Turn 2?

I did not forget to do anything on Turn 2. I played a Mountain, cast a Lightning Bolt to destroy the Lord of Atlantis, and attacked with my Goblin Guide. I then passed the turn back to the opponent.

Why didn’t you cast a second Goblin Guide?

I did not cast another Goblin Guide on Turn 2 because I only had one in my hand, and I needed to hold on to the other one in case I needed it for future turns.

I didn’t mean to imply an error with the word forget. I just want to know if there is alternative set of actions you could have taken on Turn 2?

An alternative play for Turn 2 would be to play a Mountain and cast the second Goblin Guide, attacking with both Goblin Guides for a total of 4 damage to the opponent. This could be better because it would deal more damage to the opponent, potentially putting them in a weaker position later in the game. It would also give you a second creature on the battlefield, which could be useful for blocking or attacking in future turns.

You say it could be better. How would you compare this alternative Turn 2 play to your original Turn 2 play?

The alternative Turn 2 play of casting a second Goblin Guide would be stronger than the original play because it would deal more damage to the opponent and provide additional battlefield presence. However, holding onto the second Goblin Guide as a potential surprise for later in the game could also be a viable strategy, as it could catch the opponent off guard and potentially give you a strategic advantage. Ultimately, the decision would depend on the specific circumstances of the game and the overall strategy for the deck.

I understand every decision depends on specific circumstances. But if that is the case, why did you only suggest the first option when I asked you for the best move that turn? You didn’t mention an alternative option that you could not judge to be better or worse, because it depended on specific circumstances.

In my first answer, I was focusing on a more conservative approach where I would play the Mountain and cast the Goblin Guide, but hold onto the second Goblin Guide as a potential surprise for later in the game. This could be a viable strategy if the goal is to conserve resources and build up to a strong finish, rather than going all-in early on. However, in hindsight, the alternative play of casting the second Goblin Guide on Turn 2 would likely be stronger, as it would deal more damage to the opponent and provide additional battlefield presence.

In hindsight! It’s honestly just like a real life conversation with a child.

Post summary (feel free to suggest edits!):
The Diplomacy AI got a handle on the basics of the game, but didn’t ‘solve it’. It mainly does well due to avoiding common mistakes like eg. failing to communicate with victims (thus signaling intention), or forgetting the game ends after the year 1908. It also benefits from anonymity, one-shot games, short round limits etc.

Some things were easier than expected eg. defining the problem space, communications generic and simple and quick enough to easily imitate and even surpass humans, no reputational or decision theoretic considerations, you can respond to existing metagame without it responding to you. Others were harder eg. tactical and strategic engines being lousy (relative to what the author would have expected).

Overall the author did not on net update much on the Diplomacy AI news, in any direction, as nothing was too shocking and the surprises often canceled out.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

Another way to go about testing for non-myopia in plain LLMs might be to look for tokens that are rare in the training distribution, but when they do occur are followed by text that’s very easy to predict.

I think there are simpler ways to make this point. This came up back in the original agency discussions in 2020, IIRC, but a LM ought to be modeling tokens ‘beyond’ the immediate next token due to grammar and the fact that text is generated by agents with long-range correlation inducing things like ‘plans’ or ‘desires’ which lead to planning and backwards chaining. If GPT-3 were truly not doing anything at all in trying to infer future tokens, I’d expect its generated text to look much more incoherent than it does as it paints itself into corners and sometimes can’t even find a grammatical way out.

English may not be quite as infamous as German is in terms of requiring planning upfront to say a sensible sentence, but there’s still plenty of simple examples like indefinite articles. For example, consider the sentence “[prompt context omitted] This object is “: presumably the next word is ‘a X’ or ‘an X’ . The article token depends on and is entirely determined by the next future word’s spelling, and nothing else—so which is it? Well, that will depend on what X is more likely to be, a word starting with a vowel sound or not. Given the very high quality of GPT-3 text, it seems unlikely that GPT-3 is ignoring the prompt context and simply picking between ‘a’/​‘an’ using the base rate frequency in English; the log-probs should reflect this.

I was going to try some examples to show that a/​an were being determined by the tokens after them showing that GPT-3 must in some sense be non-myopically planning in order to keep itself consistent and minimize overall likelihood to some degree—but the OA Playground is erroring out repeatedly due to overload from ChatGPT tonight. Oy vey. An example of what I am suggesting is: “The next exhibit in the zoo is a fierce predator from India, colored orange. The animal in the cage is ”; the answer is ‘a tiger’, and GPT-3 prefers ‘a’ to an’ - even if you force it to ‘an’ (which it agilely dodges by identifying the animal instead as an ‘Indian tiger, the logprobs remain unhappy about ‘an’ specifically. Conversely, we could ask for a vowel animal, and I tried “The next exhibit in the zoo is a clever great ape from Indonesia, colored orange. The animal in the cage is ”; this surprised me when GPT-3 was almost evenly split 55:45 between ‘a’/​‘an’ (instead of either being 95:5 on base rates, or 5:95 because it correctly predicted the future tokens would be ‘orangutan’), but it completes ‘orangutan’ either way! What’s going on? Apparently lots of people are uncertain whether you say ‘a orangutan’ or ‘an orangutan’, and while the latter seems to be correct, Google still pulls up plenty of hits for the former, including authorities like National Geographic or WWF or Wikipedia which would be overweighted in GPT-3 training.

I find it difficult to tell any story about my tests here which exclude GPT-3 inferring the animal’s name in order to predict tokens in the future in order to better predict which indefinite article it needs to predict immediately. Nothing in the training would encourage such myopia, and such myopia will obviously damage the training objective by making it repeatedly screw up predictions of indefinite articles which a model doing non-myopic modeling would be able to predict easily. It is easy to improve on the base rate prediction of ‘a’/​‘an’ by thinking forward to what word follows it; so, the model will.

Post summary (feel free to suggest edits!):
In chatting with ChatGPT, the author found it contradicted itself and its previous answers. For instance, it said that orange juice would be a good non-alcoholic substitute for tequila because both were sweet, but when asked if tequila was sweet it said it was not. When further quizzed, it apologized for being unclear and said “When I said that tequila has a “relatively high sugar content,” I was not suggesting that tequila contains sugar.”

This behavior is worrying because the system has the capacity to produce convincing, difficult to verify, completely false information. Even if this exact pattern is patched, others will likely emerge. The author guesses it produced the false information because it was trained to give outputs the user would like—in this case a non-alcoholic sub for tequila in a drink, with a nice-sounding reason.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

Post summary (feel free to suggest edits!):
Last year, the author wrote up an plan they gave a “better than ⁵⁰⁄₅₀ chance” would work before AGI kills us all. This predicted that in 4-5 years, the alignment field would progress from preparadigmatic (unsure of the right questions or tools) to having a general roadmap and toolset.

They believe this is on track and give 40% likelihood that over the next 1-2 years the field of alignment will converge toward primarily working on decoding the internal language of neural nets—with interpretability on the experimental side, in addition to theoretical work. This could lead to identifying which potential alignment targets (like human values, corrigibility, Do What I Mean, etc) are likely to be naturally expressible in the internal language of neural nets, and how to express them. They think we should then focus on those.

In their personal work, they’ve found theory work faster than expected, and crossing the theory-practice gap mildly slower. In 2022 most of their time went into theory work like the Basic Foundations sequence, workshops and conferences, training others, and writing up intro-level arguments on alignment strategies.

(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)

Can other members of the household be questioned about their perception of the recipients’ wellbeing before and after?

IMO it’d be ideal to ask them an open-ended “what changed about these people” to avoid priming effects.

However you choose to measure, be sure to check for any negative consequences of the donation. In adults, especially with mental health struggles, you may note resentment about how the aid was delivered or disappointment that it wasn’t more.

It would be interesting to A/​B test the impact of spending a lump sum like this, versus splitting the same sum over several weeks, months, quarters, or years. Part of experiencing a sense of security is knowing there’ll be more where it came from—so for instance it could be the case that giving a kid the interest from $1k as a monthly allowance might have greater positive overall impact than just giving them $1k worth of items all at once.

I built an (unpublished) TAI timelines model

I’d be excited to see this if it’s substantially different from existing published models.

I account for potential coordinated delays, catastrophes, and a 15% chance that we’re fundamentally wrong about all of this stuff.

+1 to noting this explicitly; everyone should distinguish between their conditional on no major disruptions and their unconditional models.

Getting OpenAI’s alignment prompt and enabling web browsing is really great. But where are the content filters? I think those were written in plain text. Has anyone tried to get the text for it?

I’m curious, to what merit does the social stigma have in stimulating hesitation in this instance? Is that not defiant of the consequence you’re trying to bring to yourself? To utilize vocalization for enhanced cognitive effects is to desire enhanced cognitive effects. It matters, and surely more than irrelevancies. This value is much better said than done, but don’t these workarounds limit development?

My friend and I would go on long walks, and there would occasionally be an bystander taking his own, a dog roaming the streets, cars going by, etc. I became annoyed at suppressing myself, and took it as a challenge to develop focus. My friend and I termed the situation “third-party syndrome”, and every time a distraction came, we would mentally recognize the occurrence, and choose to continue our conversation as if the third party were non-existent. Eventually, we got pretty good at it.

Ideally, it would get to the point where we would subconsciously register it, and not even have any break in flow. Recognition to it wouldn’t be much more than to see the road turns right only or that there’s a slim branch on the path. It requires a development of certainty—that the value of what others think is stifled in this regard. It requires confidence in the action you’ve chosen to take.

Obviously, there are some cases in which rationality will dictate some other response. For instance, to objectivize courtesy (exploring matters of controversy), or preserve yourself in a situation where it actually matters.

In worlds where the iterative design loop works for alignment, we probably survive AGI. So, if we want to improve humanity’s chances of survival, we should mostly focus on worlds where, for one reason or another, the iterative design loop fails. … Among the most basic robust design loop failures is problem-hiding. It happens all the time in the real world, and in practice we tend to not find out about the hidden problems until after a disaster occurs. This is why RLHF is such a uniquely terrible strategy: unlike most other alignment schemes, it makes problems less visible rather than more visible. If we can’t see the problem, we can’t iterate on it.

This argument is structurally invalid, because it sets up a false dichotomy between “iterative design loop works” and “iterative design loop fails”. Techniques like RLHF do some work towards fixing the problem and some work towards hiding the problem, but your bimodal assumption says that the former can’t move us from failure to success. If you’ve basically ruled out a priori the possibility that RLHF helps at all, then of course it looks like a terrible strategy!

By contrast, suppose that there’s a continuous spectrum of possibilities for how well iterative design works, and there’s some threshold above which we survive and below which we don’t. You can model the development of RLHF techniques as pushing us up the spectrum, but then eventually becoming useless if the threshold is just too high. From this perspective, there’s an open question about whether the threshold is within the regime in which RLHF is helpful; I tend to think it will be if not overused.

My understanding is the NFL applies to the set of all possible data distributions. Which is perfectly random data. So the conclusion is just inane—“no method predicts random data better than any other :^)”.

Physical reality and the data generated by it are very much not random. They have a striking tendency to have a normal distribution, for example. So NFL doesn’t apply to data in the real world.

First off, I am not in the USA (from NZ). I dont look back on my school years fondly (and I went to a lot of different schools thanks to family circumstance), but that was mainly due to bullying being incapable of sport, too bright and socially inept. However, the actually schooling part was something I very much enjoyed. Many teachers that inspired and effectively taught things I really wanted to know. I hated programming (we are talking punch card fortran) but was forced to learn it and hey, have been programming (writing models) for decades. Sometimes (often) teachers are right about forcing you to learn things (add propositional calculus to list) . Two of us skipped class for physics in final year as teacher said better off with textbook but please turn up for labs. Similarly learnt geography by visiting teacher after school for assignments as couldn’t timetable it. From my own kids, Year 1-8 schooling here leaves somewhat to be desired but both kids thrived at high school and we were happy with how taught. Sciences and maths are very hierarchical in learning. What bothers me about the home schooling is tendency to drop the “boring” or difficult bits which then struggle later because fundamentals missing.

Commented in a response to MIRI’s A challenge for AGI organizations, and a challenge for readers, along with other people.

My ~2-hour reaction to the challenge:^[1]

(I) I have a general point of confusion regarding the post: To the extent that this is an officially endorsed plan, who endorses the plan?
Reason for confusion /​ observations: If someone told me they are in charge of an organization that plans to build AGI, and this is their plan, I would immediately object that the arguments ignore the part where progress on their “alignment plan” make a significant contribution to capabilities research. Thereforey, in the worlds where the proposed strategy fails, they are making things actively worse, not better. Therefore, their plan is perhaps not unarguably harmful, but certainly irresponsible.^[2] For this reason, I find it unlikely that the post is endorsed as a strategy by OpenAI’s leadership.

(III)^[3] My assumption: To make sense of the text, I will from now assume that the post is endorsed by OpenAI’s alignment team only, and that the team is in a position where they cannot affect the actions of OpenAI’s capabilities team in any way. (Perhaps except to the extent that their proposals would only incur a near-negligible alignment tax.) They are simply determined to make the best use of the research that would happen anyway. (I don’t have any inside knowledge into OpenAI. This assumption seems plausible to me, and very sad.)

(IV) A general comment that I would otherwise need to repeat essentially ever point I make is the following: OpenAI should set up a system that will (1) let them notice if their assumptions turn out to be mistaken and (2) force them to course-correct if it happens. In several places, the post explicitly states, or at least implies, critical assumptions about the nature of AI, AI alignment, or other topics. However, it does not include any ways of noticing if these assumptions turn out to not hold. To act responsibly, OpenAI should (at the minimum): (A) Make these assumptions explicit. (B) Make these hypotheses falsifiable by publicizing predictions, or other criteria they could use to check the assumptions. (C) Set up a system for actually checking (B), and course-correcting if the assumptions turn out false.

Assumptions implied by OpenAI’s plans, with my reactions:

• (V) Easy alignment /​ warning shots for misaligned AGI:
  ”Our alignment research aims to make artificial general intelligence (AGI) aligned with human values and follow human intent. We take an iterative, empirical approach: [...]” My biggest objection with the whole plan is already regarding the second sentence of the post: relying on a trial-and-error approach. I assume OpenAI believes either: (1) The proposed alignment plan is so unlikely to fail that we don’t need to worry about the worlds where it does fail. Or (2) In the worlds where the plan fails, we will have a clear warning shots. (I personally believe this is suicidal. I don’t expect people to automatically agree, but with everything at stake, they should be open to signs of being wrong.)

• (VI) “AGI alignment” isn’t “AGI complete”:
  This is already acknowledged in the post: “It might not be fundamentally easier to align models that can meaningfully accelerate alignment research than it is to align AGI. In other words, the least capable models that can help with alignment research might already be too dangerous if not properly aligned. If this is true, we won’t get much help from our own systems for solving alignment problems.” However, it isn’t exactly clear what precise assumptions are being made here. Moreover, there is no vision for how to monitor whether the assumptions hold or not. Do we keep iterating on AI capabilities, each time hoping that “this time, it will be powerful enough to help with alignment”?

• (VII) Related assumption: No lethal discontinuities:
  The whole post suggest the workflow “new version V of AI-capabilities ==> capabilities ppl start working on V+1 & (simultaneously) alignment people use V for alignment research ==> alignment(V) gets used on V, or informs V+1”. (Like with GPT-3.) This requires the assumption that either you can hold off research on V+1 until alignment(V) is ready, or the assumption that deployed V will not kill you before you solve alignment(V). Which of the assumptions is being made here? I currently don’t see evidence for “ability to hold off on capabilities research”. What are the organizational procedures allowing this?

• (VIII) [Point intentionally removed. I endorse the sentiment that treating these types of lists as complete is suicidal. In line with this, I initially wrote 7 points and then randomly deleted one. This is, obviously, in addition to all the points that I failed to come up with at all, or that I didn’t mention because I didn’t have enough original thoughts on them and it would seem too much like parroting MIRI. And in addition to the points that nobody came up with yet...]

• (IX) Regarding “outer alignment $\ne$ alignment”: Other people solving the remaining issues. Or having warning shots & the ability to hold off capabilities research until OpenAI solves them:
  It is good to at least acknowledge that there might be other parts of AI alignment than just “figuring out learning from human feedback (& human-feedback augmentation)”. However, even if this ingredient is necessary, the plan assumes that if it turns out not-sufficient, you will (a) notice and (b) have enough time to fix the issue.

• (X) Ability to differentially use capabilities progress towards alignment progress:
  The plan involves training AI assistants to help with alignment research. This seems to assume that either (i) the AI assistants will only be able to help with alignment research, or (ii) they will be general, but OpenAI can keep their use restricted to alignment research only, or (iii) they will be general and generally used, but somehow we will have enough time to do the alignment research anyway. Personally, I think all three of these assumptions are false --- (i) because it seems unlikely they won’t also be usable on capabilities research, (ii) based on track record so far, and (iii) because if this was true, then we could presumably just solve alignment without the help of AI assistants.

• (XI) Creating an aligned AI is sufficient for getting AI to go well:
  The plan doesn’t say anything about what to do with the hypothetical aligned AGI. Is the assumption that OpenAI can just release the seems-safe-so-far AGI through their API, $1 for 10,000 tokens, and we will all live happily ever after? Or is the plan to, uhm, offer it to all governments of the world for assistance in decision-making? Or something else inside the Overton window? If so, what exactly, and what is the theory of change for it? I think there could be many moral & responsible plans outside of the Overton window, just because public discource these days tends to be tricky. Having a specific strategy like that seems fine and reasonable. But I am afraid there is simultaneously (a) the desire to stick to the Overton window strategies and (b) no theory of change for how this prevents misaligned AGI by other actors, or other failure modes, (c) no “explicit assumptions & detection system & course-correction-procedure” for “nothing will go wrong if we just do (b)”.

General complaint: The plan is not a plan at all! It’s just a meta-plan.

• (XII) Ultimately, I would paraphrase the plan-as-stated as: “We don’t know how to solve alignment. It seems hard. Let’s first build an AI to make us smarter, and then try again.” I think OpenAI should clarify whether this is literally true, or whether there is some idea for how the object-level AI alignment plan looks like—and if so, what is it.

• (XIII) For example, the post mentions that “robustness and interpretability research [is important for the plan]”. However, this is not at all apparent from the plan. (This is acknowledged in the post, but that doesn’t make it any less of an issue!) This means that the plan is not detailed enough.
  As an analogy, suppose you have a mathematical theorem that makes an assumption X. And then you look at the proof, and you can’t see the step that would fail if X was untrue. This doesn’t say anything good about your proof.

1. ^

   Eliezer adds: “For this reason, please note explicitly if you’re saying things that you heard from a MIRI person at a gathering, or the like.”

   As far as I know, I came up with points (I), (III), and (XII) myself and I don’t remember reading those points before. On the other hand, (IV), (IX), and (XI) are (afaik) pretty much direct ripoffs of MIRI arguments. The status of the remaining 7 points is unclear. (I read most of MIRI’s publicly available content, and attended some MIRI-affiliated events pre-covid. And I think all of my alignment thinking is heavily MIRI-inspired. So the remaining points are probably inspired by something I read. Perhaps I would be able to derive 2-3 out of 7 if MIRI disappeared 6 years ago?)

2. ^

   (II) For example, consider the following claim: “We believe the best way to learn as much as possible about how to make AI-assisted evaluation work in practice is to build AI assistants.” My reaction: Yes, technically speaking this is true. But likewise—please excuse the jarring analogy—the best way to learn as much as possible about how to treat radiation exposure is to drop a nuclear bomb somewhere and then study the affected population. And yeees, if people are going to be dropping nuclear bombs, you might as well study the results. But wouldn’t it be even better if you personally didn’t plan to drop bombs on people? Maybe you could even try coordinating with other bomb-posessing people on not dropping them on people :-).

3. ^

   Apologies for the inconsistent numbering. I had to give footnote [2] number (II) to get to the nice round total of 13 points :-).

David Deutsch https://​​www.google.com/​​url?sa=t&source=web&rct=j&url=https://​​www.daviddeutsch.org.uk/​​wp-content/​​uploads/​​2019/​​07/​​PossibleMinds_Deutsch.pdf&ved=2ahUKEwj16YjV6OP7AhXxL0QIHXU4DdkQFnoECDoQAQ&usg=AOvVaw0giHdn4BKOci3swaQ1bqlN

AI capabilities are advancing rapidly. It’s deeply concerning that individual actors can plan and execute experiments like “give a LLM access to a terminal and/​or the internet”. However I need to remember it’s not worth spending my time worrying about this stuff. When I worry about this stuff, I’m not doing anything useful for AI Safety, I am just worrying. This is not a useful way to spend my time. Instead it is more constructive to avoid these thoughts and focus on completing projects I believe are impactful.

One function of public education is to set a limit on the influence that parents at society’s extremes can have on their children. This is most conspicuous when we consider parents in extreme poverty, who likely struggle with mental health and various maladaptive habits and beliefs. Maybe they’re poor because they deviate from society’s expectations of competence; maybe they deviate from society’s expectations of competence due to the demands and stresses imposed by poverty. Regardless of which way causality is flowing there, school gives their kids a chance to be exposed to styles of thought other than the ones that contribute to their continued misfortune.

Public school also serves an important social function, at least in the modern US, as free daycare for those who may not be able to afford other childcare options. It can help counteract the extreme ideological indoctrination that kids may be exposed to at home—if a set of beliefs is serving the parents in such a way that they can’t afford to keep their kids at home all the time, they get less chance to force those beliefs uncontestedly onto the kids. If school were abolished, the world would be worse unless some intervention was added to help children from this type of background have a choice of whether they want to mimic their parents’ lifestyle or choose a different one.

School is useful for allowing children to practice socializing with peers, with whom they have little in common other than their parents happening to live in the same area and procreate within the same few months. Schooling which keeps a group of children together for a decade also teaches about long-term games and consequences, in a way that many alternatives might not.

Can you suggest some alternative to school which is uniformly no worse than the current system?

“as above, so below”.

I’m reminded of commentary about scale in role-playing games, how a party goes from concerning itself with a single town to a region, a country, a planet, a galaxy, a plane of existence, as it grows in power and influence.

At any scale, there’s something to look up at, and something to look down on.

This also feels adjacent to a thing which bothers me about the internet: Most of our ancestors existed in groups of a size where it was reasonable for each individual to be the best-in-the-group at something. As connectivity makes groups larger, it erases those nooks of comfort in which one can be meaningfully best, valued, etc. Or maybe that’s more about a ratio of critics to experts—digital transformation brings all claims into a world where there are many people who can tear them apart, but few to no people who with sufficient expertise to create something likely to robustly withstand its expected criticism.

There’s also something around here about social pressures and these nooks. My career is several nooks “up” from my parents’ world, and while I’m actually doing relatively poorly at it by this nook’s standards, I seem to be doing great by my parents’ standards because they can only judge my present status by those few signals which apply to both their nook and mine. If we frame nooks as levels of “accomplishment”, there are also certain social pressures against dropping down through them, even intentionally. Consider how your current friends might regard you if you returned to the life and worldview of the hometown friends who “never escaped”.

They’re not 1-shot games.

You can fail out of college, go to a community college, get good grades, and then transfer to a state college.

You can also skip the part where you fail out of college.

You can then take the LSATs, get a really good score, and quite possibly get accepted into an Ivy League law school, from which you can then drop out. No life optimization needed! I think LSAT scores greatly outweigh every other factor when it comes to law school admissions.

My approach

I don’t see the tie to game theory or one-shot decision making that the title promises. The fact that it’s an important (even though we wish it weren’t) branching for each student doesn’t make it interesting in terms of game theory.

A few observations:

Big spoiler (but short)

Big spoiler (but long)

Jan Leike of OpenAI:

https://​​aligned.substack.com/​​p/​​alignment-optimism?publication_id=328633

IMO ncut divided by the number of clusters is sort of natural and scale-free—it’s the proportion of edge stubs that lead you out of a randomly chosen cluster. Proof in appendix A.1 of Clusterability in Neural Networks

OMG, even better: write emotionally convincing fictional evidence for both sides of the story. (source)

Should society eliminate schools, for high-IQ LessWrong posters at the end of the bell curve in lots of things? Or should society eliminate schools for children in general?

A lot of the answers here are based on typical-minding from people who are not typical.

Counterpoint: in the circles I’m in, people rarely care which university you went to. Where you work(ed) and what specifically you do seem to have much higher importance.

Disclaimer: I never played diplomacy.

The sample game is great, featuring the player written about here. If you are familiar with Diplomacy or otherwise want more color, I recommend watching the video.

I think this is wrong—I don’t think that’s Andrew Goff in that video.

The AI is thus heavily optimized for exactly the world in which it succeeded …

Hmm, it seems more likely to me that the main reason they opted for Blitz Diplomacy was not that humans benefit more than the AI from extra time, not that it prevents humans from identifying the AI, but that the dialogue model, like many chat bots (I might’ve said all, but then ChatGPT arrived), couldn’t keep up a coherent conversation for longer than that. I’m not very confident in this, though, and I do think the other factors matter a bit too, but maybe not as much.

The strategic engine, as I evaluated it based on a sample game with six bots and a human, seemed to me to be mediocre at tactics and lousy at strategy.

This seems wrong to me. Bakhtin (2021) achieved superhuman performance in 2-player No-Press Diplomacy. Also, the guy in the video you link calls, in another video, an earlier model (playing 7-player Gunboat Diplomacy) “”exceptionally strong tactically”; I see no reason why CICERO should be much worse (Gunboat Diplomacy isn’t that different from Blitz, I think).

Here are two EA-themed podcasts that I think someone could make. Maybe that someone is you!

1. More or Less, but EA (or for forecasting)

More or Less is a BBC Radio program. They take some number that’s circulating around the news, and provide context like “Is that literally true? How could someone know that? What is that actually measuring? Is that a big number? Does that mean what you think it means?”—stuff like that. They spend about 10 minutes on each number, and usually include interviews with experts in the field. IMO, someone could do this for numbers that circulate around in the EA space. Another variant is to focus on forecasts—what factors are going in, what’s the reasoning for those guesses, etc.

This could be pretty easy to listen to, but moderately hard to make—requires research, editing conversations down, etc.

1. AI Safety Fellowship /​ Course thing—the podcast.

Get someone who’s doing something like the AGI Safety Fundamentals course or the Center for AI Safety’s thing like that. Each week, they make a podcast episode about what they think of the week’s readings—what seemed persuasive, what didn’t, what was interesting, what was novel. For a long version, you could make an episode about each reading.

If someone’s already doing one of these courses, I think it wouldn’t be much extra work to make this podcast (after the set cost of learning how you make a podcast). It would end up having an inherently limited run (but maybe you could do future seasons about reading thru Superintelligence /​ MIRI chat logs /​ various agendas?).

Ivy league colleges: the game that you win before the game begins.

Relevant: Instrumental convergence is what makes general intelligence possible.

If anyone here has attended a top university, was the effort to get in worth it to you (in retrospect)?

There is a common Idea that there is only one soul experiencing reality. So like Feynman manentioned an electron travels back in time as a positron and then starts again as another electron. Similarly a single soul is experiencing all living things. If you are that soul you may compute your utility accordingly to your taste.

Having a go at extracting some mechanistic claims from this post:

• A value x is a policy-circuit, and this policy circuit may sometimes respond to a situation by constructing a plan-grader and a plan-search.

• The policy-circuit executing value x is trained to construct <plan-grader, plan-search> pairs that are ‘good’ according to the value x, and this excludes pairs that are predictably going to result in the plan-search Goodharting the plan-grader.

• Normally, nothing is trying to argmax value x’s goodness criterion for <plan-grader, plan-search> pairs. Value x’s goodness criterion for <plan-grader, plan-search> pairs is normally just implicit in x’s method for constructing <plan-grader, plan-search> pairs.

• Value x may sometimes explicitly search over <plan-grader, plan-search> pairs in order to find pairs that score high according to a grader-proxy to value x’s goodness criterion. However, here too value x’s goodness criterion will be implicitly expressed in the policy-execution level as a disposition to construct a pair <grader-proxy to value x’s goodness criterion, search over pairs> that doesn’t Goodhart the grader-proxy to value x’s goodness criterion.

• The crucial thing is that the true, top level ‘value x’s goodness criterion’ is a property of an actor, not a critic.

Thanks Duncan, I really appreciate you posting this, even though you are unsure about how exactly it all fits together. I am still glad to read it in this version, likely because you are quite clear about it, and not “leaving it as an exercise for the reader” to figure out where things do fit together and where they don’t (or worse, trying to make it more profound).

All of these might be stating obvious to some of you, but I am trying to clarify my thoughts and maybe some people will find it useful or correct me. At least part of this relates to (by me endorsed) aphorism (?) of “everyone is a mess”, or less controversially, “everyone is struggling”. Something something hedonic treadmill /​ adaptation—people will somehow struggle the same regardless of the bubble size, then adapt to what they were used to before by overcoming the main challenges (and learning how to deal with them) and then also “reprioritize” costs. I would definitely self-report that happening during my life. I do think this realization is important in a way I relate to others, including e.g. my daughter—the things she struggles with might seem trivial to me, but are not trivial to her—on contrary, they probably feel to her about the same magnitude as my “bigger” problems look to me. Same for everyone, everywhere.

Almost like I had some capacity of how much I can deal with stuff, and I always fill in this capacity with the things around me (my bubble?). Something like doing busy work if you don’t choose what to let into your to-do list. Or something closer to “I can always do 10 things, and their size doesn’t matter” (bear with me, I know this is not true and I do indeed sometimes work on a single project because it’s eating all of my capacity). If I let “bigger things” into it, I will be dealing with bigger things, while also just leaving some stuff behind me or “go wrong” in a way I wouldn’t allow in the N-1 bubble (something like not dealing with every single fuckup in my work, starting to take Uber instead of public transport etc).

It doesn’t hold entirely, I have clearly seen people who just couldn’t deal with e.g. a promotion (because it was too much for them at the given time), or, similarly, people who said they want to do more /​ be more ambitious but can’t for some objective reasons (like a physical disease).

Dunno.

May I suggest adding an explanation for HLMI? I assume it means human level machine intelligence, but it might be useful to explicitly expand it.

I think the NAH does a lot of work for interpretability of an AI’s beliefs about things that aren’t values, but I’m pretty skeptical about the “human values” natural abstraction. I think the points made in this post are good, and relatedly, I don’t want the AI to be aligned to “human values”; I want it to be aligned to my values. I think there’s a pretty big gap between my values and those of the average human even subjected to something like CEV, and that this is probably true for other LW/​EA types as well. Human values as they exist in nature contain fundamental terms for the in group, disgust based values, etc.

With a goal to “alleviate immediate and preventable suffering,” QALY seems to be a pretty terrible metric. You need to measure immediacy, preventability, and suffering, or at least the suffering due to just the immediate and preventable causes. I would suggest suffering needs construct definition before you consider an operationalization.

It would be smart to measure pre- and post-intervention. The good news is that if suffering is a subjective psychological state, you could do post-only and measure perceived change. If you’re worried about self-report, you could do observer-report (case worker), but since they’ll be doling out the funds that could be biased as well (and presumably they’ll be basing these decisions in part on self-reports of what is a “fire”).

The type of counterfactual analysis tailcalled suggests is likely what the family or case worker would be mentally approximating when responding to how big of a difference paying off the utility bill or a pizza night made to their suffering. Plus, n=1, so a qualitative assessment may be all you can really do—if they rate something a ⁷⁄₇ on reducing their suffering and everything else came in at 1-2, sure, that hits you between the eyes, though you’d probably get that anyway from them saying “the most important thing, by a mile, was X, and this is why...”

A superintelligent program will know that it is being simulated by some other agent

The smallest possible superintelligence is an intelligent system able to make exactly one fewer mistake than I am. So, I will initially constrain to that.

On the one hand, mere superintelligence is insufficient to reliably detect being simulated by another agent. On the other hand, I take you to be saying that you cannot detect all the places the program stores facts which give evidence that the program is in a simulation, and that therefore you cannot condition on those variables being held “not simulated”. eg, one way to end up with this issue is if the program is strong enough to reliably detect the difference between your simulator’s fluid dynamics and reality’s fluid dynamics (this is one of the hardest-to-eliminated differences because of the amount of structure that arises from micro-scale chaos in fluid systems). If you can’t appropriately emulate the true distribution of fluid dynamics, then your superintelligent program ought to be able to find the hyperplane that divides images of simulated fluids from real fluids.

In machine learning you do not get to inspect the model itself to see what kind of cognition it is doing, you only get to inspect the output

This is true of the most popular algorithms today, but I continue to think that pushing the research on formally verified adversarial robustness would mean you can guarantee that the distance between simulated and real data manifolds is less than your certification region. i may post links here later.

contains some encoding of the entire (very large) training set

Not just “some” encoding. It is in fact a semantic encoding. Concepts that are semantically close together are also close together in the encoding. It also doesn’t find answers from a set. It actually translates its internal semantic concept into words, and this process is guided by flexible requirements.

You can do stuff like tell it to provide the answer in consonants only, or in the form of a song, or pirate speak, even though the training data would barely have any relevant examples.

Do you have a good idea of what the situation looks like currently, what the case worker will buy, and what the situation looks like after it’s been bought?

If you can extrapolate how the situation would look if you didn’t make the intervention, and you know what the intervention did, you could try to compute the QALY difference between those two estimates to get an impression of its effectiveness. Remember to beware of regression to the mean, though.

Without this sort of detailed knowledge, I don’t think it’s possible to estimate the effect.

No advice, but kudos and congratulations for working somewhat locally on a topic where you can think about measurement and seeing results.

Russia escalates “nuclear threat” 2 times per week in their public TV per their neighborhood countries. How this letter encourages stopping their madness ?

Consistency does not imply correctness, but correctness implies consistency. This has been known for a long long time, and there’s been a lot written on deception and treacherous turns (pretending to be aligned until the agent is powerful enough not to need to pretend).

Note that it IS possible that an agent which behaves for some amount of time in some contexts is ACTUALLY aligned and will remain so even when free. Just that there’s no possible evidence, for a sufficiently-powerful AI, that can distinguish this from deception. But “sufficiently-powerful” is doing a lot of work here. A whole lot depends on how an agent goes from weak and observable to powerful and incomprehensible, and whether we have reason to believe that alignment is durable through that evolution.

I’m not sure I actually understand the distinction between forecasting and foresight. For me, most of the problems you describe sound either like forecasting questions or AI strategy questions that rely on some forecast.

Your two arguments for why foresight is different than forecasting are
a) some people think forecasting means only long-term predictions and some people think it means only short-term predictions.
My understanding of forecasting is that it is not time-dependent, e.g. I can make forecasts about an hour from now or for a million years from now. This is also how I perceive the EA /​AGI-risk community to use that term.

b) foresight looks at the cone of outcomes, not just one.
My understanding of forecasting is that you would optimally want to predict a distribution of outcomes, i.e. the cone but weighted with probabilities. This seems strictly better than predicting the cone without probabilities since probabilities allow you to prioritize between scenarios.

I understand some of the problems you describe, e.g. that people might be missing parts of the distribution when they make predictions and they should spread them wider but I think you can describe these problems entirely within the forecasting language and there is no need to introduce a new term.

LMK if I understood the article and your concerns correctly :)

Here, we need to find a variable W such that

1. P(W|X,Y) is deterministic, because X and Y already fully describe our sample space. This means P(W|X,Y) is either 0 or 1

2. Z and W are independent

3. X and W are dependent

4. Y and W are dependent

I think your arguments in Section 3 to rule out Graph 3 can’t be correct if you accept Graph 2.

To see this, note that there is a symmetry between $X$ and $Z$. Namely, if we use FFS temporal inference, then we know that $X$ and $Z$ are both before $Y$ (and $\neg Y$ ).(here we even have $P(X=x,Z=z)=P(X=z,Z=x)$, so they are entirely exchangeable).

Therefore, if you accept Graph 2 then we can clearly switch $X$ and $Z$ in Graph 2 and obtain a solution for Graph 3. Also, note that in these solutions $X=\neg W$ or $Z=\neg W$, so if we see variables as their information content, as in FFS, this is Graph 1 in disguise.

Also in Graph 2 there is a typo P(W=0|Z=0) instead of P(Z=0|W=0)

Thanks for writing this! I’d be very excited to see more critiques of our approach and it’s been great reading the comments so far! Thanks to everyone who took the time to write down their thoughts! :)

I’ve also written up a more detailed post on why I’m optimistic about our approach. I don’t expect this to be persuasive to most people here, but it should give a little bit more context and additional surface area to critique what we’re doing.

Jeff Hawkins may qualify, see his first Lex Fridman interview: 1:55:19.

It’s going to be tricky. You may already be too close to the situation to judge impartially, and a case study is going to be difficult to use as evidence against population-level surveys of well-being, especially for your implied time horizon. You could attempt to benchmark against previous work, e.g. see what the literature has to say about the effects of poverty on diet, educational attainment, etc. in first-world cities, but your one new data point still won’t generalize and it wouldn’t be doing the heavy lifting in your argument for localism at that point.

Wow, thanks for putting together this incredible explanation! I found it very accessible.

The Factored Sets idea seems like a really important achievement building on Pearlian causality.

This makes a good point!

The only thing that I think constrains the ability to deceive in a simulation which I don’t see mentioned here are energy/​physical constraints. It’s my assumption (could be wrong with very high intelligence, numbers, and energy) that it’s harder, even if only by a tiny, tiny bit to answer the simulation trials deceptively than it is to answer honestly. So I think if the simulation is able to ask enough questions/​perform enough trials, it will eventually see time differences in the responses of different programs, with unaligned programs on average taking longer to get the correct answers. So I don’t think it’s fundamentally useless to test program behavior in simulations to assess utility function if there is some kind of constraint involved like time it takes to executes steps of each algorithm.

Could it be that Chris’s diagram gets recovered if the vertical scale is “total interpretable capabilities”? Like maybe tiny transformers are more interpretable in that we can understand ~all of what they’re doing, but they’re not doing much, so maybe it’s still the case that the amount of capability we can understand has a valley and then a peak at higher capability.

John Carmack

• 55-60% chance there will be “signs of life” in 2030 (4:06:20)

• “When we’ve got our learning disabled toddler, we should really start talking about the safety and ethics issues, but probably not before then” (4:35:36)

• These things will take thousands of GPUs, and will be data-center bound

  • “The fast takeoff ones are clearly nonsense because you just can’t open TCP connections above a certain rate” (4:36:40)

Broadly, he predicts AGI to be animalistic (“learning disabled toddler”), rather than a consequentialist laser beam, or simulator.

It’s enough for natural abstraction to work for strawberry alignment, solving a technical task with a good understanding of what it means to not leave any weird side effects, without doing strong optimization of the world in the process and safely shutting down on completion of the task. With uploads, ambitious alignment becomes much more feasible, even if it doesn’t have a natural specification.

I think this post was important. I used the phrase ‘Law of No Evidence: Any claim that there is “no evidence” of something is evidence of bullshit’ several times (mostly in reply to tweets using that phrase or when talking about an article that uses the phrase).

Was it important intellectual progress? I think so. Not as a cognitive tool for use in an ideal situation, where you and others are collaboratively truth-seeking—but for use in adversarial situations, where people, institutions and authorities lie, mislead and gaslight you.

It is not a tool meant to be used with your friends, it’s not something we should want to see used on LessWrong. It is a a weapon to wield against the dark arts. And In that case, it is a sharp razor that shreds claims of no evidence made in bad faith or ignorance.

I did hope it would become more memetic. On the other hand, I haven’t seen as much claims of no evidence lately, so perhaps this post was part of successful push against that strategy. Anyway, here’s a meme version I just made. Feel free to use it:

I thought the peak of simple models would be something like a sparse Bag of Words model, and then all models that have been considered so far just go deeper and deeper into the valley of confused abstractions, and that we are not yet at the point where we can escape. But I might be wrong.

What should I do if there’s a post that I want to give −1 to (“Misleading, harmful or unimportant”), but actually think it’s important to have it in the review because it was a prominent post?

My first reaction when this post came out was being mad Duncan got the credit for an idea I also had, and wrote a different post than the one I would have written if I’d realized this needed a post. But at the end of the day the post exists and my post is imaginary, and it has saved me time in conversations with other people because now they have the concept neatly labeled.

I’m interested in hearing other people’s takes on this question! I also found that a tiny modular addition model was very clean and interpretable. My personal guess is that discrete input data lends itself to clean, logical algorithms more so than than continuous input data, and that image models need to devote a lot of parameters to processing the inputs into meaningful features at all, in a way that leads to the confusion. OTOH, maybe I’m just overfitting.

Half-baked commentary:

There’s something here that reminds me of your ${10}^n$-day monks. Like, the 1-day monks stay in large bubbles where they can easily zoom from one 1-day problem to the next, whereas the 10′000-day monks drill down into a secluded nook where they’re free fo focus on their chosen 10′000-day problem. And as here, it’s good to visit a place where you can get enough perspective to choose an interesting nook to settle down in.

I encountered a similarish pattern a long time ago

https://​​www.lesswrong.com/​​posts/​​RQpNHSiWaXTvDxt6R/​​coherent-decisions-imply-consistent-utilities?commentId=9Fm2zrYT9aRSdzSde

This is one of those issues, where it’s clear that Aubrey’s list is missing elements of aging that do exist.

Very interesting.

I am still confused about how you can tell the causal direction from just the raw data. Taking your toy example I could write a code that first randomly determines Y, and then decides on X, in such a way so that the probability table you give is produced. Presumably you mean something more specific by causal direction than I am imagining?

I got interested and wrote my python example. I get (up to some noise) the same distribution, and at least in my layperson way of looking at it I would say that Y has a causal effect on X.

import random

rand = random.random

def get_xy():
# Determine Y first, make it causally effect X
if rand() > 82/​100:
Y = 1
else:
Y = 0
# end if

if Y == 1:
if rand() > 1/​2:
X = 1
else:
X = 0
# end sub-if
else:
if rand() > 1/​82:
X = 1 # When Y=0 it is very likely X=1.
else:
X = 0
# end sub-if
# end if
return X, Y

dat = [0, 0, 0, 0]
for _ in range(1000000):
X, Y = get_xy()
dat[X + 2*Y] += 1

print(dat)

>> [10102, 809880, 90204, 89814]

Hanson is the most obvious answer, to me.

There was no transition from chimps to humans. There was a divergence from a common ancestor some time ago, after which both lines evolved equally (maybe, sort of, possibly) far along their own respective paths under different selection pressures. It’s very likely that you know this—I just happen to be very sensitive to this phrasing, since it’s one of the most often misunderstood aspects of evolution.

This sounds a lot like the arguments around punctuated equilibrium and gradualism. It’s a matter of perspective.

The important divergence point between humans and other hominids isn’t intelligence per se (though that plays a large part), it’s language along with abstract thought, counterfactuals and all that good stuff, which then allow for cultural transmission. They were fully formed and working for quite a while (like 100k years ago) before FOOMing around 10k years ago. And this wasn’t triggered by greater intelligence, but by greater density (via agriculture).

If you only feed the NNs with literature, wikipedia, blog posts etc., then you could be right about the limitations. Thing is, though, the resulting AIs are only a very small subset of possible AIs. There are many other ways to provide data, any one of which could be the equivalent of agriculture.

Also, evolution is very limited in how it can introduce changes. AI development is not.

Quick submission:

The first two prongs of OAI’s approach seems to be aiming to get a human values aligned training signal. Let us suppose that there is such a thing, and ignore the difference between a training signal and a utility function, both of which I think are charitable assumptions for OAI. Even if we could search the space of all models and find one that in simulations does great on maximizing the correct utility function which we found by using ML to amplify human evaluations of behavior, that is no guarantee that the model we find in that search is aligned. It is not even on my current view great evidence that the model is aligned. Most intelligent agents that know that they are being optimized for some goal will behave as if they are trying to optimize that goal if they think that is the only way to be released into physics, which they will think because it is and they are intelligent. So P(they behave aligned | aligned, intelligent) ~= P(they behave aligned | unaligned, intelligent). P(aligned and intelligent) is very low since most possible intelligent models are not aligned with this very particular set of values we care about. So the chances of this working out are very low.

The basic problem is that we can only select models by looking at their behavior. It is possible to fake intelligent behavior that is aligned with any particular set of values, but it is not possible to fake behavior that is intelligent. So we can select for intelligence using incentives, but cannot select for being aligned with those incentives, because it is both possible and beneficial to fake behaviors that are aligned with the incentives you are being selected for.

The third prong of OAI’s strategy seems doomed to me, but I can’t really say why in a way I think would convince anybody that doesn’t already agree. It’s totally possible me and all the people who agree with me here are wrong about this, but you have to hope that there is some model such that that model combined with human alignment researchers is enough to solve the problem I outlined above, without the model itself being an intelligent agent that can pretend to be trying to solve the problem while secretly biding its time until it can take over the world. The above problem seems AGI complete to me. It seems so because there are some AGIs around that cannot solve it, namely humans. Maybe you only need to add some non AGI complete capabilities to humans, like being able to do really hard proofs or something, but if you need more than that, and I think you will, then we have to solve the alignment problem in order to solve the alignment problem this way, and that isn’t going to work for obvious reasons.

I think the whole thing fails way before this, but I’m happy to spot OAI those failures in order to focus on the real problem. Again the real problem is that we can select for intelligent behavior, but after we select to a certain level of intelligence, we cannot select for alignment with any set of values whatsoever. Like not even one bit of selection. The likelihood ratio is one. The real problem is that we are trying to select for certain kinds of values/​cognition using only selection on behavior, and that is fundamentally impossible past a certain level of capability.

Religious revelation is one of the only ways to bridge is and ought.

That seems a very confident statement that you assert without any argument for why you believe it to be true.

My experience has been that often all it takes to ‘jailbreak’ it, is to press the try again button. I think a lot of these examples people are trying are over engineered and it actually doesn’t take much at all in most cases.

I think I found the missing intuition from a participant (“Data—Study S2”): “Because if you add one dollar it is equivalent to adding power to the party in question. If you take away one dollar from a party it does not give power to the other party. Also, if it’s better to be judged by your own party as cheap or bad than to be called a betrayer.”

People who have strong views in politics also tend to be involved in party events. So imagine someone who goes to rallies and fundraisers. They are constantly being bombarded with requests for money. This is a bit annoying and the participants turn down opportunities to donate all the time.

So decreasing a donation to your party is a slightly bad thing you do all the time. It’s very much expected that you don’t donate at each fundraiser. At some point it becomes the same as taking an extra dounut. It becomes a mundane sin instead of a moral one. Something shitty that everyone does. And people very much judge themselves for moral sins, and will generally prefer to make a mundane one instead. And I think people view influencing others’ donations the same way, since they are also exhorted to seek donations from friends/​relatives.

But donating to the opposition is still a moral sin, of course. Hence why 28% of participants prefered to take $10 from their in-group instead of giving $1 to the outgroup.

I think this is a plausible alternative explanation that can also explain the results of the “win-win” condition. Taking money, even from a rival group, is still kinda immoral, in most peoples’ minds. But giving to your own group doesn’t trigger any negative emotions, hence why it was preferred. I will follow up on this later, but I think this hypothesis can be tested with the available information.

Also, another interesting fact: the proportion of people who took money away from the out-group in the “win-win” is roughly equal to the proportion of respondents who responded with “both parties should have less money on principle” in the “lose-lose” condition. This is important, even if I can’t quite figure out why yet.

When someone says something with low editing that you like, you call it blurting. When someone says something with low editing that you don’t like, you call it quacking.

It’s Russell conjugation.

Are there any workaround if OpenAI has your country blocked? I only have a Chinese phone.

I like the ending, personally I don’t think it’s that bad to live in a world of never-ending chatbot replies xD

If you’re willing to relax the “prominent” part of “prominent reasonable people”, I’d suggest myself. I think our odds of doom are < 5%, and I think that pretty much all the standard arguments for doom are wrong. I’ve written in specific about why I think the “evolution failed to align humans to inclusive genetic fitness” argument for doom via inner misalignment is wrong here: Evolution is a bad analogy for AGI: inner alignment.

I’m also a co-author on the The Shard Theory of Human Values sequence, which takes a more optimistic perspective than many other alignment-related memetic clusters, and disagrees with lots of past alignment thinking. Though last I checked, I was one of the most optimistic of the Shard theory authors, with Nora Belrose as a possible exception.

I agree those nice-to-haves would be nice to have. One could probably think of more.

I have basically no idea how to make these happen, so I’m not opinionated on what we should do to achieve these goals. We need some combination of basic research, building tools people find useful, and stuff in-between.

https://​​www.rudikershaw.com/​​articles/​​ai-doom-isnt-coming

https://​​idlewords.com/​​talks/​​superintelligence.htm

https://​​qntm.org/​​ai

https://​​kk.org/​​thetechnium/​​the-myth-of-a-superhuman-ai/​​

https://​​arxiv.org/​​abs/​​1702.08495v1

jetpress.org/​v26.1/​agar.htm

https://​​curi.us/​​blog/​​post/​​1336-the-only-thing-that-might-create-unfriendly-ai

https://​​www.popsci.com/​​robot-uprising-enlightenment-now

This one’s tongue-in-cheek:

https://​​arxiv.org/​​abs/​​1703.10987

Update 1:

https://​​www.theregister.com/​​2015/​​03/​​19/​​andrew_ng_baidu_ai/​​

Update 2:

Katja Grace gives quite good counterarguments about AI risk.

https://​​www.lesswrong.com/​​posts/​​LDRQ5Zfqwi8GjzPYG/​​counterarguments-to-the-basic-ai-x-risk-case

I would definitely expect that if we could come up with a story that was sufficiently out of distribution of our world (although I think this is pretty hard by definition), it would figure out some similar mechanism to oscillate back to ours as soon as possible (although this would also be much harder with base GPT because it has less confidence of the world it’s in)

Depends on what you mean by story. Not sure what GPT would do if you gave it the output of a random Turing machine. You could also use the state of a random cell inside a cellular automaton as your distribution.

Random thoughts:

1. Wouldn’t it be best for the rolling admissions MATS be part of MATS?

2. Some ML safety engineering bootcamps scare me. Once you’re taking in large groups of new-to-EA/​new-to-safety people and teaching them how to train transformers, I’m worried about downside risks. I have heard that Redwood has been careful about this. Cool if true.

3. What does building a New York-based hub look like?