Johannes C. Mayer

• Johannes C. Mayer 26 Nov 2023 18:02 UTC

  1 point

  0

  on: [Er­ror com­mu­ni­cat­ing with LW2 server]

  Filler Filler

  This is a test comment.

• Johannes C. Mayer 25 Nov 2023 14:19 UTC

  3 points

  on: Jo­hannes C. Mayer’s Shortform

  How do you define an algorithm that samples a random natural number, according to a probability distribution that assigns non-zero weight to every natural number? Meditate on it, before reading the solution.

  ---

  def sample_nat(p_heads=0.5):
      i = 0
      while True:
          if coin_flip(p_heads) == 'heads':
              return i
          i += 1
  

  With p_heads=0.5 we implicitly define the probability distribution: $$P(n:N):=\frac{1}{2^{(n+1)}}$$

  This program is pretty weird because the probability that it will not have halted after n steps is non-zero, for any n.

  [Edit 2023-11-26]

  Now I understand what is going on. This program is in fact a non-halting program, that halts with probability 1.

  This is similar to how the probability that you get a rational number when you sample uniformly from $[0,1]\subseteq R$ is zero.

  (I am unsure about this paragraph.) In practice, this depends somewhat on what pseudo-random number generator we use. The pseudo-random number generator that we use might be such that there is no possible initial state, such that the generated sequence would be all tails, in which case the program would be guaranteed to terminate.

  [END Edit]

• Johannes C. Mayer 23 Nov 2023 13:34 UTC

  1 point

  on: Ro­bust­ness to Scale

  It seems there is a generalization of this. I expect there to be many properties of an AI system and an environment such that if the value of the property changes alignment doesn’t break.

  The hard part is to figure out which properties are most useful to pay attention to. Here are a few:

  • Capability (as discussed in the OP)

    • Could be something very specific like the speed of compute

    • Pseudo Cartesianess (a system might be effectively cartesian at a certain level of capability before it figures out how to circumvent some constraints we put on it)

  • Alignment (ideally we would like to detect and shut down if the agent becomes misaligned)

  • Complexity of the system (maybe your alignment scheme rests on being able to understand the world model of the system, in which case it might stop working as we move from modeling toy worlds to modeling the real world)

  • etc.

  ---

  I think it can be useful to consider the case where alignment breaks as you decrease capabilities. For example, you might think of constructing a minimal set of assumptions such that you would know how to solve alignment. One might be, having an arbitrary amount of compute and memory available that can execute any halting program arbitrarily fast. If we want to remove this assumption it might break alignment. It’s pretty easy to see how alignment could break in this case, but it seems useful to have the concept of the generalized version.

Sat­u­rat­ing the Difficulty Levels of Alignment

• Johannes C. Mayer 21 Nov 2023 22:26 UTC

  1 point

  in reply to: Johannes C. Mayer’s comment on: Jo­hannes C. Mayer’s Shortform

  Upon reflection, I’m unsure what you mean by the program being simpler. What is your preferred way to represent modular addition? I could of course write down 20 % 11. I know exactly what that means. But first of all, this is not an algorithm. It just talks about the concept of modular arithmetic without specifying how to compute it. And understanding the concept at a high level is of course easier than representing the entire algorithm all at once in my mind.

  I guess the normal way you would compute the modulo would be to take a number $a$ and then subtract $b$ from it until what is left is smaller than $b$. What is left is then the modulo. Ok, that seems simpler so never mind.

  It does seem an important distinction to think about the way we represent a concept and the actual computation associated with obtaining the results associated with that concept. I got confused because I was conflating these two things.

• Johannes C. Mayer 18 Nov 2023 14:55 UTC

  1 point

  in reply to: Neel Nanda’s comment on: Jo­hannes C. Mayer’s Shortform

  Do you think this might be a significant obstacle in the future? For example, do you think it is likely that the algorithms inside of an AGI-neural-network built by SGD will be so complicated that they are not humanly understandable, because of their sheer size? I am especially thinking about the case where an algorithm exists that is just as capable but understandable.

  This seems more likely if we end up with an AGI-neural-network that mushes together the world model and the algorithms that use the world model (e.g. update it, use it to plan), such that there are no clear boundaries. If the AGI is really good at manipulating the world, it probably has a pretty good model of the world. As the world contains a lot of algorithmic information, the AGI’s model of the world will be complex. If the system is mushed we might need to understand all that complexity to an intractable extent.

  I expect that if you can have a system where the world model is factored out into its own module, it will be easier to handle the complexity in the world because then we can infer properties of the world model based on the algorithms that construct and use it. I expect the world model will still be very complex, and the algorithms that construct and use it will be simple. Therefore infering properties of the world model based on these simple algorithms might still be tractable.

  Do you think this problem is likely to show up in the future?

• Johannes C. Mayer 18 Nov 2023 12:58 UTC

  1 point

  on: Jo­hannes C. Mayer’s Shortform

  Imagine all possible programs that implement a particular functionality. Imagine we have a neural network that implements this functionality. If we have perfect mechanistic interpretability we can extract the algorithms of a neural network that implements that functionality. But what kind of program do we get? Maybe there are multiple qualitatively different algorithms that all implement the functionality. Some of them would be much easier to understand for human. The algorithm the neural network finds might not be that program that is easiest to understand to a human.

[Question] Is there Work on Embed­ded Agency in Cel­lu­lar Au­tomata Toy Models?

[Question] Would this be Progress in Solv­ing Embed­ded Agency?

• Johannes C. Mayer 13 Nov 2023 14:46 UTC

  7 points

  on: Jo­hannes C. Mayer’s Shortform

  I made this collage of people I think are cool and put it in my room. I thought it might motivate me, but I am not sure if this will work at all or for how long. Feel free to steal. Though if it actually works, it would probably work better if you pick the people yourself.

The Science Al­gorithm AISC Project

• Johannes C. Mayer 13 Nov 2023 10:12 UTC

  4 points

  2

  on: The Fun­da­men­tal The­o­rem for mea­surable fac­tor spaces

  Consider adding more context at the beginning of the post about why the reader should care. Something like:

  “Finite factored sets have the fundamental theorem, which is X. This is an analogous theorem but for the more general case of using measurable spaces. It is formulated using the causal graph construction.”

  This is important because it allows us to solve “Concrete problem P” which we were not able to solve before. (Or whatever the reason is that this is progress.) An alternative to this would be to first introduce a problem that is clearly important and then show how finite factored sets can’t handle this problem very well.

  Of course, if there are writeups doing these things you may supplement links to them. Though even then it is probably good to give a very brief summary still.

  I think making these changes would be almost no work, compared to doing all the math, and it would make it a lot more accessible. Right now this text seems to assume that you are intimately familiar with finite factored sets, to the extent that you would not even look up again what the fundamental theorem for finite factored sets is (there is no link to it at the appropriate point).

  As I don’t quite understand what you did, some of the details in my suggestions might be wrong, but I think the overall points I am trying to make should still hold up.

• Johannes C. Mayer 11 Nov 2023 11:19 UTC

  1 point

  0

  in reply to: RogerDearnaley’s comment on: Pivotal Acts might Not be what You Think they are

  Letting Loose a Rogue AI is not a Pivotal Act

  It seems likely that governments who witnessed an “almost AI takeover” will try to develop their own AGI. This could happen even if they understand the risks. They might think that other people are rushing it, and that they could do a better job.

  But if they don’t really understand the risks, like right now, then they are even more likely to do it. I don’t count this as a pivotal act. If you can get the outcome you described then it would be a pivotal act, but the actions you propose would not have that outcome with high probability. I would guess with much less than 50%. Probably less than 10%.

  There might be a version of this, with a much more concrete plan, such that we can see that the outcome would actually follow from executing the plan.

  On Having an AI explain how Alignment is Hard

  I think your second suggestion is interesting. I’d like to see you write a post about it exploring this idea further.

  If we build a powerful AI and then have it tell us about all the things that can go wrong with an AI, then we might be able to generate enough scientific evidence about how hard alignment is, and how likely we will die, e.g. in the current paradigm, such that people would stop.

  I am not talking about conceptual arguments. To me at least I think the current best conceptual arguments already strongly point in that direction. But extremely concrete rigorous mathematical arguments, or specific experimental setups that show how specific phenomena do in fact arise. For example, if you had an experiment that showed that Eliezer’s arguments are correct, that when you train hard enough on a general enough objective, you will in fact get out more and more general cognitive algorithms, up to and including AGI. If the system also figures out some rigorous formalisms to correctly present these results, then this could be valuable.

  The reason why this seems good to me, at first sight, is that false positives are not that big of an issue. If an AI finds all the things that can go wrong, but 50% of them are false positives in the sense that they would not be a problem in practice, we may get saved, because we are aware of all the ways things can go wrong. When solving alignment, false positives, i.e. thinking that something is safe when it is not, kill you.

  Intuitively it also seems that evaluating whether something describes a failure case is a lot easier than evaluating whether something can’t fail.

  When doing this you are much less prone to the temptation of delaying a system in practice with the insights you got. Understanding failure modes does not necessarily imply that you know how to solve them (though it is the first step, and definitely can do this).

  Pitfalls

  That being said there are a lot of potential pitfalls with this idea, but I don’t think they disqualify the idea:

  • An AI that could tell you how alignment is hard might already be so capable that it is dangerous.

  • When telling you how things are dangerous, it might formulate concepts that are also very useful for advancing capabilities.

  • If the AI is very smart it could probably trick you. It could present to you a set of problems with AI, such that it looks like if you solved all the problems you would have solved alignment. But in fact, you would still get a misaligned AGI that would then reward the AI that deceived you.

    • E.g. if the AI roughly understands its own cognitive reasoning process, and notices how it is not really aligned, it would give the AI information about what parts of alignment the humans have figured out already.

  • Can we make the AI figure out the useful failure modes? There are tons of failure modes, but ideally, we would like to discover new failure modes such that eventually we can paint a convincing picture of the hardness of the problem. An even better would be a list of problems corresponding to having new important insights (though this would go beyond what this proposal tries to do).

  Prefered Planning

  Let’s go back to your first pivotal act proposal. I think I might have figured out where you miss-stepped.

  Missing step plans are a fallacy, and thinking of them I realize that I think you probably committed another type of planning fallacy here. I think you generated a plan and then assumed some preferred outcome would occur. That outcome might be possible in principle but not what would happen in practice. This seems very related to The Tragedy of Group Selectionism.

  This fallacy probably shows up when generating plans, because if there are no other people involved and the situation is not that complex, it is probably a very good heuristic. When you are generating a plan, you don’t want to fill in all the details of the plan. You want to make the planning problem as easy as possible. So our brain might implicitly make the assumption that we are going to optimize for the successful completion of the plan. That means that the plan can be useful as long as it roughly points in the correct direction. Mispredicting an outcome is fine, because later on when you realize that the outcome is not what you wanted, you can just apply more optimization pressure, changing the plan, such that now the plan again has the desired outcome. As long as you were walking roughly in the right direction, and things you have been doing so far don’t turn out to be completely useless, this heuristic is great for reducing the computational load of the planning task.

  Details can be filled in later, corrections can be made later. At least as long as you will reevaluate your plan later on. You could do this by reevaluating the plan when:

  • A step is completed

  • You notice a failure when executing the current step

  • You notice that the next step has not been filled in yet.

  • After a specific amount of time passed.

  Sidenote: Making an abstract step more concrete might seem like a different operation from regenerating in the case where you notice that the plan does not work. But it could just involve the same planning procedure. In one case with a different starting point, and in the other with a different set of constraints.

  I expect part of the failure mode here is that you generate a plan and then to evaluate the consequences of the plan, you implicitly plug yourself into the role of the people who would be impacted by the plan, to predict their reaction. Without words, you think “What would I do if I observed a rouge AI almost taking over the world, if I were China?” Probably without realizing, that this is what you are doing. But the resulting prediction is wrong.

• Johannes C. Mayer 7 Nov 2023 15:12 UTC

  1 point

  0

  on: [Er­ror com­mu­ni­cat­ing with LW2 server]

  I am not sure. It seems I figured out some high level stuff that I am not sure other GOFAI people have figured out yet, by thinking for tens of hours. When skiming AI a modern approach I did not find these insights in there.

  Test dialog comment.

• Johannes C. Mayer 7 Nov 2023 12:35 UTC

  1 point

  0

  in reply to: Sammy Martin’s comment on: Pivotal Acts might Not be what You Think they are

  I think you are correct, for a particular notion of pivotal act. One that I think is different from Eliezer’s notion. It’s certainly different from my notion.

  I find it pretty strange to say that the problem is that a pivotal act is a single action. Everything can be framed in terms of a single action.

  For any sequence of actions, e.g.[X, Y, Z] I can define a new action ω := [X, Y, Z], which executes X, then Y, and then Z. You can do the same for plans. The difference between plans and action sequences is that plans can have things like conditionals. For example, choosing the next sequence of actions based on the current state of the environment. You could also say that a plan is a function that tells you what to do. Most often this function takes in your model of the world.

  So really you can see anything you could ever do as a single plan that you execute. If there are multiple steps involved you simply give a new name to to all these steps, such that you now have only a single thing. That is how I am thinking about it. After this definition, we can have a pivotal act that is composed of many small actions that are distributed across a large timespan.

  The usefulness of the concept of a pivotal act comes from the fact that a pivotal act needs to be something that saves us with a very high probability. It’s not important at all that it happens suddenly, or that it is a single action. So your criticism seems to miss the mark. You are attacking the concept of a pivotal act for having properties that it simply does not have.

  “Upload a human” is something that requires many steps dispersed throughout time. We just use the name “Upload a human” such that we don’t need to specify all of these individual steps in detail. That would be impossible right now anyway, as we don’t know exactly how to do it.

  So if you provide a plan that is composed of many actions distributed throughout time, that will save us with a very high probability, I would count this as a pivotal act.

  Note that being a pivotal act is a property of a plan in relation to the territory. There can be a plan P that saves us when executed. But I might fail to predict this. I.e. it is possible to misestimate is_pivotal_act(P). So one reason for having relatively simple, abstract plans like “Upload a human”, is that these plans specify a world state, with easily visible properties. In the “Upload a human” example we would have a superintelligent human. Then we can evaluate the is_pivotal_act property, and based on that we have created a superintelligent human. I am heavily simplifying here, but I think you get the idea.

  I think your “positively transformative AI” just does not capture what a pivotal act is about (I haven’t read the article, I am guessing based on the name). You could have positive transformative AI, that makes things increasingly better and better, by a lot. And then somebody builds a misaligned AGI and everybody dies. One doesn’t exclude the other.

• Johannes C. Mayer 6 Nov 2023 7:28 UTC

  1 point

  0

  in reply to: faul_sname’s comment on: Pivotal Acts might Not be what You Think they are

  Yes, if you can make the article’s contents be in all the brains that would be liable to accidentally create a misaligned AGI, now and in the future, and we also assume that none of these brains want to intentionally create a misaligned AGI, then that would count as a pivotal act in my book.

  This might work without the assumption that nobody wants to create a misaligned AGI, through a different mechanism than described in the OP. Then it seems relatively likely that there is enough oomph to push for effective regulations.

• Johannes C. Mayer 6 Nov 2023 7:14 UTC

  1 point

  0

  in reply to: Raemon’s comment on: Pivotal Acts might Not be what You Think they are

  Thank you for the feedback. That’s useful.

  I agree that you need to be very careful about who you upload. There are less than 10 people I would be really confident in uploading. That point must have been so obvious in my own mind that I forgot to mention it.

  Depending on the setup I think an additional important property is how resistant the uploaded person is, to going insane. Not because the scan wasn’t perfect, or the emulation engine is buggy, but because you would be very lonely (assuming you only upload one person and don’t immediately clone yourself) if you run that much faster. And you need to handle some weird stuff about personal identity that comes up naturally, through cloning, simple self-modifications, your program being preempted by another process, changing your running speed, etc.

• Johannes C. Mayer 5 Nov 2023 17:54 UTC

  3 points

  0

  on: Pivotal Acts might Not be what You Think they are

  In retrospect, I am somewhat confused about what I am trying to do with this article. I am glad that I did publish it, because I too frequently don’t publish writeups that are almost complete. It all started out by trying to give a brief summary that people could look at to get less confused about pivotal acts. Basically in a different writeup, a person said that it is unclear what I mean by pivotal act. Instead of writing this article, I should probably just have added a note to the original article that pivotal act actually means something very specific and that it is easily conflated with other things, and then linked to the original article. I did link to the original article in the document I was writing, but apparently, that was not enough.

  I think it does an okay job of succinctly stating the definition. I think stating some common confusions is probably a good thing to do, to preemptively prevent people from falling into these failure modes. And the ones I listed seem somewhat different from the ones in the original article. So maybe they add a tiny bit of value.

  I think the most valuable thing about writing this article is to make me slightly less confused about pivotal acts. Before writing this article my model was implicitly telling me that any pivotal act needs to be directly about putting some entity into a position of great enough power that it can do the necessary things to save the world. After writing this article it is clear that this is not true. I now might be able to generate some new pivotal acts that I could not have generated before.

  If I want the articles I write to be more valuable to other people, I should probably plan things out much more precisely and set a very specific scope for the article that I am writing beforehand. I expect that this will decrease the insights I will generate for myself during writing, but make the writing more useful to other people.

Pivotal Acts might Not be what You Think they are

• Johannes C. Mayer 2 Nov 2023 5:22 UTC

  1 point

  in reply to: jimrandomh’s comment on: Open Thread, Au­gust 2010-- part 2

  It seems that Benfotiamine (none of the other tiamines are over the counter in Germany) had a similar effect on me. I feel a lot better now, whereas before I would feel constantly tired. Before I felt like I could not do anything most of the time without taking stimulants. Now my default is probably more than 50% towards what I felt like on a medium stimulant dose. I did try a lot of interventions in the same time period, so I am not sure how much Benfotiamin contributed on its own, but I expect it to contribute between 25-65% of the positive effects. I also figured out that I am borderline diabetic, which is evidence in favor of Benfotiamine being very significant.