AI Safety person currently working on multi-agent coordination problems.
Jonas Hallgren
Karma: 1,142
I really appreciated this post.
I didn’t know that you had concepts for aliveness and boggling within the rationality sphere as I find these two of my most previous states that I’ve been cultivating over the last couple of years and they’ve always felt semi-orthogonal to more classic rationality (which I associate more with the betting, TDT and deep empiricism stuff).
Meditation seems to bring aliveness, boggling and focusing forth quite well and I just really appreciate that they’re things you place high value on as I find them some of the best ways of getting out of pre-existent frames. (Which for me seems one of the best ways of becoming more rational)
On character alignment for LLMs.
I would like to propose that we think of a John Rawls style original position (https://en.wikipedia.org/wiki/Original_position) as one view when looking at character prompting for LLMs. More specifically I would want you to imagine that you’re on a social network or similar and that you’re put into a word with mixtures of AI and human systems, how do you program the AI in order to make the situation optimal? You’re a random person among all of the people, this means that some AIs are aligned to you some are not. Most likely, the majority of AIs will be run by larger corporations since the amount of AIs will be proportional to the power you have.
How would you prompt each LLM agent? What are their important characteristics? What happens if they’re thought of as “tool-aligned”?
If we’re getting more internet based over time and AI systems are more human in that they can flawlessly pass the turing test, I think the veil of ignorance style thinking becomes more and more applicable.
Think more of how you would design a societly of LLMs and what if the entire society of LLMs had this alignment rather than just the individual LLM.
This is a nice way to get around the problems raised in Andrew Critch’s post on consciousness as well since it is a lot less conflationary
Adults will pre-mortem plan by thinking most of their plans will fail. They will therefore they have a dozen layers of backups and action plans prepared in advance. This is also so that other people can feel that they’re safe because someone had already planned this. (“Yeah, I knew this would happen” type of vibe.)
The question would then be “How will my first 10 layers of plans go wrong and how can I take this into account?”
A quick example of this might look like this:
Race dynamics will happen, therefore we need controls and coordination (90%)
Coordination will break down therefore we need better systems (90%)
We will find these better systems yet they will not be implemented by default (85%)
We will therefore have to develop new coordination systems yet they can’t be too big since they won’t be implemented (80%)
We should therefore work on developing proven coordination systems where they empirically work and then scale them
(Other plan maybe around tracking compute or other governance measures)
And I’ve now coincidentally arrived at the same place as Audrey Tang...
But we need more layers than this because systems will fail in various more ways as well!
(Warning: relatively hot take and also not medical advice.)
Firstly, there’s a major underlying confounder effect here which is the untracked severity of insomnia and it’s correlation with the prescription of melatonin. If these are majorly coupled it could amount to most of the effect?
Secondly, here’s a model and a tip for melatonin use as most US over the top pills I’ve seen are around 10mg which is way too much. I’ll start with the basic mental model for why and then say the underlying thing we see.
TL;DR:
If you want to be on the safe side don’t take more than 3mg of it per night. (You’re probably gonna be fine anyway due to the confounder effects of long-term insomnia having higher correlation with long-term melatonin use but who knows how that trend actually looks like.)
Model:
There’s a model for sleep which I quite like from Uri Alon (I think it’s from him at least) and it is mainly as the circadian rhythm and sleep mechanism as a base layer signal for your other bodily systems to sync to.
The reasoning goes a bit like: Which is the most stable cycle that we could stabilise to? Well we have a day rhythm that is coupled to 24 hours a day each day, very stable compared to most other signals. That’s the circadian rhythm which is maintained by your system’s sleep.
What sleep does is that it is a reset period for the biological version of this as it sends out a bunch of low-range stable signals that are saying “hey gather around let’s coordinate, the time is approximately night time.” These brain signals don’t happen in non-sleep and so they are easy to check.
Melatonin is one of the main molecules for regulating this pattern and you actually don’t need more than 0.3 mg (remember bioavaliability here) to double your existing amount that you already have in the body. Most over the counter medicine is around 10mg which is way too much. Imagine that you change one of the baseline signals of your base regulatory system and just bloop it the fuck out of the stratosphere for a concentrated period once everyday. The half life of melatonin is also something like 25-50 minutes so it decays pretty quickly as well which means that the curve ends up looking like the following:
If you don’t do this then your more natural curve looks something more like this:
Section 3 of the following talk about a desentisation of the MT2 part of melatonin as something that happens quite quickly: https://www.sciencedirect.com/topics/neuroscience/melatonin-receptor
So if you stratosphere your melatonin with 10 mg then your REM sleep will be fine but the sensitivity to your MT2 receptor will be a bit fucked which means worse deep sleep (which is important for your cardiovascular system!). Hence you will fall asleep but with worse deep sleep (Haven’t checked if this is true but this could probably be checked pretty easily experimentally).
The bioavaliability of melatonin varies between 10 and 30% so if you aim for approximately your own intragenous generation of melatonin you should take 10 to 3x the amount existent in the system. For my own optimal sleep that is 0.5 mg of melatonin but that’s because my own system already works pretty well and I just need a smaller nudge. The area under the curve part of the model is also a good reason to take slow release melatonin as it better approximates your normal melatonin curve.
(I need to get back to work lol but hopefully this helps a bit)
Also, what an amazing post. You’ve expressed something that I’ve wanted to express for a while on a level of depth that I wouldn’t have been able to do and I got literal chills when reading the last part. Well done.
Theoretical physics could produce a recipe for ruin if it turns out that there’s a way to crash the universe through something like a buffer overflow. I’ve seen enough arbitrary code execution glitches to intuitively understand that there’s nothing in principle that says we can’t find a way to bug out some underlying level of abstraction in our physics and break everything. Interestingly enough a bug of this type would complicate interstellar expansion because it would mean that the potential value drift as you exit communication range with other parts of your civilization could lead to a series of events that destroy everything for everyone. Knowledge of such a bug could therefore be one explanation for the Fermi Paradox.
On the 4D Chess takes section I think if you combine that idea with this idea of an ever evolving cosmology where we have selection for the proliferation of intelligent life then it does make sense that we would also have some unintentional bugs come up: https://www.youtube.com/watch?v=7OCY9ppY34Q&lc=UgxFQvrJXcgA6U1Xvf54AaABAg
(The TL;DR of the video is that our cosmos could quite nicely be explained through evolution through replicators that are black holes basically. It’s quite cool and from this year’s ILIAD.)
I also just want to point out that there should be a base rate here that’s higher context in the beginning since before MATS and similar there weren’t really that many AI Safety training programs.
So the intiial people that you get will automatically be higher context because the sample is taken from people who have already worked on it/learnt about it for a while. This should go down over time due to the higher context individuals being taken in?
(I don’t know how large this effect would be but I would just want to point it out.)
I liked this book too and I just wanted to share a graphic that was implied in the book between guidance and expertise. It’s a pretty obvious idea but for me it was just one of those things you don’t think about. The lower context someone has the more guidance they need and vice versa (the trend is not necessarily linear though):
Okay (if possible), I want you to imagine I’m an AI system or similar and that you can give me resources in the context window that increase the probability of me making progress on problems you care about in the next 5 years. Do you have a reading list or similar for this sort of thing? (It seems hard to specify and so it might be easier to mention what resources can bring the ideas forth. I also recognize that this might be one of those applied knowledge things rather than a set of knowledge things.)
Also, if we take the cryptography lens seriously here, an implication might be that I should learn the existing off the shelf solutions in order to “not invent my own”. I do believe that there is no such thing as being truly agnostic to a meta-philosophy since you’re somehow implicitly projecting your own biases on to the world.
I’m gonna make this personally applicable to myself as that feels more skin in the game and less like a general exercise.
There are a couple of contexts to draw from here:
Traditional philosophy. (I’ve read the following):
A history of western philosophy
(Plato, Aristotles, Hume, Foucault, Spinoza, Russell, John Rawls, Dennett, Chalmers and a bunch of other non continental philosophers)
Eastern philosophy (I’ve read the following):
Buddhism, Daoism, Confucianism (Mainly tibetan buddhism here)
Modern more AI related philosophy (I’ve read the following):
Yudkowsky, Bostrom
(Not at first glance philosophy but): Michael Levin (Diverse Intelligence), Some Category Theory (Composition),
Which one is the one to double down on? How do they relate to learning more about meta ethics? Where am I missing things within my philosophy education?
(I’m not sure this is a productive road to go down but I would love to learn more about how to learn more about this.)
Okay, this is quite interesting. I’ll try to parse this by mentioning a potential instantiation of this and maybe you could let me know if I got it wrong or right/somewhere in between?
The scenario is that I’m trying to figure out what I should do when I wake up in the morning and I’m on a walk. What do I is that I then listen to the world in some way, I try to figure out some good way to take actions. One way to do this is somato sensory experiencing, I listen into my sub-parts. Yet a problem with this is that there’s a egree o fpassivity here. Yes my stomach is saying go away and hide and my shoulders are saying that I’m carrying weight whilst maybe my face is smiling and is curious. This listening has some sort of lack of integration within it? I now know this but that doesn’t mean that my sub-parts have had a good conversation.
We can extend this even further for why is the best basis something like emotions? Why can’t we sense things like a degree of extended cognition within our social circles and with different things that we do?
The practice is then somehow figure out how to listen and do good bargaining and to come up with good solutions for the combined agency that you have, whatever that might be? And the more extended and open you can make that cognition, the better it is?
Yet, you shouldn’t fully identify with your social network or the world, neither should you identify with nothing, you should identify with something in between (non-duality from buddhism?). You should try to find the most (causally?) relevant actor to identify with and this is situation dependent and an art?
So that is the process to engage in and the answer is different for each person? (Let me know if I’m off the mark or if this is kind of what you mean)
n=1 evidence but I thought I would share from a random external perspective who enjoys doing some writing and who’s in the potential audience for a future version (if run) of inkhaven.
For this version, I had no clue how it would be and so I thought it was too high risk to gamble on it being good. Given what I’ve seen of the setup this year I would basically be a guaranteed sign up if it was less than 15-20% of my cash reserve to go next year. Potentially upwards of 30-40% (reference class: currently doing more or less independent work and getting by with money for going to uni from the swedish state).
(For some reason I feel a bit weird about making this comment but I also want to practice sending unfinished comments that are more random feedback as it is often useful for the individuals involved given the assumption that my perspective is somewhat indicative of a larger audience.)
Could you elaborate here?
Is there a specific example of the difference between just somatic sensing and having it being intuitively reflected through your thoughts? I feel like you’re saying something important but I’m not fully sure how it manifests more concretely and I might want to work on this skill.
I do feel like there’s something to be said for a more integrated emotion system not being as somatic but being more implicit in the system, like your thoughts and feelings are more one-pointed which is kind of where my experience has shifted to over time, I don’t know if this is what you mean?
Also, don’t listen to me, listen to this successful person!: https://youtube.com/shorts/QEsc1ObYeFk?si=X-3PicapZqJ16DHg
(Ethos guru argument successfully applied!)
(This take is like literaly a copy paste from Dr.K validated through my own experience.)
Firstly, that is a pretty amazing data gathering exercise and I’m really impressed. From the frame of the data I would completely agree with you that it doesn’t seem to help.
I think my frame here is slightly different and specifically about non-cold approaches?
(I want to acknowledge the lack of skin in the game that this view has created for me, I do not care as much about relationships as I find myself quite peaceful and happy without it.)
It is for repeated interactions more? It’s also something that kind of changes the approach vector a bit? I don’t think I could go through the amount of cold approaches that you have here as I don’t care enough for it?
Let me try to give you a mental model of how I think about it and let me know if it makes sense:
Analogously, I would want to imagine that everytime you have a conversation with someone else you create a space, a room. This room can either be cozy with a bunch of nice cushions, maybe it is quite sterile like an operating hall or if it is a more nerdy relationship it might be filled with whiteboards or whatever, there’s a vibe. Meditation (or more specifically awareness + metta meditation) is a bit like creating an openness for that room? You’re allowing the other person space to place their own things in that room and you can more meet them where they’re at and so the conversations become a lot more natural and enjoyable as a consequence. “Oh, you really really want that specific lamp, I guess it doesn’t matter to me but that’s good to know as I can then place my couch here, instead of where the lamp would be”.
When I’m in a warm, open and concentrated state I’m a lot better at conversations.
Do you have any concrete measurable predictions for what would happen in that case?
What I would track is my personal enjoyment of conversations that I have with people, if I did that sort of meditation I would expect myself to enjoy conversations with others more. (With the caveat of adding some sort of metta practice on top).
More statistically, If we model relationship probability as a markov chain we get something like (first meeting → date → date 2 → dating → relationship) and I think your transition probability from first meeting to date to anything beyond that goes up by quite a lot. I think the problem here is that it is more of a poission distribution so it is a bit difficult to do linear prediction on it? (unless you’re poly?) It’s more like a heuristic optimisation problem where the more warmth you have, the easier it is to have giving conversations with other people?
Also, it seems to me that long-term relationships seem to more naturally mature from activities with longer time horizons where you meet people repeatedly? (I could find some stats on this but the basic intuition here is that one of the main criteria for women wanting a long-term relationship is safety which is hard to build without repeated interactions. An optimisation setup is then to repeatedly show up at the right sort of events such as interesting book clubs, dance, meditation or other dependent on your preferences for the base person who shows up at such an event.)
Fair warning is that there’s some unsolicited armchair psychologist advice below but I want to give a meta comment on the “relationship John arc”.
I find it fun, interesting, and sometimes useful to read through these as an underlying investigation of what is true when it comes to dating. (Starting a year ago or so)
So I used to do this cognitive understanding and analysis of relationships a lot but that all changed when the meditation nation attacked? There was this underlying need for love and recognition through a relationship and this underlying want and need for that to feel whole or similar. It’s just kind of gone away more and more and I just generally feel happier in life as a consequence? It kind of feels like you’re looking to resolve that need through relationships and my brain is like “Why doesn’t he just meditate?”
Given the goal is happiness and well-being from this (which it might not be), are there any specific reasons here why you’re going the relationship route? From my own research, all (not all) the cool people (QRI & happiness researchers) agree that meditation gives you better vibes than the courtship stuff?
Finally a weird claim that I’ll make is that the relationship stuff is a lot easier when I’m in a good place when it comes to meditation as I find it a lot easier to read and understand people from this place. I like to go salsa dancing and I feel a lot more relaxed and playful when doing it compared to when I was “looking” for romance? I just bring a different more secure energy and I just stop worrying and start vibing? I agree with you that people’s signals are extremely unclear but it kind of doesn’t matter from that perspective? (You might also already be doing this but meditation probably can make you do this more.)
Therefore, part of me is like, “man he should really stop thinking and start to just sharpen his awareness and attention based systems and he’s gonna be a lot better off in these skills compared to the current investigation”.
So start meditating for an hour a day for 3 months using the mind illuminated as an experiment (getting some of the cool skills mentioned in Kaj Sotala’s sequence?) and see what happens?
I’m however very much enjoying the series of John applying his intelligence to relationships. So uh, do what you want and have fun!
I really like this direction of work, I think it is quite important to elucidate the connection between power-seeking systems and RL and a more generalised version of variational inference that can be applied to collectives.
It feels a bit like you did what the following post is pointing at in a better and more formal way, I thought it might be interesting to share it (to potentially help with some framings of how to explain it intuitively?): https://www.lesswrong.com/posts/KYxpkoh8ppnPfmuF3/power-seeking-minimising-free-energy
Looking forward to more in this area!
I think it’s a fair suggestion that is adjacent, I do think the mechanisms are different enough that it’s wrong though. Some of what we know of the mechanisms of dreaming and emotional regulation through sleep are gone through here (Dreams, Emotional regulation) and one of the questions there is to what extent yogic sleep is similar to REM sleep.
For your lucid dreaming angle, I would say the main dangerous thing is the inhibition of bodily action that leads to this spiral of anxiety when you can’t move? (Sleep paralysis)
I’m like ~70% (50-90%) certain that this does not occur during yoga nidra and that yoga nidra is a technique that actually helps you if you’ve had these problems before.
I also read this book to get the vibe of it, it doesn’t have the best epistemic rigour but the person writing it has a psychiatry practice specifically focused on yoga nidra and one of the main things that this person claims it helps with is PTSD and sleep related problems. I think it has a specific activation pattern that can be very healing if done correctly, if you’re worried you can probably find a ACT psychologist or similar to do the practice with but I do think it is one of the safer practices you can do.
First and foremost, yes changing your brain in major ways is dangerous and you should be careful with what you’re doing. I do think that there are safe ways of doing it and that it is dependent on the teachings that you follow and I go into more depth below. The main takeaway is basically that doing loving kindness and cultivating positive mindstates is a prerequisite to doing concentration practice if you’re in a bad state of mind.
I’m basically repeating the takes of my teacher who’s a thai forest tradition monk for like 30 years with some degrees in the background. He’s been a bit of a rogue and tried out a bunch of different meditation practices and the thing that he recommends to people who might be at risk for negative experiences is awareness practice and loving kindness practice and I think this makes a lot of mechanistic sense. (Do take this with a grain of salt but this is my current best theory and I’ve not seen anyone go astray with this advice.)
The basic problem of general meditation is that it is focused on concentration. This is so that your mind can stabilise yet higher concentration states can generally lead to an amplification of existing emotions which can be a large problem if you have lots of negative emotions. This isn’t necessarily the case for awareness practices and loving kindness practice as they induce different mental states for you. They’re not about intensifying experience and only letting a small amount through, they’re about expanding and seeing more. (see more on this model here)
So the advice is, if you’re worried about the downside you can most likely safely do things like: Yoga Nidra, Loving Kindness practice or awareness practice as it is unlikely that you will be absorbed into negative states of mind even though you’re coming from a worse state (since it’s not absorption based!). It is generally the most direct path to healing and acceptance (imo) and it is what has helped my mother for example the most as she’s a lot more calm and accepting of her current illnesses. My guess is that you could probably do up towards an hour or two a day of this practice without any problems at all. (especially yoga nidra and loving kindness practices)
A bit more detailed on sub points:
1. On the feasibility of there being different types of practice:
I did some research on this before and there’s a bunch of interesting contemplative neuroscience out on the differences in activation in brain areas for meditation. Different brain areas are activated and this is something that is also mentioned in altered traits which is a pop science book on the science of meditation. (I did a quick report for a course a while back on this here which might have some interesting references in the end, the writing is kinda bad though (report) (here are the links that are most relevant in the underlying papers: paper 1, paper 2)
2. One of the main concerns later on in the practice is “The Dark Night of the Soul”. According to my teacher this is more of a concept within practices that are based on the burmese tradition and through retreats that are focused on “dry” (meaning non-joyful) concentration like goenka vipassyana retreats and daniel ingram’s books. One of the underlying things he has said is that there’s a philosophical divide between the non-dual and theravadan styles of practicing about whether you “die” or whether your experience transforms into something that it already was (returning to the unborn) which can be quite important for changing your frame of self.
Also final recommendation is to do it alongside therapy for example something like ACT as it will then also tie you to reality more and it will allow both western and eastern healing to work on you in tandem!
Hopefully this might help somewhat? The basic idea is just to cultivate joy and acceptance before training your amplification abilities as positive emotions would be what is amplified instead.
Based on a true map of the territory. (I really like this advice a good exploration strat seems similar to the one about taking photographies, it is really just about taking a bunch of them and you’ll learn what works over time.)