Suppose the agent’s utility function is concave, i.e. the agent prefers (50% double wealth, 50% lose everything) over (100% wealth stays the same).
I think you meant to write convex here.
Morpheus
Karma: 1,286
Toy Models of Superposition in the dense regime
Nice work in keeping up your public journal.
There is a lot of variance in decision-making quality that is not well-accounted for by how much information actors have about the problem domain, and how smart they are.
I currently believe that the factor that explains most of this remaining variance is “paranoia”. In-particular the kind of paranoia that becomes more adaptive as your environment gets filled with more competent adversaries. While I am undoubtedly not going to succeed at fully conveying why I believe this, I hope to at least give an introduction into some of the concepts I use to think about it.
I don’t know if this was intended, but up until the end I was reading this post thinking you meant in this paragraph that the variance is explained by people not being paranoid enough or not paranoid in the right way and that is why you explain in this post how to be paranoid properly.
I like this post. What I really wish though was if I was better at explaining this to my friends and family. Has anyone on here ever had any success explaining this to an extent where you feel like the other person is really getting it? Perhaps I should become a truth cleric for a while and see if I can convert literal people on the street.
I like this post. I’ve been thinking for a while that I feel like I am doing pretty well in terms of epistemic rationality, but I have quite some trouble figuring out what I want or what I even endorse on reflection. I noticed with your wizard post that this was not something I would ever have come up with, because I would not have looked for “true names” of the thing I want in fiction.
Below I was brainstorming some examples where I could get more of what I want.
Notice: With my ego-dystonic wants I probbaly have more room for improvement. Perhaps the goal should be to not have ego-dystonic wants? They are the main drivers why I have a hard time with agentic-ness.
With Ego-syntonic wants, I already do this. For example just before reading this post, I was asking myself if there could be a company doing long-read sequencing for consumers like John who are peculiar and want to understand themselves better (soon concluded this would be worse than MetaMed, so then thought about other people who might be interested in long-read sequencing).
My ego-dystonic interests I don’t know that well how to deal with. I remember one of my post-rationalist friends commenting that it seems like I seem to only do things I consider useful. For example I tried to get rid of all my useless hobbies I pursued in the past after they ceased being useful. An ego-dystonic interest that I don’t know how to integrate well in a useful way is competitiveness. I get absolutely addicted to improving and competing on metrics. Number go up! For example, hobbies/games that sucked me deep in the past include: juggling, cubing, chess, dominion, learning all japanese kanji with anki (and just staring at the stats ~5-25% of the time), making predictions on metaculus (trying to not be too tempted to maximize points), the universal paperclips game etc..
I now don’t pursue any of the above, because improving on these doesn’t give me enough improvement in other areas of my life I care about. I also notice unless there is a competitive element where I feel like I have worthy competition, the metrics loose their appeal after some time. Problem with Japanese was also that the only reason to do this particular one was to proove to myself that memorization is not that hard. I recently started using anki more again to remember math and science knowledge, but it doesn’t quite feel as addictive when I have to curate all the cards myself. With the kanji, I had premade cards. I was allowed to just grind through.
With Metaculus I had strong frustration that the thing I was competing on was easily goodharted into something that wasn’t teaching me anything. I enjoyed Manifold because the incentives were in line, but then the new problem was that this was incentivising me to be more distracted than I would like, so I stopped using Manifold much. I absolutely loved the thinking physics question challenge. My main bottleneck here was friends who were capable and motivated enough to compete with. I had thought of starting a local workshop in Melbourne to work together on problems we don’t understand. My thinking there was that the hard step seems to be finding problems that everyone is excited to work on. Now I am thinking the best solution is probably just having some array of challening problems to pick from and then you choose something that everyone finds interesting. Perhaps the first challenge is to come up with lots of cool problems.
Part of me is thinking though, tradeoffs are terrible. Perhaps playing chess, cubing or playing Zelda some of the time and spending some of the other time working on illegible problems despite less outside motivation might be the way to go. Obviously, most of the real value is in places where no one can compete with you sadly. Any place where it’s convenient to compete (online games with elo matching being the prototypical example), is where the least of the value is. Finding creative ways to improve my skills by being motivated by competition might be an exception here though. Like running workshops of the sort Raemon is running.
Hm… writing this took me 90 minutes. Ben claims you can write a reasonably long lesswrong comment in under 30 minutes. I already failed the halfhaven challenge, because I would not be able to think of something neat that felt like a round idea to put in a blogpost. Also writing my blogposts took way too long. I did notice that the 500 word lower limit was holding me back there in not publishing short things (I hated the blogpost drafts where I would have a neat 100 word idea and then expanding them to 500 words felt absolutely impossible and wrong). I do think I often like reading rambly comments. I don’t like reading super rambly comments. I do find it hard to find the balance here (in general I find it hard to write about internal conflicts as they are happening). Here at the end I went back and forth between writing out what I thought was my takeaway from this. I do think internal conflict is a huge part that makes my writing slow.
Yep I use all of those you mentioned in evil mode. Except I rarely use paragraph and sentence level, which in practice I just use ”/” to search the right place at that point. You can go overboard overoptimising here and I’ve certainly done that in the past.
Why do we have pimples/acne? Pimples are kind of a confusing phenomenon. As far as I can tell, a majority of people finds popping pimples compelling while knowing that this is obviously not “good for you” in the sense that you look worse afterwards, and if you just wait they often do in fact go away on their own again without risking an infection. My first thought was coming up with some lazy evolutionary psychology explanation that maybe popping pimples is so compelling because social grooming (Thinking of apes grooming each others backs) is good for you because that is how you get rid of ticks and make friends. Once people have a drive that compells them to groom other peoples skin, it is now advantageous to have pimples to make friends. That would explain pimples on your back, but it would not explain pimples on your face that well (it makes you look bad). After that I asked claude why people have pimples. Claudes main observations were that hunter gatherers have few to no pimples, which lines up with studies I find on google scholar[1]. People are blaming some type of change in diet. Given that it’s related to growth hormones, maybe it is just that people are gettting more calories than most modern hunter gatherers, who tend to be in places where food is not always abundant, so people are both taller and have more acne, because somehow growth and sebum production are coupled for some reason that I don’t understand. I guess my evolutionary explanation is probably wrong for why we have pimples, but it might explain why people enjoy messing with them so much.
- ↩︎
I wish these studies were including pictures. How am I supposed to know whether the way they assess acne between studies is consistent at all?
- ↩︎
Is this type of goodness about memetic bullshit value claims or something else? Funnily enough, when I thought I had an example of this with washing vegetables it was somewhat controversial.
Yes. I think using count modifiers on “j” and “k” in vim is insane. Spamming up and down is definitely less mental load than trying to calculate how many rows down you want to move (I remember one tutorial on vim was even supplying some config to remind to unlearn this “bad mental habit” of spamming j and k). But probably this is one of these instances where different peoples minds work very differently. I almost never use macros, but use regex inside of my editor often.
Obviously the legibility of problems is not only a problem in AI safety. For example if I was working on ageing fulltime, I think most of the alpha would be in figuring out which root cause of aging is in fact correct and thinking of the easiest experiments (like immortal yeast) to make this legible to all the other biologists as fast as possible. I wouldn’t try to come up with a drug at all.
Keeping Ants and Spotting Queens
Fine. I accept that this was overdue. I need to do this today. Thanks for writing this :). My last full digital declutter was in ~April of 2022.
A flip side of this analysis is that the detrimental effects of the aforementioned cognitive distortions might be much higher than is usually supposed or realized, perhaps sometimes causing multi-year/decade delays in important approaches and conclusions, and can’t be overcome by others even with significant IQ advantages over me. This may be a crucial strategic consideration, e.g., implying that the effort to reduce x-risks by genetically enhancing human intelligence may be insufficient without other concomitant efforts to reduce such distortions.
Since I have been working on germline engineering, I have been thinking about the same thing. My intuition is that if I could magically just increase everyone’s IQ by 5 points that would result in a marginally saner world. But creating a few babies with ~160+ IQs doesn’t seem obviously beneficial. Even if 3⁄4 get coordination problems etc. what if the fourth one decides to work at OpenAI or Anthropic on capabilities, because working on capabilities is just so much more exciting. If GWAS for personality were working better I’d be more optimistic about selecting for something like capacity to gather wisdom? With editing, you could also go for cognitively enhanced clones of people we consider wise (Main bottleneck with this option would be PR). Problem is that I am going to disagree with people whom we consider wise. Although perhaps we can all agree, we would not like to clone the CEOs of the AGI labs. Perhaps really good education would also help. Not an expert on education.
How does your purely causal framing escape backward induction? Pure CDT agents defect in the iterated version of the prisoners’ dilemma, too. Since at the last time step you wouldn’t care about your reputation.
In conclusion, if you find yourself freely choosing between options, it’s rational to take a dominating strategy, like two-boxing in Newcomb’s problem, or defecting in the sorted prisoner’s dilemma. However, given the opportunity to actually pre-commit to decisions that get you better outcomes provided your pre-commitment, you should do so.
How do you tell if you are in a “pre-commitment” or in a defecting situation?
This is not the correct level for thinking about decision theory—we don’t think about any of our decisions that way. Decision theory is about determining the output of the specific choice-making procedure “consider all available options and pick the best one in the moment”.
Categorical imperative has been popular for a long while:
Act only according to that maxim whereby you can at the same time will that it should become a universal law.
I don’t think this is incompatible with making the best decision in the moment. You just decide in the moment to go with the more sophisticated version of the categorical imperative, because that seems best? If I didn’t reason like this, I would not vote and I would have a harder time to stick to commitments. I agree thinking about decisions in a way that is not purely greedy is complicated.
Brainstorming 25 Questions I Am Interested In
A more general claim is: if something can predict your action with better than random accuracy no matter how hard you try to prevent them, you don’t have free will over that action. (I’m not addressing the question of whether free will exists in general, only whether a particular action is chosen freely.)
The whole framing here strikes me as confused (although maybe I am confused). The way you are phrasing it already assumes that you are in conflict with someone (“no matter how hard you try to prevent them”). Your setup already assumes away you have free will. Both Cooperate Bot and Defect Bot do in fact not have free will. The whole point of game-theory is that you have agents that can simulate each other to some extent. A more useful game theory bot is a function that takes the function of another agent as input. When you assume that you just exist in one place and time, then you are already assuming a setup where game theory is not useful. If you are predicted by someone else, then you (your computation) is being simulated, you don’t exist in just one place and time, you exist in different places and times (although not all of these versions are the full You). You don’t get to choose where you exist, but in this framing you do get to choose your output.
I like this post a lot. It might explain why I feel like an expert at addition, but not on addition. I notice when I am struggling with things like this in math, I often start blaming my own intellect instead of trying to understand what is making this hard and if this is perhaps just bad design that is to blame. The second approach seems much more likely to solve the problem. Noticing that word problems are harder seems like a good thing to notice, especially if you want to become an expert at using a particular math tool. For example I don’t think I currently really get exterior products and searching for relevant word problems might be a good way to practice. LLMs might be useful in creating problems I can’t solve (although I found it astonishing a while ago when Sonnet 3.5 was not able to consistently create word problems for applying bayes rule (~50% were just wrong)).