awenonian

Karma: 116

awenonian 27 Sep 2025 21:18 UTC
6 points
2
on: Christian homeschoolers in the year 3000
I think I agree with a vibe I see in the comments that an AI that causes this problem is perhaps threading a very small needle.
Yudkowsky wrote The Hidden Complexity of Wishes to explain that a genie that does what you say will almost certainly cause problems. If people have this kind of Superintelligence, it won’t take long before someone asks it to get them as many paperclips as possible and we all die. The kind of AI that does what humans want without killing everyone is one that does what we mean.
But how does this work? If you ask such a superintelligence to pull your kid from the rubble of a collapsed building, does it tell you no, because disturbing the rubble could cause it to collapse further and injure your kid more? That you have to wait for better equipment? If not, it probably causes paperclipping problems. If so, it knows when to not do the things you ask, because they won’t accomplish what you “really want”. This is necessarily paternalist.
Would such an AI still listen when people ask it to isolate themselves or others like this? I’m having trouble thinking of one that thinks that being manipulated into a certain set of beliefs is what is best, but is still “aligned” in a way that doesn’t kill everyone.
Admittedly, I shear pretty close to Yudkowsky on doomerism, so that may be the crux. That I don’t see much space between “we all die” and “robustly solved alignment ushers in techno-utopia” (given superintelligence). So arbitrarily targettable hyper-manipulative AI that don’t cause either “AI takeover” or “massive swings in human power” just don’t seem like a real through-line.
(Like, if someone asks their AI to convince everyone else that they are the king of the world. Does it do that? Does it succeed? Do any protections against “massive swings in human power” prevent this? Do passive AI protections everyone has know to defend against this? Do they not apply to convincing people that “Jesus is your Lord”? Does human civilization end as soon as some drunk person says “Hey GPT go convince everyone you’re king of the world” or something? If I make a bubble and you tell an AI “go tell everyone in that bubble the truth,” what happens? Does an AI war break out? Does it not hurt anyone somehow?)

awenonian 26 Aug 2025 0:49 UTC
1 point
0
in reply to: DaystarEld’s comment on: Emotions Make Sense
To some degree. And I agree on most emotions, they exist for a reason and someone who discounts them without reflection is making a mistake. But I think Envy, on reflection, still strikes me as something better for the goals of evolution and in the environment of our ancestors than one that “makes sense” for us and in the modern world.
I think, insofar as Envy drives people to steal, it decreases people’s likelihood to survive and thrive (jail isn’t the optimal place for either of those, and if you’re stealing from Envy, not desperation or something, it probably wasn’t worth the risk). Cheating, another behavior driven by Envy, can lead to suffering violence at the hands of the spurned party (tho if you have “has more sex than otherwise” as a non-trivial term in “thrive” then possibly this one is a wash).
To me, Envy seems to be the drive to defect against a cooperator in some cases, which is, let’s call it “effective” (to differentiate “good/nice”) to take advantage of when you can. But it’s calibrated for a situation where there are tribal levels of coalition with the cooperators, and now there are societal levels of coalition with the cooperators, so this is a much worse value proposition.
It “makes sense” that it evolved the way it did. And of course, if it didn’t, it wouldn’t have evolved that way. But that doesn’t mean it must continue to “make sense” and I’m not sure it does.

awenonian 18 Aug 2025 22:34 UTC
1 point
0
in reply to: DaystarEld’s comment on: Emotions Make Sense
Sure, but if the positives of Envy can be better obtained by less destructive emotions, does it really serve a purpose?
I mean, if Inside Out 2 was about the exploration of Envy being gone, and the best it could do is “Riley is still motivated, but just, a bit less than she would be otherwise” it wouldn’t really make for a dramatic change.
And if I were building a person, and had the option to give them Envy, instead of upping their Admiration or Pleasure (as you name), I don’t know why I would.
(Edit: that may be overly unfair. I can come up with reasons why. But they feel anti-altruistic. For example, Envy could push you to steal from people who have things you don’t. This is good for you, because then you’ll have more stuff. But in the same way society and the people in it are better off if no one steals, they’d also be better off if no one felt Envy. So I am still struggling to come up with reasons why I should endorse Envy.)

awenonian 13 Aug 2025 18:52 UTC
0 points
0
on: Emotions Make Sense
So I picked Envy for the exercise (though meant Jealousy in your wording), and came up with basically the example you gave in your section on it, but dismissed it, since it felt like a justification, not a reason. That is, it felt like what I would say if I needed to argue that Jealousy was useful, not an actual reason why Jealousy is useful.
Like, I don’t imagine Jealousy is our only source of motivation. We would (but maybe only could?) want to do better or have more, without seeing someone else have more. The people in Tribe A would want more fish because they’d feel Happy if they had more, or Sad if they didn’t have enough, or Angry if they went to long being denied fish. Presumably, Tribe C came up with better fishing for those reasons, since they didn’t have anyone to be Jealous of. Tribe A, now seeing that better fishing is possible, not doing it because they don’t feel Jealousy, would be like Tribe C not doing it after they came up with it, because there was no one to be Jealous of.
I think the evolutionary story of “why Jealousy” is simpler as just “because evolution cares about relative genetic fitness, and if someone is doing better than you, that’s bad relatively, even if it isn’t bad absolutely”. (And so, unless you do care about relative genetic fitness, you should probably be very skeptical of Jealousy)
And in general, I would find it very surprising to learn that an emotional landscape optimized for hunting and gathering in the savanna is also optimized for working a 9 to 5 desk job. At the same time, I would be very surprised to learn that those two situations were so disparate that none of the emotional landscape translates. (So they’re like an unreliable friend. Possibly helpful, but possibly untrustworthy)
But I think this is where I put your Envy example. Yeah, your tribe outcompeting and then destroying mine would be bad. But if I feel Envious of my richer friend, it’s not because I think he’ll use his richer-ness to destroy me. And if America is less productive and ends up losing a war to China, it’d be hard to explain through insufficient Envy, on account of how little anyone feels Envious of Chinese people they’ve never met and know nothing of the life circumstances of. I doubt “Envy” is the driving force of military strategy in the modern age.

awenonian 6 May 2025 13:16 UTC
1 point
0
in reply to: kilgoar’s comment on: Playing in the Creek
I interpreted the thrust of the essay as something else. But more broadly, I get why you’d find intuition pump essays manipulative, but not why you’d find them unconvincing.
The point is to take someone who says “I don’t like any dish with onions”, and give them a dish that has onions in it. They’d normally refuse to eat that, so you don’t tell them, and then when they like it you can show them that their claim is wrong.
(I will note that I think “writing an essay with a surprising conclusion” is a significantly smaller violation than “feed someone food they explicitly asked not to be fed”)
I think this essay is aimed at people who think things like “Better AI is always good, because it will do things better, and that’s good.” If those people were not saying “No, you should’ve kept making better bows until you put an arrow through your friend’s chest” then maybe they don’t actually think “better” is always good. (Basically the Orthogonality Thesis)

awenonian 14 Jan 2025 14:07 UTC
3 points
0
in reply to: lsusr’s comment on: Stream Entry
Thank you for the reply. I think I’ll need to look into things more.
One clarification I wanted to make. I wouldn’t have normally said that I need to put effort into listening. I think I generally feel like it doesn’t take effort. But somewhat recently I had an interaction with someone go poorly. It was a date, and they said afterwards that they didn’t feel like I wanted to get to know them, because I hadn’t asked enough things about them^[1].
So I figure there’s something lacking in my interaction with people. Something that I’m not doing that I should, and doing things requires effort, so therefore I’m lacking in effort in some way.
I think you’re saying in some state it won’t be effortful, because it will be the path of least resistance. But I think my problem was not knowing what they wanted, and I don’t see how a state of mind would give me that information “for free”.
1. ^
  Possibly this was just idiosyncracy on their part. Other people have not expressed this feeling to me, so I have considered I’m putting too much weight on a single interaction.

awenonian 13 Jan 2025 14:05 UTC
3 points
0
on: Stream Entry
Then the wave crested.

Is there any more you can say about this?
People already tell me I have good vibes, and feel like I listen to them^[1]. I give off an air of nonchalance, because I think I kill^[2] my emotions in public.
But I do it because it’s one more thing to deal with that I generally don’t have energy for. I’m already bothered by all the light and sound.
It seems like you’re saying “I meditated, and while at first that made sensory issues worse, eventually they just stopped.” And I’d like to know why, if that was legible to you.
Would appreciate any advice. I can probably say more about my experiences, but I’m not sure what more, if any, would be helpful.
1. ^
  I don’t know if it’s the same as you. I think I’m maybe not as active a listener? I don’t think I’m incapable of the way I see you listen in your videos, but I’m not sure I put the energy into it.
2. ^
  I use a visceral word because I don’t approve of this practice in myself. Perhaps “ignore” or “block” would be more accurate?

awenonian 24 Dec 2024 17:38 UTC
13 points
5
on: When Is Insurance Worth It?
Whether or not to get insurance should have nothing to do with what makes one sleep – again, it is a mathematical decision with a correct answer.
Don’t be overly naive consequentialist about this. “Nothing” is an overstatement.
Peace of mind can absolutely be one of the things you are purchasing with an insurance contract. If your Kelly calculation says that motorcycle insurance is worth $899 a month, and costs $900 a month, but you’ll spend time worrying about not being insured if you don’t buy it, and won’t if you do, I fully expect that is worth more than $1 a month.
But do be actual consequentialist about it. If the value of the insurance is more like $10, but the cost is $900, I doubt peace of mind about this one thing is worth $890 a month.

awenonian 6 Jul 2023 22:08 UTC
1 point
0
on: When do “brains beat brawn” in Chess? An experiment
I have a habit of reading footnotes as soon as they are linked, and your footnote says that you won with queen odds before the call to guess what odds you’d win at, creating a minor spoiler.

awenonian 7 Feb 2023 3:31 UTC
1 point
0
on: Introduction to abstract entropy
Is it important that negentropy be the result of subtracting from the maximum entropy? It seemed a sensible choice, up until it introduces infinities, and made every state’s negentropy infinite. (And also that, if you subtract from 0, then two identical states should have the same negentropy, even in different systems. Unsure if that’s useful, or harmful).
Though perhaps that’s important for the noting that reducing an infinite system to a finite macrostate is an infinite reduction? I’m not sure if I understand how (or perhaps when?) that’s more useful than having it be defined as subtracted from 0, such that finite macrostates have finite negentropy, and infinite macrostates have -infinite negentropy (showing that you really haven’t reduced it at all, which, as far as I understand with infinities, you haven’t, by definition).

awenonian 18 Jan 2023 4:12 UTC
3 points
0
on: Models Don’t “Get Reward”
Back in Reward is not the optimization target, I wrote a comment, which received a (small I guess) amount of disagreement.
I intended the important part of that comment to be the link to Adaptation-Executers, not Fitness-Maximizers. (And more precisely the concept named in that title, and less about things like superstimuli that are mentioned in the article) But the disagreement is making me wonder if I’ve misunderstood both of these posts more than I thought. Is there not actually much relation between those concepts?
There was, obviously, other content to the comment, and that could be the source of disagreement. But I only have that there was disagreement to go on, and I think it would be bad for my understanding of the issue to assume that’s where the disagreement was, if it wasn’t.

awenonian 16 Jan 2023 1:14 UTC
2 points
0
on: We don’t trade with ants
When I tried to answer why we don’t trade with ants myself, communication was one of the first things (I can’t remember what was actually first) I considered. But I worry it may be more analogous to AI than argued here.
We sort of can communicate with ants. We know to some degree what makes them tick, it’s just we mostly use that communication to lie to them and tell them this poison is actually really tasty. The issue may be less that communication is impossible, and more that it’s too costly to figure out, and so no one tries to become Antman even if they could cut their janatorial costs by a factor of 7.
The next thought I had was that, if I were to try to get ants to clean my room, I think it’s likely that the easiest line towards that is not figuring out how to communicate, but breeding some ants with different behavior (e.g. search for small bits of food, instead of large bits. This seems harder than that sentence suggests, but still probably easier than learning to speak ant). I don’t like what that would be analogous to in human-AI interactions.
I think it’s possible that an AI could fruitfully trade with humans. While it lacks a body, posting an ad on Craigslist to get someone to move something heavy is probably easier than figuring out how to hijack a wifi-enabled crane or something.
But I don’t know how quickly that changes. If the AI is trying to build a sci-fi gadget, it’s possible that an instruction set to build it is long or complicated enough that a human has trouble following it accurately. The costs of writing intuitive instructions, and also designing the gadget such that idiot-proof construction is possible could be high enough that it’s better to do it itself.

Things roll downhill

awenonian6 Dec 2022 15:27 UTC

19 points

0 comments1 min readLW link

awenonian 26 Jul 2022 22:07 UTC
3 points
−5
in reply to: paulfchristiano’s comment on: Reward is not the optimization target
I interpret OP (though this is colored by the fact that I was thinking this before I read this) as saying Adaptation-Executers, not Fitness-Maximizers, but about ML. At which point you can open the reference category to all organisms.
Gradient descent isn’t really different from what evolution does. It’s just a bit faster, and takes a slightly more direct line. Importantly, it’s not more capable of avoiding local maxima (per se, at least).
What links here?
- awenonian's comment on Models Don’t “Get Reward” by Sam Ringer (18 Jan 2023 4:12 UTC; 3 points)

awenonian 4 Jul 2022 19:31 UTC
3 points
0
in reply to: anonymousaisafety’s comment on: I No Longer Believe Intelligence to be “Magical”
So, I want to note a few things. The original Eliezer post was intended to argue against this line of reasoning:
I occasionally run into people who say something like, “There’s a theoretical limit on how much you can deduce about the outside world, given a finite amount of sensory data.”
He didn’t worry about compute, because that’s not a barrier on the theoretical limit. And in his story, the entire human civilization had decades to work on this problem.
But you’re right, in a practical world, compute is important.
I feel like you’re trying to make this take as much compute as possible.
Since you talked about headers, I feel like I need to reiterate that, when we are talking to a neural network, we do not add the extra data. The goal is to communicate with the neural network, so we intentionally put it in easier to understand formats.
In the practical cases for this to come up (e.g. a nascent superintelligence figuring out physics faster than we expect), we probably will also be inputting data in an easy to understand format.
Similarly, I expect you don’t need to check every possible esoteric format. The likelihood of the image using an encoding like 61 bits per pixel, with 2 for red, 54 for green and 5 for blue is just, very low, a priori. I do admit I’m not sure if only using “reasonable” formats would cut down the possibilities into the computable realm (obviously depends on definitions of reasonable, though part of me feels like you could (with a lot of work) actually have an objective likeliness score of various encodings). But certainly it’s a lot harder to say that it isn’t than just saying “f(x) = (63 pick x), grows very fast.”
Though, since I don’t have a good sense for whether “reasonable” ones would be a more computable number, I should update in your direction. (I tried to look into something sort of analogous, and the most common 200 passwords cover a little over 4% of all used passwords, which, isn’t large enough for me to feel comfortable expecting that the most “likely” 1,000 formats would cover a significant quantity of the probability space, or anything.)
(Also potentially important. Modern neural nets don’t really receive things as a string of bits, but instead as a string of numbers, nicely split up into separate nodes. (yes, those numbers are made of bits, but they’re floating point numbers, and the way neural nets interact with them is through all the floating point operations, so I don’t think the neural net actually touches the bit representation of the number in any meaningful way.)

awenonian 30 Jun 2022 16:57 UTC
3 points
0
in reply to: anonymousaisafety’s comment on: I No Longer Believe Intelligence to be “Magical”
“you’re jumping to the conclusion that you can reliably differentiate between...”
I think you absolutely can, and the idea was already described earlier.
You pay attention to regularities in the data. In most non-random images, pixels near to each other are similar. In an MxN image, the pixel below is a[i+M], whereas in an NxM image, it’s a[i+N]. If, across the whole image, the difference between a[i+M] is less than the difference between a[i+N], it’s more likely an MxN image. I expect you could find the resolution by searching all possible resolutions from 1x<length> to <length>x1, and finding which minimizes average distance of “adjacent” pixels.
Similarly (though you’d likely do this first), you can tell the difference between RGB and RGBA. If you have (255, 0, 0, 255, 0, 0, 255, 0, 0, 255, 0, 0), this is probably 4 red pixels in RGB, and not a fully opaque red pixel, followed by a fully transparent green pixel, followed by a fully transparent blue pixel in RGBA. It could be 2 pixels that are mostly red and slightly green in 16 bit RGB, though. Not sure how you could piece that out.
Aliens would probably do a different encoding. We don’t know what the rods and cones in their eye-equivalents are, and maybe they respond to different colors. Maybe it’s not Red Green Blue, but instead Purple Chartreuse Infrared. I’m not sure this matters. It just means your eggplants look red.
I think, even if it had 5 (or 6, or 20) channels, this regularity would be born out, between bit i and bit i+5 being less than bit i and i+1, 2, 3, or 4.
Now, there’s still a lot that that doesn’t get you yet. But given that there are ways to figure out those, I kinda think I should have decent expectations there’s ways to figure out other things, too, even if I don’t know them.
I do also think it’s important to zoom out to the original point. Eliezer posed this as an idea about AGI. We currently sometimes feed images to our AIs, and when we do, we feed them as raw RGB data, not encoded, because we know that would make it harder for the AI to figure out. I think it would be very weird, if we were trying to train an AI, to send it compressed video, and much more likely that we do, in fact, send it raw RGB values frame by frame.
I will also say that the original claim (by Eliezer, not the top of this thread), was not physics from one frame, but physics from like, 3 frames, so you get motion, and acceleration. 4 frames gets you to third derivatives, which, in our world, don’t matter that much. Having multiple frames also aids in ideas like the 3d → 2d projection, since motion and occlusion are hints at that.
And I think the whole question is “does this image look reasonable”, which you’re right, is not a rigorous mathematical concept. But “‘looks reasonable’ is not a rigorous concept” doesn’t get followed by “and therefore is impossible” Above are some of the mathematical descriptions of what ‘reasonable’ means in certain contexts. Rendering a 100x75 image as 75x100 will not “look reasonable”. But it’s not beyond thinking and math to determine what you mean by that.

awenonian 26 Jun 2022 14:56 UTC
2 points
0
on: Worked Examples of Shapley Values
“the addition of an unemployable worker causes … the worker’s Shapley values to drop to $208.33 (from $250).”
I would emphasize here that the “workers’” includes the unemployed one. It was not obvious to me, until about halfway through the next paragraph, and I think the next paragraph would read better with that in mind from the start.

awenonian 19 Jun 2022 3:58 UTC
2 points
0
in reply to: LVSN’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
I’d be interested to know why you think that.
I’d be further interested if you would endorse the statement that your proposed plan would fully bridge that gap.
And if you wouldn’t, I’d ask if that helps illustrate the issue.

awenonian 17 Jun 2022 20:47 UTC
1 point
0
on: Contra EY: Can AGI destroy us without trial & error?
It seems odd to suggest that the AI wouldn’t kill us because it needs our supply chain. If I had the choice between “Be shut down because I’m misaligned” (or “Be reprogrammed to be aligned” if not corrigible) and “Have to reconstruct the economy from the remnants of human civilization,” I think I’m more likely to achieve my goals by trying to reconstruct the economy.
So if your argument was meant to say “We’ll have time to do alignment while the AI is still reliant on the human supply chain,” then I don’t think it works. A functional AGI would rather destroy the supply chain and probably fail at its goals, than be realigned and definitely fail at its goals.
(Also, I feel this is mostly a minor thing, but I don’t really understand your reference class on novel technologies. Why is the time measured from “proof of concept submarine” to “capable of sinking a warship”? Or from “theory of relativity” to “Atom Bomb being dropped”? Maybe that was just the data available, but why isn’t it “Wright brothers start attempting heavier than air flight” to “Wright brothers do heavier than air flight”? Because when reading my mind immediately wondered how much of the 36 year gap on mRNA vaccines was from “here’s a cool idea” to “here’s a use case”, instead of “here’s a cool idea” to “we can actually do that”)

awenonian 17 Jun 2022 3:31 UTC
1 point
0
in reply to: LVSN’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
Surely creating the full concrete details of the strategy is not much different from “putting forth as-good-as-human definitions, finding objections for them, and then improving the definition based on considered objections.” I at least don’t see why the same mechanism couldn’t be used here (i.e. apply this definition iteration to the word “good”, and then have the AI do that, and apply it to “bad” and have the AI avoid that). If you see it as a different thing, can you explain why?