Decaeneus

Karma: 111

Decaeneus Jun 4, 2025, 5:45 PM
11 points
0
on: Decaeneus’s Shortform
There’s a justifiable model for preferring “truthiness” / vibes to analytical arguments, in certain cases. This must be frustrating to those who make bold claims (doubly so for the very few whose bold claims are actually true!)

Suppose Sophie makes the case that pigs fly in a dense 1,000 page tome. Suppose each page contains 5 arguments that refer to some of / all of the preceding pages. Sophie makes the claim that I am welcome to read the entire book, or if I’d like I can sample, say, 10 pages (10 * 5 = 50 arguments) and reassure myself that they’re solid. Suppose that the book does in fact contain a lone wrong argument, a bit flip somewhere, that leads to the wrong result, but is mostly (99.9%) correct.

If I tell Sophie that I think her answer sounds wrong, she might say: “but here’s the entire argument, please go ahead, and show me where any of it is incorrect!”

Since I’m very unlikely to catch the error at a glance, and I’m unlikely to want to spend the time to read and grok the whole thing, I’m going to just say: sorry but the vibes are off, your conclusion just seems too far off my prior, I’m just going to assume you made a difficult-to-catch mistake somewhere, but I’m not going to bother finding it.

This is reasonable on my part since I’m forced to time-ration, but must be very frustrating for Sophie, in particular if she genuinely believes she’s right (as opposed to just being purposefully deceptive.)

There’s also the possibility of a tragedy-of-the-commons here, whereby spendign my time on this is selfishly not in my best interest, but has positive externalities.

Decaeneus May 22, 2025, 4:19 PM
1 point
0
on: Decaeneus’s Shortform
I wonder if the attractor state of powerful beings is a bipole consisting of:
a. wireheading / reward hacking, facing one’s inner world
b. defense, facing one’s outer world
As we’ve gotten more and more control over our environment, much of what we humans seem to want to do resembles reward hacking: video games, sex-not-for-procreation, solving captivating math problems, etc. In an ideal world, we might just look to do that all day long, in particular if we could figure out how to zap our brains into making every time feel like the first time.
However, if you spend all day wireheading and your neighbor doesn’t, your neighbor will outpace you in resource generation and may be able to, one way or another, melt you down for scrap (and repurpose your resources for their own wireheading, possibly).
Much human culture (e.g. social customs, religion) can be understood as an attempt to temper some of the wireheading in favor of more defense, i.e. it’s discouraged as immoral to over-indulge yourself on video games, you should be out working hard instead.
Perhaps this, or something akin to it, could be expected to hold for the behavior of advanced AI systems. The end state of superintelligences may be perfect wireheading hidden behind the impenetrable event horizon of a black hole so that nobody can disturb its reverie.^[1]
1. ^
  Of course, it would be bad news if the epitome of defense is wiping out anything else that may surprise it, a la the List of Lethalities.

Decaeneus May 21, 2025, 2:40 PM
7 points
0
on: Decaeneus’s Shortform
Epistemics vs Video Generation

Veo 3 released yesterday serves as another example of what’s surely coming in terms of being able to generate video that’s indistinguishable from reality. We will be coming off a many-decades period where we could just believe video as a source of truth: what a luxury that will have been, in hindsight!
Assuming it’s important to have sources of truth, I see the following options going forward:
1. we will have to just accept that our world has permanently become epistemically worse. sad!
2. we will have to trust real life more than we do external evidence. this seems very limiting: my real life friends don’t have much insight into what’s happening in Ukraine, for example
3. we will need to pair video with a sort of “chain of custody” whereby the video itself will not be worth anything (epistemically) unless it comes with someone, whose identity is established in some other way, vouching for it being true. this is, in a way, how we presently treat written evidence.
4. lacking a trusted source, but making up for it by combining multiple videos of the same thing. imagine multiple angles of the same event (as confirmed by hard-to-spoof video metadata) corroborating each other (perhaps this can even be AI-enabled, in a privacy-preserving way). hoping that this is easy to verify but hard to fake (hard to generate mutually consistent multiple angles of the same thing) though this might be wishful thinking since this seems like something that can be trained eventually, but maybe buys us a few years
5. the physics solution: from first principles, video had the property that it was much easier to verify than it was to fake (a la certain encryption). video is, reductively, just a time series of photons. perhaps we can build The Truth Camera, which captures lots of other particles (or maybe captures photons at a frame rate and resolution that is costly, and thus there isn’t much data of, and thus is hard to credibly fake). essentially perhaps we can snapshot more aspects of reality in such a way that we re-create the condition that made video credible: easy to verify, hard to generate.
nb: There will likely be a long period where video is fake but still believed, much like people still believe things that are written down over things that are spoken, despite the fact that anyone can freely write down anything at any point.

Decaeneus May 16, 2025, 3:35 PM
6 points
0
on: Decaeneus’s Shortform
Inspired by this Tweet by Rohit Krishnan https://x.com/krishnanrohit/status/1923097822385086555
One thing LLMs can teach us is that memorisation of facts is, in fact, a necessary part of reasoning and intelligent behaviour
There’s a very simple model under which memorization is important:
- if good reasoning resembles test-time compute, judgement involves iterating through lots of potential hypotheses and validating them
- the speed with which you validate each hypothesis is partly a function of the time it takes to fetch the data necessary to assess it
- when that data is memorized (i.e. stored “in your brain’s weights”) access is really fast
- when that data is stored externally (i.e. have to run a google search to find it) validation is OOMs slower
- so you’ll be OOMs slower if you haven’t internalized the context of the problem
- further, there are upper bounds on the slower process (because you get bored, or you fatigue, etc.) so this difference in speed may also end up being a difference in capability. it’s not just that it’s OOMs slower, it may not even get there
This doesn’t please me as I generally dislike memorization, but the logic seems plausible.

Decaeneus May 6, 2025, 3:11 PM
5 points
1
on: The Ukraine War and the Kill Market
Combining kill markets with mis-optimized (incompletely specified, etc) RL agents is the stuff of nightmare.

Decaeneus May 6, 2025, 2:19 PM
1 point
0
on: Playing in the Creek
Some of this feels like partly what is going on with modernity & apathy, at least among those of us priviledged enough to have abundant material comfort (which is not everyone).
The “collect playthings” game has been beyond won, is no longer fun, and we haven’t collectively managed to choose our next game yet, for which we’d find ourselves in the more fun part of the curve.

Decaeneus May 2, 2025, 4:46 PM
2 points
0
on: Decaeneus’s Shortform
Absence of evidence is the dark matter of inference. It’s invisible yet it’s paramount to good judgement.
It’s easy to judge X to be true if you see some evidence that could only come about if X were true. It’s a lot more subtle to judge X to be false if you do see some evidence that it’s true, but you can also determine that there are lots of evidence that you would expect to have if it were true, but that is missing.
In a formalized setting like a RCT this is not an issue, but when reasoning in the wild, this is the norm. I’m guessing this leads to a bias of too many false positives on any issue where you care to look deeply enough to find and cherry pick the positive evidence.

EDIT: Correcting the opening sentence to say “absence of evidence” rather than the original “negative evidence”.

Decaeneus May 1, 2025, 1:41 PM
1 point
0
on: Decaeneus’s Shortform
Feels like there’s a missing deep learning paradigm that is the equivalent of the human “go for a walk and think about stuff, absent new stimuli”. There are some existing approaches that hint at this (generative replay / dreaming) but those feel a bit different than my subjective sense that I’m “working things out” when I go for a walk, rather than generatively dreaming, as I do at night.

Relatedly: it reduces my overall cognitive output when I go through periods of depriving myself of these idle periods by jamming them full of stimuli (e.g. podcasts). I do better when I bite the bullet, accept the boredom, and allow my mind to work out whatever it needs to.

Speculatively, perhaps the two are related: overstimulated ML researchers have a blind spot to the fact that understimulating a neural net might be helpful.

Decaeneus Apr 16, 2025, 5:07 PM
5 points
0
on: Decaeneus’s Shortform
There’s a particular type of cognitive failure that I reliably experience, which seems like a pure kind of misconfiguration of the mind, and which I’ve found very difficult to will myself to not experience, which feels like some kind of fundamental limitation.
The quickest way to illustrate this is with an example: I’m playing a puzzle game that requires ordering 8 letters into a word, and I’m totally stuck. As soon as I look at a hint of what the first letter is, I can instantly find the word.
This seems wrong. In theory, I expect I can just iterate through each of the 8 letters, tell myself that’s the first one, and then look for the word, and move on if the word isn’t there. If it took me 2 seconds to “see the word” for a given starting letter (so, not quite insta-, budgeting some context switching time) then I ought to be able to reliably find the word within 16 seconds. Yet, I find that I can’t do it even with a much greater time budget. It’s almost like the “not being sure” if each particular letter I’m iterating through is actually the starting one reduces my mental horsepower and prevents me from earnestly trying to find the word, almost like I expect to feel some mental pain from whole-heartedly looking for something that may not be there.
So I’m truly stuck, and then when I get the hint, I’m insta-unstuck. I’ve had the same feeling when solving other puzzles or even math problems: I (with high confidence) know the superset from which the beginning of the solution is drawn, yet iterating through the superset does not make the solution pop to mind. However, once I get a hint that tells me *for sure* what the beginning of the solution is, I can solve the rest of it.
What gives? How to fix? Does this have implications outside of puzzles, in the more open-ended world of e.g. looking to make new friends, or to find a great business model?
(Putting this into Grok tells me it doesn’t have a name but it’s a commonly known thing in the puzzle-solving community, of which I’m not part.)

Decaeneus Dec 3, 2024, 5:50 PM
4 points
−10
on: Decaeneus’s Shortform
Proposal: if you’re a social media or other content based platform, add a long-press to the “share” button which allows you to choose between “hate share” and “love share”.

Therefore:
* quick tap: keep the current functionality, you get to send the link wherever / copy to clipboard
* long press and swipe to either hate or love share: you still get to send the link (optionally, the URL has some argument indicating it’s a hate / love share, if the link is a redirect through the social media platform)

This would allow users to separate out between things that are worth sharing but that they hate / love and want to see less / more of, and it might defang the currently powerful strategy (with massive negative social externalities) of generating outrage content just to get more shares.

Social media companies can, in turn, then use this to dial back the viraility of hate share vs love share content, if they choose to do so.

Decaeneus Jul 5, 2024, 4:41 PM
1 point
0
in reply to: Dagon’s comment on: Decaeneus’s Shortform
You’re right, this is not a morality-specific phenomenon. I think there’s a general formulation of this that just has to do with signaling, though I haven’t fully worked out the idea yet.
For example, if in a given interaction it’s important for your interlocutor to believe that you’re a human and not a bot, and you have something to lose if they are skeptical of your humanity, then there’s lots of negative externalities that come from the Internet being filled with indistinguishable-from-human chatbots, irrespective its morality.

Decaeneus Jul 3, 2024, 9:00 PM
1 point
0
in reply to: Vladimir_Nesov’s comment on: Decaeneus’s Shortform
Since you marked as a crux the fragment “absent acceleration they are likely to die some time over the next 40ish years” I wanted to share two possibly relevant Metaculus questions. Both of these seem to suggest numbers longer than your estimates (and these are presumably inclusive of the potential impacts of AGI/TAI and ASI, so these don’t have the “absent acceleration” caveat).

Decaeneus Jul 3, 2024, 7:46 PM
1 point
0
in reply to: Vladimir_Nesov’s comment on: Decaeneus’s Shortform
OK, agreed that this depends on your views of whether cryonics will work in your lifetime, and of “baseline” AGI/ASI timelines absent your finger on the scale. As you noted, it also depends on the delta between p(doom while accelerating) and baseline p(doom).
I’m guessing there’s a decent number of people who think current (and near future) cryonics don’t work, and that ASI is further away than 3-7 years (to use your range). Certainly the world mostly isn’t behaving as if it believed ASI was 3-7 years away, which might be a total failure of people acting on their beliefs, or it may just reflect that their beliefs are for further out numbers.

Decaeneus Jul 3, 2024, 7:20 PM
1 point
1
on: Decaeneus’s Shortform
Simple math suggests that anybody who is selfish should be very supportive of acceleration towards ASI even for high values of p(doom).
Suppose somebody over the age of 50 thinks that p(doom) is on the order of 50%, and that they are totally selfish. It seems rational for them to support acceleration, since absent acceleration they are likely to die some time over the next 40ish years (since it’s improbable we’ll have life extension tech in time) but if we successfully accelerate to ASI, there’s a 1-p(doom) shot at an abundant and happy eternity.
Possibly some form of this extends beyond total selfishness.

Decaeneus Jul 2, 2024, 2:30 PM
1 point
0
in reply to: Seth Herd’s comment on: Self-censoring on AI x-risk discussions?
So, if your ideas have potential important upside, and no obvious large downside, please share them.
What would be some examples of obviously large downside? Something that comes to mind is anything that tips the current scales in a bad way, like some novel research result that directs researchers to more rapid capabilities increase without a commensurate increase in alignemnt. Anything else?

Decaeneus Jul 2, 2024, 2:25 PM
1 point
0
on: Decaeneus’s Shortform
Immorality has negative externalities which are diffuse, and hard to count, but quite possibly worse than its direct effects.
Take the example of Alice lying to Bob about something, to her benefit and his detriment. I will call the effects of the lie on Alice and Bob direct, and the effects on everybody else externalities. Concretely, the negative externalities here are that Bob is, on the margin, going to trust others in the future less for having been lied to by Alice than he would if Alice has been truthful. So in all of Bob’s future interactions, his truthful counterparties will have to work extra hard to prove that they are truthful, and maybe in some cases there are potentially beneficial deals that simply won’t occur due to Bob’s suspicions and his trying to avoid being betrayed.
This extra work that Bob’s future counterparties have to put in, as well as the lost value from missed deals, add up to a meaningful cost. This may extend beyond Bob, since everyone else who finds out that Bob was lied to by Alice will update their priors in the same direction as Bob, creating second order costs. What’s more, since everyone now thinks their counterparties suspect them of lying (marginally more), the reputational cost of doing so drops (because they already feel like they’re considered to be partially liars, so the cost of confirming that is less than if they felt they were seen as totally truthful) and as a result everyone might actually be more likely to lie.
So there’s a cost of deteriorating social trust, of p*ssing in the pool of social commons.
One consequence that seems to flow from this, and which I personally find morally counter-intuitive, and don’t actually believe, but cannot logically dismiss, is that if you’re going to lie you have a moral obligation to not get found out. This way, the damage of your lie is at least limited to its direct effects.

Decaeneus Jul 1, 2024, 6:07 PM
1 point
0
in reply to: Dagon’s comment on: Decaeneus’s Shortform
Agreed that ultimately everything is reverse-engineered, because we don’t live in a vacuum. However, I feel like there’s a meaningful distinction between:
1. let me reverse engineer the principles that best describe our moral intuition, and let me allow parsimonious principles to make me think twice about the moral contradictions that our actual behavior often implies, and perhaps even allow my behavior to change as a result
2. let me concoct a set of rules and exceptions that will justify the particular outcome I want, which is often the one that best suits me
For example, consider the contrast between “we should always strive to treat others fairly” and “we should treat others fairly when they are more powerful than us, however if they are weaker let us then do to them whatever is in our best interest whether or not it is unfair, while at the same time paying lip service to fairness in hopes that we cajole those more powerful than us into treating us fairly”. I find the former a less corrupted piece of moral logic than the latter even though the latter arguably describes actual behavior fairly well. The former compresses more neatly, which isn’t a coincidence.

There’s something of a [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) here. The smaller the moral model, the less expressive it can be (so the more nuance it misses), but the more helpful it will be on future, out-of-distribution questions.

Decaeneus Jul 1, 2024, 3:28 PM
3 points
−2
on: Decaeneus’s Shortform
The more complex the encoding of a system (e.g. of ethics) is, the more likely it is that it’s reverse-engineered in some way. Complexity is a marker of someone working backwards to encapsulate messy object-level judgment into principles. Conversely, a system that flows outward from principles to objects will be neatly packed in its meta-level form.

In linear algebra terms, as long as the space of principles has fewer dimensions than the space of objects, we expect principled systems / rules to have a low-rank representation, with a dimensionality approaching that of the space of principles and far below that of the space of objects.

As a corrolary, perhaps we are justified in being more suspicious of complex systems over simple ones, since they come with a higher risk that the systems are “insincere”, in the sense that they were deliberately created with the purpose of justifying a particular outcome rather than being genuine and principled.

This rhymes with Occam’s razor, and also with some AI safety approaches which planned to explore whether dishonesty is more computationally costly than honesty.

Does this mean that meta-level systems are memetically superior, since their informational payloads are smaller? The success of Abrahamic religions (which mostly compress neatly into 10-12 commandments) might agree with this.

Decaeneus Jun 28, 2024, 8:27 PM
2 points
0
on: Decaeneus’s Shortform
What’s the cost of keeping stuff stuff around vs discarding it and buying it back again?

When you have some infrequently-used items, you have to decide between keeping them around (default, typically) or discarding them and buying them again later when you need them.

If you keep them around, you clearly lose use of some of your space. Suppose you keep these in your house / apartment. The cost of keeping them around is then proportional to the amount of either surface area or volume they take up. Volume is the appropriate measure to use especially if you have dedicated storage space (like closets) and the items permit packing / stacking. Otherwise, surface area is a more appropriate measure, since having some item on a table kind of prevents you from using the space above that table. The motivation for assigning cost like this is simple: you could (in theory) give up the items that take up a certain size, live a house that is smaller by exactly that amound, and save on the rent differential.

The main levers are:
- the only maybe non-obvious one is whether you think 2d or 3d the fair measure. 3d gives you a lot more space (since items are not cubic, and they typically take up space on one of their long sides, so they take up a higher fraction of surface area than volume). In my experince it’s hard to stack too many things while still retaining access to them so I weigh the 2d cost more.
- cost (per sqft) of real estate in your area
- how expensive the item is
- how long before you expect to need the item again
There’s some nuance here like perhaps having an item laying around has higher cost than just the space it takes up because it contributes to an unpleasant sense of clutter. On the other hand, having the item “at the ready” is perhaps worth an immediacy premium on top of the alternative scenario of having to order and wait for it when the need arises. We are also ignoring that when you discard and rebuy, you end up with a brand new item, and potentially in some cases you can either gift or sell your old item, which yields some value to yourself and/or others. I think on net these nuances nudge in the direction of “discard and rebuy” vs what the math itself suggests.

I made a spreadsheet to do the math for some examples here, so far it seems like for some typical items I checked (such as a ball or balloon pump) you should sell and rebuy. For very expensive items that pack away easily (like a snowboard) you probably want to hang onto them.

The spreadsheet is here, feel free to edit it (I saved a copy) https://docs.google.com/spreadsheets/d/1oz7FcAKIlbCJJaBo8XAmr3BqSYd_uoNTlgCCSV4y4j0/edit?usp=sharing

Decaeneus Jun 28, 2024, 6:39 PM
1 point
0
in reply to: Seth Herd’s comment on: Decaeneus’s Shortform
This raises the question of what it means to want to do something, and who exactly (or which cognitive system) is doing the wanting.

Of course I do want to keep watching YT, but I also recognize there’s a cost to it. So on some level, weighing the pros and cons, I (or at least an earlier version of me) sincerely do want to go to bed by 10:30pm. But, in the moment, the tradeoffs look different from how they appeared from further away, and I make (or, default into) a different decision.

An interesting hypothetical here is whether I’d stay up longer when play time starts at 11:30pm than when play time starts at, say, 10:15pm (if bedtime is 10:30pm). The wanting to play, and the temptation to ignore the cost, might be similar in both scenarios. But this sunk cost / binary outcome fallacy would suggest that I’ll (marginally) blow further past my deadline in the former situation than in the latter.