There’s something very creepy to me about the part of research consent forms where it says “my participation was entirely voluntary.”
Do they really think an involuntary participant wouldn’t sign that? If they understand that they would, what purpose could this possibly serve, other than, as is commonly the purpose of contracts; absolving themselves of blame and moving blame to the participant? Which would be downright monstrous. Probably they just aren’t fucking consequentialists, but this is all they end up doing.
This is a minor thing, but it adds an additional creepy garnish: Nothing is 100% voluntary, because everything is a function of the involuntary base reality that other people command force and resources and we want to use them for things so we have to go along with what other people want to some extent. I’m at peace with this, and I would prefer not to have to keep denying it, and it feels like I’m being asked to participate in the addling of moral philosophy.
Maybe it’s some legal hack, like maybe in some situations you can’t dismiss unethical research, but you can dismiss fraudulent research… and a research where people were forced to falsely write that their participation was voluntary, is technically fraudulent.
I notice it also makes sure that if the participants know anything at all about the research, they know it’s supposed to be voluntary, even if they’re still forced to sign it, they learn that the law is supposed to be on their side and there is in theory someone they could call for help.
The reason is to prevent the voluntary participant from later claiming that their participation was involuntary and telling that to the IRB.
‘Well if your participation was involuntary, why did you sign this document?’
It kind of limits the arguments someone could make attacking the ethics of the study. The attacker would have to allege coercion on the order of people being forced to lie on forms under threat.
If someone explicitely writes into their consent forms “my participation was entirely voluntary” and the participation isn’t voluntary it might be easier to attack the person running the trial later.
The voluntary consent of the human subject is absolutely essential.
This means that the person involved should have legal capacity to give consent; should be so situated as to be able to exercise free power of choice, without the intervention of any element of force, fraud, deceit, duress, over-reaching, or other ulterior form of constraint or coercion; and should have sufficient knowledge and comprehension of the elements of the subject matter involved, as to enable him to make an understanding and enlightened decision. This latter element requires that, before the acceptance of an affirmative decision by the experimental subject, there should be made known to him the nature, duration, and purpose of the experiment; the method and means by which it is to be conducted; all inconveniences and hazards reasonably to be expected; and the effects upon his health or person, which may possibly come from his participation in the experiment.
The duty and responsibility for ascertaining the quality of the consent rests upon each individual who initiates, directs or engages in the experiment. It is a personal duty and responsibility which may not be delegated to another with impunity
Important to remember and stand by the Nuremberg Code in these contexts.
There’s a lot of “neuralink will make it easier to solve the alignment problem” stuff going around the mainstream internet right now in response to neuralink’s recent demo.
I’m inclined to agree with Eliezer, that seems wrong; either AGI will be aligned in which case it will make its own neuralink and wont need ours, or it will be unaligned and you really wouldn’t want to connect with it. You can’t make horses competitive with cars by giving them exoskeletons.
But, is there much of a reason to push back against this?
Providing humans with cognitive augmentation probably would help to solve the alignment problem, in a bunch of indirect ways.
It doesn’t seem like a dangerous error at all. It feeds a public desire to understand how AGI might work. Neuralink itself is a great project for medical science. Generally, wrong beliefs cause bad consequences, but I’m having difficulty seeing what they’d be here.
The obvious bad consequence is a false sense of security leading people to just get BCIs instead of trying harder to shape (e.g. delay) AI development.
″ You can’t make horses competitive with cars by giving them exoskeletons. ” <-- this reads to me like a separate argument, rather than a restatement of the one that came before.
I agree that BCI seems unlikely to be a good permanent/long-term solution, unless it helps us solve alignment, which I think it could. It could also just defuse a conflict between AIs and humans, leading us to gracefully give up our control over the future light cone instead of fighting a (probably losing) battle to retain it.
...Your post made me think more about my own (and others’) reasons for rejecting Neuralink as a bad idea… I think there’s a sense of “we’re the experts and Elon is a n00b”. This coupled with feeling a bit burned by Elon first starting his own AI safety org and then ditching it for this… overall doesn’t feel great.
I’ve never been mad at elon for not having decision theoretic alignmentism. I wonder, should I be mad. Should I be mad about the fact that he has never talked to eliezer (eliezer said that in passing a year or two ago on twitter) even though he totally could whenever he wanted.
Also, what happened at OpenAI? He appointed some people to solve the alignment problem, I think we can infer that they told him, “you’ve misunderstood something and the approach you’re advocating (proliferate the technology?) wouldn’t really be all that helpful”, and he responded badly to that? They did not reach mutual understanding?
(instutitional reform take, not important due to short timelines, please ignore)
The kinds of people who do whataboutism, stuff like “this is a dangerous distraction because it takes funding away from other initiatives”, tend also to concentrate in low-bandwidth institutions, the legislature, the committee, economies righteously withering, the global discourse of the current thing, the new york times, the ivy league. These institutions recognize no alternatives to them, while, by their nature, they can never grow to the stature required to adequately perform the task assigned to them. I don’t think this is a coincidence, and it makes it much easier for me to sympathize with these people: They actually believe that we can’t deal with more than one thing at a time.
They generally have no hope for decentralized decisionmaking, and when you examine them closely you find that they don’t really seem to believe in democracy, they’ve given up on it, they don’t talk about reforming it, they don’t want third parties, they’ve generally never heard of decentralized public funding mechanisms, certainly not futarchy. So it’s kind of as simple as that. They’re not being willfully ignorant. We just have to show them the alternatives, and properly, we basically haven’t done it yet. The minarchists never offered a solution to negative externalities or public goods provision. There are proposals but the designs are still vague and poorly communicated. There has never been an articulation of enlightened technocracy, which is essentially just succeeding at specialization or parallelization in executive decisionmaking. I’m not sure enlightened technocracy was ever possible until the proposal of futarchy, a mechanism by which non-experts can hold claimed experts accountable.
Theory: Photic Sneezing (the phenotype where a person sneezes when exposed to a bright light, very common) evolved as a hasty adaptation to indoor cooking or indoor fires, clearing the lungs only when the human leaves the polluted environment. The newest adaptations will tend to be the roughest, I’m guessing it arose only in the past 500k years or so as a response to artificial dwellings and fire use.
Considering doing a post about how it is possible the Society for Cryobiology might be wrong about Cryonics, it would have something to do with the fact that at least until recently, no cryobiologist who was seriously interested in cryonics was allowed to be a member,
but I’m not sure… their current position statement is essentially “it is outside the purview of the Society for Cryobiology”, which, if sincere, would have to mean that the beef is over?
I have this draft, Extraordinary Claims Routinely Get Proven with Ordinary Evidence, a debunking of that old Sagan line. We actually do routinely prove extraordinary claims like evolution or plate tectonics with old evidence that’s been in front of our faces for hundreds of years, and that’s important.
But Evolution and plate tectonics are the only examples I can think of, because I’m not really particularly interested in the history of science, for similar underlying reasons to being the one who wants to write this post. Collecting buckets of examples is not as useful as being able to deeply interpret and explain the examples that you have.
But I’m still not posting this until someone gives me more examples! I want the post to fight and win on the terms of the people it’s trying to reach. Subdue the stamp collectors with stamps. It’s the only way they’ll listen.
The true thing that Sagan’s line might be interpreted to mean is “A claim which is very unlikely on priors needs very strong evidence to end up with a posterior probability close to 1.” “Extraordinary evidence” would ideally have been stated as “extraordinarily strong evidence”, but that makes the line a bit clunkier. Unfortunately, there is often a tradeoff between accuracy and pithiness. Many pithy sayings require a bit of interpretation/reconstruction to get the correct underlying idea. I think anyone who invokes a catchphrase should be aware of this, though I don’t know how many people share this perspective.
Are there in fact a significant number of people who take it at the face value of “extraordinary evidence” and think it must mean it was obtained via super-advanced technology or something?
Strong evidence is incredibly ordinary, and that genuinely doesn’t seem to be intuitive. Like, every time you see a bit string longer than a kilobyte there is a claim in your corpus that goes from roughly zero to roughly one, and you are doing that all day. I don’t know about you, but I still don’t think I’ve fully digested that.
Some extraordinary claims established by ordinary evidence:
Stomach ulcers are caused by infection with Helicobacter Pylori. It was a very surprising discovery that was established by a few simple tests.
The correctness of Kepler’s laws of planetary motion was established almost entirely by analyzing historical data, some of it dating back to the ancient Greeks.
Special relativity was entirely a reinterpretation of existing data. Ditto Einstein’s explanation of the photoelectric effect, discovered in the same year.
I assume you mean, by “stamp collectors”, people on the biology/chemistry/materials science side of things, rather than on the math/theoretical physics side of things, and by “extraordinary claims” you mean something along the lines of “claims that a specific simple model makes good predictions in a wide variety of circumstances”, and by “ordinary evidence” you mean something along the lines of “some local pieces of evidence from one or a few specific experiments”. So with that in mind:
Biology:
Cell theory (“If you look at a tissue sample from a macroscopic organism, it will be made of cells.”)
Homeostasis (“If you change the exterior environment of an organism, its responses will tend to keep its internal state within a certain range in terms of e.g. temperature, salinity, pH, etc).
DNA->RNA->protein pipeline (“If you look at an organism’s DNA, you can predict the order of the amino acid residues in the proteins it expresses,, and every organism uses pretty much the same codon table which is blah blah”)
Chemistry:
Acid-base chemistry
Bond geometry and its relation to orbitals (e.g. “bond angles will tend to be ~109º for things attached to a carbon that has only single bonds, because that’s the angle that two vertices of a tetrahedron make across the center”).
Bond energy (i.e. “you can predict pretty well how much energy a given reaction will produce just by summing the bond energy of each individual bond before and after”)
Resonance/delocalization
Law of Mass Action: (i.e. “for every chemical reaction, there is an equilibrium ratio of reactants to products at a constant temperature. That equilibrium is computable based on the number of molecules in the reactants and products, and the energy contained within those molecules”)
For organic chemistry, literally hundreds of “if you put a molecule with this specific structure in with these specific reagents in these specific conditions, you will get a molecule that is transformed in this one specific way with no other important changes”. For a concrete example: if you have a Grignard Reagent RMgX, and an aldehyde R’HO, you can combine them to form R-CH(OH)-R’. Individually, these “laws” are perhaps not so satisfying, but in combination they say “for pretty much any organic compound, you can synthesize that compound from relatively cheap inputs by using some combination of these reactions”.
Misc other fields
The photovoltaic effect, demonstrated in 1839, and its relation to the band gap—the fact that some materials have energy levels that are “forbidden” to electrons led to unexplained empirical observations all the way back in 1839, and understanding the phenomenon (and tinkering a whole bunch, because analytical and computational methods don’t even come close to being good enough) paved the way to the information age.
Fourier Transforms aren’t directly a physical phenomenon, but the fact that you can convert a series of values of any complex periodic system down into a sum of simple sine waves, knowing only the input frequencies but not the input amplitudes, meant that you could e.g. mechanically predict the future tides for a location based only on the past tides for that location.
I’m not so sure how well these examples will demonstrate that “collecting buckets of examples is not as useful as being able to deeply interpret and explain the examples that you have”, but also I’m pretty sure that’s just false a lot of the time—you may have a deep theory of everything which is in principle sufficient, but that doesn’t mean your deep theory of everything is computationally tractable for solving the specific problem you have in front of you.
Noticing I’ve been operating under a bias where I notice existential risk precursors pretty easily (EG, biotech, advances in computing hardware), but I notice no precursors of existential safety. To me it is as if technologies that tend to do more good than harm, or at least, would improve our odds by their introduction, social or otherwise, do not exist. That can’t be right, surely?...
When I think about what they might be… I find only cultural technologies, or political conditions: the strength of global governance, the clarity of global discourses, perhaps the existence of universities. But that can’t be it. These are all low hanging fruit, things that already exist. Differential progress is about what could be made to exist.
Probably has something to do with the fact that a catastrophe is an event, and safety is an absence of something. It’s just inherently harder to point at a thing and say that it caused fewer catastrophes to happen. Show me the non-catastrophes. Bring them to me, put them on my table. You can’t do it.
I’d say it’s an aspect of negativity bias, where we focus more on the bad things than on the good things. It’s already happening in AI safety, and AI in general, so your bias is essentially a facet of negativity bias.
There’s a sense in which negativity bias is just rationality; you focus on the things you can improve, that’s where the work is. These things are sometimes called “problems”. The thing is, the healthy form of this is aware that the work can actually be done, so, should be very interested in, and aware of technologies of existential safety, and that is where I am and have been for a long time.
The problem is that focusing on a negative frame enabled by negativity bias will blind you to solutions, and is in general a great way to get depressed fast, which kills your ability to solve problems. Even more importantly, the problems might be imaginary, created by negativity biases.
It’s essentially a frame that views things in a negative light, or equivalently a frame that views a certain issue as by default negative unless action is taken.
For example, climate change can be viewed in the negative, which is that we have to solve the problem or we all die, or as a positive frame where we can solve the problem by green tech
I was hoping to understand why people who are concerned about the climate ignore greentech/srm.
One effect, is that people who want to raise awareness about the severity of an issue have an incentive to avoid acknowledging solutions to it, because that diminishes its severity. But this is an egregore-level phenomenon, there is no individual negative cognitive disposition that’s driving that phenomenon as far as I can tell. Mostly, in the case of climate, it seems to be driven by a craving for belonging in a political scene.
The point I was trying to make is that we click on and read negative news, and this skews our perceptions of what’s happening, and critically the negativity bias operates regardless of the actual reality of the problem, that is it doesn’t distinguish between the things that are very bad, just merely bad but solvable, and not bad at all.
In essence, I’m positing a selection effect, where we keep hearing more about the bad things, and hear less or none about the good things, so we are biased to believe that our world is more negative than it actually is.
And to connect it to the first comment, the reason you keep noticing precursors to existentially risky technology but not precursors existentially safe technology, or why this is happening:
To me it is as if technologies that tend to do more good than harm, or at least, would improve our odds by their introduction, social or otherwise, do not exist. That can’t be right, surely?...
Is essentially an aspect of negativity bias because your information sources emphasize the negative over the positive news, no matter what reality looks like.
Some biotech contributes to existential risk but others doesn’t. A lot of vaccine technology doesn’t increase existential risk but reduces it because of reduced danger from viruses. Phage therapy is the same for reducing the risk from infectious bacteria.
LessWrong itself is a form of technology that’s intended to lead to existential risk reduction by facillitating a knowledge community to exist that otherwise wouldn’t.
The general idea of CFAR is that social technology they developed like double crux helps people to think more clearly and thus reduce existential risk.
Observation from playing Network Wars: The concept of good or bad luck is actually crucial for assessing one’s own performance in games with output randomness (most games irl). You literally can’t tell what you’re doing well in any individual match without that, it a sensitivity that lets you see through the noise and learn more informative lessons from each experience.
Well, yes, but nobody’s going to get paid (in status) by putting it that way. It needs to be much more obfuscated and have a lot of math that requires mental contortions to apply to actual human choices.
More seriously, yes, but most of those words need multiple books worth of exploration to fully understand. “basically just”: what are the limits, edge cases, and applicability of exceptions. “comparing”: on what dimensions, and how to handle uncertainty that often overwhelms the simple calculations. “alternatives”: there are a near-infinite number, how to discover and filter the ones worth thinking about. “picking”: what does this even mean? how does choice work? “best”: what does THIS even mean? How would one know if an alternative is better than another?
Oh, and I guess “right?”: what would “wrong” look like, and how would you know which it is?
Yeah I guess the “just” was in jest, we all know how complicated this gets when you’re serious about it.
I considered adding a paragraph about how and why people fail to do this, how this definition characterizes ingroup and outgroup, and could probably write an entire post about it.
Not quite, in my opinion. In practice, humans tend to be wrong in predictable ways (what we call a “bias”) and so picking the best option isn’t easy.
What we call “rationality” tends to be the techniques / thought patterns that make us more likely to pick the best option when comparing alternatives.
Idea: Screen burn correction app that figures out how to exactly negate your screen’s issues by pretty much looking at itself in a mirror through the selfie cam, trying to display pure white, remembering the imperfections it sees, then tinting everything with the negation of that from then on.
Nobody seems to have made this yet. I think there might be things for tinting your screen in general, but it doesn’t know the specific quirks of your screenburn. Most of the apps for screen burn recommend that you just burn every color over the entire screen that isn’t damaged yet, so that they all get to be equally damaged, which seems like a really bad thing to be recommending.
As I get closer to posting my proposal to build a social network that operates on curators recommended via webs of trust, it is becoming easier for me to question existing collaborative filtering processes.
And, damn, scores on posts are pretty much meaningless if you don’t know how many people have seen the post, how many tried to read it, how many read all of it, and what the up/down ratio is. If you’re missing one of those pieces of information, then there exists an explanation for a low score that has no relationship to the post’s quality, and you can’t use the score to make a decision as to whether to give it a chance.
Hmm. It appears to me that Qualia are whatever observations affect indexical claims, and anything that affects indexical claims is a qualia, and this is probably significant
Yes, this seems straightforwardly true, although I don’t think it’s especially significant unless I’m failing to think of some relevant context about why you think indexical claims matter so much (but then I don’t spend a lot of time thinking very hard about semantics in a formal context, so maybe I’m just failing to grasp what all is encompassed by “indexical”).
It’s important because demystifying qualia would win esteem in a very large philosophical arena, heh. More seriously though, it seems like it would have to strike at something close to the heart of the meaning of agency.
Hmm. It appears to me that Qualia are whatever observations affect indexical claims, and anything that affects indexical claims is a qualia
I don’t think so, here is a counter-example:
Alice and Bob start talking in a room. Alice has an identical twin, Alex. Bob doesn’t know about the twin and thinks he’s talking to Alex. Bob asks: “How are you today?”. Before Alice responds, Alex walks in.
Bob’s observation of Alex will surprise him, and he’ll quickly figure out that something’s going on. But more importantly: Bob’s observation of Alex alters the indexical ‘you’ in “How are you today?” (at least compared to Bob’s intent, and it might change for Alice if she realises Bob was mistaken, too).
I don’t think this is anything close to describing qualia. The experience of surprise can be a quale, the feeling of discovering something can be a quale (eureka moments), the experience of the colour blue is a quale, but the observation of Alex is not.
An indexical is, roughly speaking, a linguistic expression whose reference can shift from context to context. For example, the indexical ‘you’ may refer to one person in one context and to another person in another context.
That’s a well constructed example I think, but no that seems to be a completely different sense of “indexical”. The concept of indexical uncertainty we’re interested in is… I think… uncertainty about which kind of body or position in the universe your seat of consciousness is in, given that there could be more than one. The Sleeping Beauty problem is the most widely known example. The mirror chamber was another example.
The concept of indexical uncertainty we’re interested in is… I think… uncertainty about which kind of body or position in the universe your seat of consciousness is in, given that there could be more than one.
I’m not sure I understand yet, but does the following line up with how you’re using the word?
Indexical uncertainty is uncertainty around the exact matter (or temporal location of such matter) that is directly facilitating, and required by, a mind. (this could be your mind or another person’s mind)
Notes:
“exact” might be too strong a word
I added “or temporal location of such matter” to cover the sleeping beauty case (which, btw, I’m apparently a halfer or double halfer according to wikipedia’s classifications, but haven’t thought much about it)
Edit/PS: I think my counter-example with Alice, Alex, and Bob still works with this definition.
I think I’m saying those are going to to turn out to be the same thing, though I’m not sure exactly where that intuition is coming from yet. Could be wrong.
Decision theoretic things that I’m not sure whether are demonic, or just real and inescapable and legitimate, and I genuinely don’t fucking know which, yet:
extortion/threats/building torturizers to gain bargaining power
(or complying to with extortionary threats)
assigning bargaining power in proportion to an agent’s strength or wealth, as opposed to in proportion to its phenomenological measure.
(arguably wrong if you extend a rawlsian veil back beyond even your awareness of which observer you are or what your utility function is, which seems mathematically elegant and morally neat.)
I’m pretty sure extortion is fine because under an advanced cooperative bargaining process the threats are never actually carried out (not even to the point of needing to build the torturizer and leave it in a warehouse), and it seems to be neater as a result of not requiring us to make assumptions about what constitutes action or inaction.
Can you expand on what you mean by “demonic”? Is it a shorthand for “indicative of broken cognition, because it’s both cruel and unnecessary”, or something else? I THINK what you’re wondering about is whether these techniques/behaviors are ever actually optimal when dealing with misaligned agents who you nonetheless consider to be moral patients. Is that close?
I think that both questions are related to uncertainty about the other agent(s). Bargaining implies costly changes to future behaviors (of both parties). Which makes signaling of capability and willingness important. Bargainers need to signal that they will change something in a meaningful way based on whatever agreement/concession is reached. In repeated interaction (which is almost all of them), actual follow-through is the strongest signal.
So, actual torture is the strongest signal of willingness and ability to torture. Building a torturizer shows capability, but only hints at willingness. Having materials that could build a torturizer or an orgasmatron is pretty weak, but not zero. Likewise with strength and wealth—it’s shows capability of benefit/reduced-harm from cooperation, which is an important prerequisite.
I don’t think you can assert that threats are never carried out, unless you somehow have perfect mutual knowledge (and then, it’s not bargaining, it’s just optimization). Thomas Schelling won a Nobel for his work in bargaining under uncertainty, and I think most of those calculations are valid, no matter how adavnced and rational the involved agents are, when their knowledge is incomplete and they’re misaligned in their goals.
Since acausal trade issues are basically spiritual, when the trade is bad I seek a word that means “spiritually bad.” You can read it as just “bad” if you want.
So, actual torture is the strongest signal of willingness and ability to torture. Building a torturizer shows capability, but only hints at willingness. Having materials that could build a torturizer or an orgasmatron is pretty weak, but not zero
Probable crux: Cognitive transparency is actually easy for advanced agencies. It’s difficult for a human to prove to a distant human that they have the means to build and deploy a torturizer without actually doing it. It wouldn’t be difficult for brains that were designed to be capable of proving the state of their beliefs, and AGI participating in a community with other AGI would want to be capable of that. (The contemporary analog is trusted computing. The number of coordination problems it could solve for us, today, if it were fully applied, is actually depressing.)
There would still be uncertainties as a result of mutual comprehensibility issues, but they could turn out to be of negligible importance, especially once nobody’s lying any more.
Ah, sorry—I missed the acausal assumption in the post. I generally ignore such those explorations, as I don’t think “decision” is the right word without causality and conditional probability.
I think you’re right that cognitive transparency is a crux. I strongly doubt it’s possible to be mutual, or possible between agents near each other in cognitive power. It may be possible for a hyperintelligence to understand/predict a human-level intelligence, but in that case the human is so outclassed that “trade” is the wrong word, and “manipulation” or “slavery” (or maybe “absorption”) is a better model.
You don’t have to be able to simulate something to trust it for this or that. EG, the specification of alphazero is much simpler than the final weights, and knowing its training process, without knowing its weights, you can still trust that it will never, say, take a bribe to throw a match. Even if it comprehended bribery, we know from its spec info that it’s solely interested in winning whatever match it’s currently playing, and no sum would be enough.
To generalize, if we know something’s utility function, and if we know it had a robust design, even if we know nothing else about its history, we know what it’ll do.
A promise-keeping capacity is a property utility functions can have.
A promise-keeping capacity is a property utility functions can have.
Yeah, definitely cruxy. It may be a property that utility functions could have, but it’s not a property that any necessarily do have. Moreover, we have zero examples of robust-designed agents with known utility functions, so it’s extremely unclear whether that will become the norm, let alone the universal assumption.
Trying to figure out why there’s so much in common between Jung’s concept of synchronicity, and acausal trade (in fact, jung seems to have coined the term acausal). Is it:
1) Scott Alexander (known to be a psychologist), or someone, drawing on the language of the paranormal, to accentuate the weird parts of acausal trade/LDT decisionmaking, which is useful to accentuate if you’re trying to communicate the novelty (though troublesome if you’re looking for mundane examples of acausal trade in human social behavior, which we’re pretty sure exist, given how much people talk about having faith in people, and how much emphasis is placed on knowing peoples’ character).
2) A coincidence
3) A synchronicity
4) Jung was actually partly trying to study LDT principles and just happened to have been fooled by some statistical magic tricks into believing in paranormal serendipity as well, and now people have convinced themselves that it was all about the paranormal shit, because 🎵 the discourse only keeps the parts it hates 🎵
I’m wondering if the “Zana just had african DNA” finding might have been a result of measurement or interpretation error: We don’t know the sasquatch markers are, so even if they diverged from sapiens like 100,000 years ago (long enough for actual evolution to occur), as long as there hadn’t been interbreeding with any other ethnicities, aren’t there are tests that totally would have just said “sapiens with no asian or caucasian background therefore african”?
My opinion is that the St Petersberg game isn’t paradoxical, it is very valuable, you should play it, it’s counterintuitive to you because you can’t actually imagine a quantity that comes in linear proportion to utility, you have never encountered one, none seems to exist.
Money, for instance, is definitely not linearly proportionate to utility, the more you get the less it’s worth to you, and at its extremes, it can command no more resources than what the market offers, and if you get enough of it, the market will notice and it will all become valueless.
Every resource that exists has sub-linear utility returns in the extremes.
(Hmm. What about land? Seems linear, to an extent)
Things that healthy people don’t have innate dispositions towards: Optimism, Pessimism, Agreeability, Disagreeability, Patience, Impatience.
Whether you are those things should completely depend on the situation you’re in. If it doesn’t, you may be engaging in magical thinking about how the world works. Things are not guaranteed to go well, nor poorly. People are not fully trustworthy, nor are they consistently malignant. Some things are worth nurturing, others aren’t. It’s all situational.
An analytic account of Depression: When the agent has noticed that strategies that seemed fruitful before have stopped working, and doesn’t have any better strategies in mind.
I imagine you’ll often see this type of depression behavior in algorithmic trading strategies, as soon as they start consistently losing enough money to notice that something must have changed about the trading environment, maybe more sophisticated strategies have found a way to dutch book them. Those strategies will then be retired, and the trader or their agency will have to search for new ones.
Chronic depression in humans (designed agencies wouldn’t tend to have such an obvious bug) kinda feels like when the behavior of searching for new strategies has itself has been caught within depression’s scope as an invalidated strategy.
Searching anew is what’s supposed to happen (I had a post called “Good Gloom” that drew an analogy to turning all of the lights out in a town so that you’ll be able to see the light pollution of a new town, and set out for it), but if the depression is blocking that too, you stop moving.
Wild Speculative Civics: What if we found ways of reliably detecting when tragedies of the commons have occurred, then artificially increased their costs (charging enormous fines) to anyone who might have participated in creating them, until it’s not even individually rational to contribute to them any more?
That sounds like punishing any usage of common resources which is likely undesireable.
Good policy to manage individual commons requires to think through how their usage is best managed. Elinor Ostrom did a lot of research into what works for setting up good systems.
I can imagine a few technical problems, like determining what level of usage is optimal (you don’t want people to overfish the lake, but you don’t know exactly how many fish are there), or the costs of policing. But it would be possible to propose a few dozen situations where this strategy could be used, and address these issues individually; and then perhaps only use the strategy in some of them. Or perhaps by examining individual specific cases, we would discover a common pattern why this doesn’t work.
Until you learn FDT, you cannot see the difference between faith and idealism, nor the difference between pragmatism and cynicism. The tension between idealism and pragmatism genuinely cannot be managed gracefully without FDT, it defines their narrow synthesis.
More should be written about this, because cynicism, idealism afflicts many.
Have you ever seen someone express stern (but valid, actionable) criticisms, conveyed with actual anger, towards an organization, and then been hired, to implement their reforms?
If that has never happened, is there a reasonable explanation for that or is it just, as it appears, almost all orgs are run by and infested with narcissism? (a culture of undervaluing criticism and not protecting critics)
There’s something very creepy to me about the part of research consent forms where it says “my participation was entirely voluntary.”
Do they really think an involuntary participant wouldn’t sign that? If they understand that they would, what purpose could this possibly serve, other than, as is commonly the purpose of contracts; absolving themselves of blame and moving blame to the participant? Which would be downright monstrous. Probably they just aren’t fucking consequentialists, but this is all they end up doing.
This is a minor thing, but it adds an additional creepy garnish: Nothing is 100% voluntary, because everything is a function of the involuntary base reality that other people command force and resources and we want to use them for things so we have to go along with what other people want to some extent. I’m at peace with this, and I would prefer not to have to keep denying it, and it feels like I’m being asked to participate in the addling of moral philosophy.
Maybe it’s some legal hack, like maybe in some situations you can’t dismiss unethical research, but you can dismiss fraudulent research… and a research where people were forced to falsely write that their participation was voluntary, is technically fraudulent.
I notice it also makes sure that if the participants know anything at all about the research, they know it’s supposed to be voluntary, even if they’re still forced to sign it, they learn that the law is supposed to be on their side and there is in theory someone they could call for help.
The reason is to prevent the voluntary participant from later claiming that their participation was involuntary and telling that to the IRB.
‘Well if your participation was involuntary, why did you sign this document?’
It kind of limits the arguments someone could make attacking the ethics of the study. The attacker would have to allege coercion on the order of people being forced to lie on forms under threat.
If someone explicitely writes into their consent forms “my participation was entirely voluntary” and the participation isn’t voluntary it might be easier to attack the person running the trial later.
Important to remember and stand by the Nuremberg Code in these contexts.
There’s a lot of “neuralink will make it easier to solve the alignment problem” stuff going around the mainstream internet right now in response to neuralink’s recent demo.
I’m inclined to agree with Eliezer, that seems wrong; either AGI will be aligned in which case it will make its own neuralink and wont need ours, or it will be unaligned and you really wouldn’t want to connect with it. You can’t make horses competitive with cars by giving them exoskeletons.
But, is there much of a reason to push back against this?
Providing humans with cognitive augmentation probably would help to solve the alignment problem, in a bunch of indirect ways.
It doesn’t seem like a dangerous error at all. It feeds a public desire to understand how AGI might work. Neuralink itself is a great project for medical science. Generally, wrong beliefs cause bad consequences, but I’m having difficulty seeing what they’d be here.
The obvious bad consequence is a false sense of security leading people to just get BCIs instead of trying harder to shape (e.g. delay) AI development.
″ You can’t make horses competitive with cars by giving them exoskeletons. ” <-- this reads to me like a separate argument, rather than a restatement of the one that came before.
I agree that BCI seems unlikely to be a good permanent/long-term solution, unless it helps us solve alignment, which I think it could. It could also just defuse a conflict between AIs and humans, leading us to gracefully give up our control over the future light cone instead of fighting a (probably losing) battle to retain it.
...Your post made me think more about my own (and others’) reasons for rejecting Neuralink as a bad idea… I think there’s a sense of “we’re the experts and Elon is a n00b”. This coupled with feeling a bit burned by Elon first starting his own AI safety org and then ditching it for this… overall doesn’t feel great.
I’ve never been mad at elon for not having decision theoretic alignmentism. I wonder, should I be mad. Should I be mad about the fact that he has never talked to eliezer (eliezer said that in passing a year or two ago on twitter) even though he totally could whenever he wanted.
Also, what happened at OpenAI? He appointed some people to solve the alignment problem, I think we can infer that they told him, “you’ve misunderstood something and the approach you’re advocating (proliferate the technology?) wouldn’t really be all that helpful”, and he responded badly to that? They did not reach mutual understanding?
(instutitional reform take, not important due to short timelines, please ignore)
The kinds of people who do whataboutism, stuff like “this is a dangerous distraction because it takes funding away from other initiatives”, tend also to concentrate in low-bandwidth institutions, the legislature, the committee, economies righteously withering, the global discourse of the current thing, the new york times, the ivy league. These institutions recognize no alternatives to them, while, by their nature, they can never grow to the stature required to adequately perform the task assigned to them.
I don’t think this is a coincidence, and it makes it much easier for me to sympathize with these people: They actually believe that we can’t deal with more than one thing at a time.
They generally have no hope for decentralized decisionmaking, and when you examine them closely you find that they don’t really seem to believe in democracy, they’ve given up on it, they don’t talk about reforming it, they don’t want third parties, they’ve generally never heard of decentralized public funding mechanisms, certainly not futarchy. So it’s kind of as simple as that. They’re not being willfully ignorant. We just have to show them the alternatives, and properly, we basically haven’t done it yet. The minarchists never offered a solution to negative externalities or public goods provision. There are proposals but the designs are still vague and poorly communicated. There has never been an articulation of enlightened technocracy, which is essentially just succeeding at specialization or parallelization in executive decisionmaking. I’m not sure enlightened technocracy was ever possible until the proposal of futarchy, a mechanism by which non-experts can hold claimed experts accountable.
Theory: Photic Sneezing (the phenotype where a person sneezes when exposed to a bright light, very common) evolved as a hasty adaptation to indoor cooking or indoor fires, clearing the lungs only when the human leaves the polluted environment.
The newest adaptations will tend to be the roughest, I’m guessing it arose only in the past 500k years or so as a response to artificial dwellings and fire use.
Considering doing a post about how it is possible the Society for Cryobiology might be wrong about Cryonics, it would have something to do with the fact that at least until recently, no cryobiologist who was seriously interested in cryonics was allowed to be a member,
but I’m not sure… their current position statement is essentially “it is outside the purview of the Society for Cryobiology”, which, if sincere, would have to mean that the beef is over?
( statement is https://www.societyforcryobiology.org/assets/documents/Position_Statement_Cryonics_Nov_18.pdf )
I have this draft, Extraordinary Claims Routinely Get Proven with Ordinary Evidence, a debunking of that old Sagan line. We actually do routinely prove extraordinary claims like evolution or plate tectonics with old evidence that’s been in front of our faces for hundreds of years, and that’s important.
But Evolution and plate tectonics are the only examples I can think of, because I’m not really particularly interested in the history of science, for similar underlying reasons to being the one who wants to write this post. Collecting buckets of examples is not as useful as being able to deeply interpret and explain the examples that you have.
But I’m still not posting this until someone gives me more examples! I want the post to fight and win on the terms of the people it’s trying to reach. Subdue the stamp collectors with stamps. It’s the only way they’ll listen.
The true thing that Sagan’s line might be interpreted to mean is “A claim which is very unlikely on priors needs very strong evidence to end up with a posterior probability close to 1.” “Extraordinary evidence” would ideally have been stated as “extraordinarily strong evidence”, but that makes the line a bit clunkier. Unfortunately, there is often a tradeoff between accuracy and pithiness. Many pithy sayings require a bit of interpretation/reconstruction to get the correct underlying idea. I think anyone who invokes a catchphrase should be aware of this, though I don’t know how many people share this perspective.
Are there in fact a significant number of people who take it at the face value of “extraordinary evidence” and think it must mean it was obtained via super-advanced technology or something?
Strong evidence is incredibly ordinary, and that genuinely doesn’t seem to be intuitive. Like,
every time you see a bit string longer than a kilobyte there is a claim in your corpus that goes from roughly zero to roughly one, and you are doing that all day. I don’t know about you, but I still don’t think I’ve fully digested that.
Some extraordinary claims established by ordinary evidence:
Stomach ulcers are caused by infection with Helicobacter Pylori. It was a very surprising discovery that was established by a few simple tests.
The correctness of Kepler’s laws of planetary motion was established almost entirely by analyzing historical data, some of it dating back to the ancient Greeks.
Special relativity was entirely a reinterpretation of existing data. Ditto Einstein’s explanation of the photoelectric effect, discovered in the same year.
I assume you mean, by “stamp collectors”, people on the biology/chemistry/materials science side of things, rather than on the math/theoretical physics side of things, and by “extraordinary claims” you mean something along the lines of “claims that a specific simple model makes good predictions in a wide variety of circumstances”, and by “ordinary evidence” you mean something along the lines of “some local pieces of evidence from one or a few specific experiments”. So with that in mind:
Biology:
Cell theory (“If you look at a tissue sample from a macroscopic organism, it will be made of cells.”)
Homeostasis (“If you change the exterior environment of an organism, its responses will tend to keep its internal state within a certain range in terms of e.g. temperature, salinity, pH, etc).
DNA->RNA->protein pipeline (“If you look at an organism’s DNA, you can predict the order of the amino acid residues in the proteins it expresses,, and every organism uses pretty much the same codon table which is blah blah”)
Chemistry:
Acid-base chemistry
Bond geometry and its relation to orbitals (e.g. “bond angles will tend to be ~109º for things attached to a carbon that has only single bonds, because that’s the angle that two vertices of a tetrahedron make across the center”).
Bond energy (i.e. “you can predict pretty well how much energy a given reaction will produce just by summing the bond energy of each individual bond before and after”)
Resonance/delocalization
Law of Mass Action: (i.e. “for every chemical reaction, there is an equilibrium ratio of reactants to products at a constant temperature. That equilibrium is computable based on the number of molecules in the reactants and products, and the energy contained within those molecules”)
For organic chemistry, literally hundreds of “if you put a molecule with this specific structure in with these specific reagents in these specific conditions, you will get a molecule that is transformed in this one specific way with no other important changes”. For a concrete example: if you have a Grignard Reagent RMgX, and an aldehyde R’HO, you can combine them to form R-CH(OH)-R’. Individually, these “laws” are perhaps not so satisfying, but in combination they say “for pretty much any organic compound, you can synthesize that compound from relatively cheap inputs by using some combination of these reactions”.
Misc other fields
The photovoltaic effect, demonstrated in 1839, and its relation to the band gap—the fact that some materials have energy levels that are “forbidden” to electrons led to unexplained empirical observations all the way back in 1839, and understanding the phenomenon (and tinkering a whole bunch, because analytical and computational methods don’t even come close to being good enough) paved the way to the information age.
Fourier Transforms aren’t directly a physical phenomenon, but the fact that you can convert a series of values of any complex periodic system down into a sum of simple sine waves, knowing only the input frequencies but not the input amplitudes, meant that you could e.g. mechanically predict the future tides for a location based only on the past tides for that location.
I’m not so sure how well these examples will demonstrate that “collecting buckets of examples is not as useful as being able to deeply interpret and explain the examples that you have”, but also I’m pretty sure that’s just false a lot of the time—you may have a deep theory of everything which is in principle sufficient, but that doesn’t mean your deep theory of everything is computationally tractable for solving the specific problem you have in front of you.
Noticing I’ve been operating under a bias where I notice existential risk precursors pretty easily (EG, biotech, advances in computing hardware), but I notice no precursors of existential safety. To me it is as if technologies that tend to do more good than harm, or at least, would improve our odds by their introduction, social or otherwise, do not exist. That can’t be right, surely?...
When I think about what they might be… I find only cultural technologies, or political conditions: the strength of global governance, the clarity of global discourses, perhaps the existence of universities. But that can’t be it. These are all low hanging fruit, things that already exist. Differential progress is about what could be made to exist.
Probably has something to do with the fact that a catastrophe is an event, and safety is an absence of something. It’s just inherently harder to point at a thing and say that it caused fewer catastrophes to happen. Show me the non-catastrophes. Bring them to me, put them on my table. You can’t do it.
I’d say it’s an aspect of negativity bias, where we focus more on the bad things than on the good things. It’s already happening in AI safety, and AI in general, so your bias is essentially a facet of negativity bias.
There’s a sense in which negativity bias is just rationality; you focus on the things you can improve, that’s where the work is. These things are sometimes called “problems”. The thing is, the healthy form of this is aware that the work can actually be done, so, should be very interested in, and aware of technologies of existential safety, and that is where I am and have been for a long time.
The problem is that focusing on a negative frame enabled by negativity bias will blind you to solutions, and is in general a great way to get depressed fast, which kills your ability to solve problems. Even more importantly, the problems might be imaginary, created by negativity biases.
What is a negative frame.
It’s essentially a frame that views things in a negative light, or equivalently a frame that views a certain issue as by default negative unless action is taken.
For example, climate change can be viewed in the negative, which is that we have to solve the problem or we all die, or as a positive frame where we can solve the problem by green tech
I was hoping to understand why people who are concerned about the climate ignore greentech/srm.
One effect, is that people who want to raise awareness about the severity of an issue have an incentive to avoid acknowledging solutions to it, because that diminishes its severity. But this is an egregore-level phenomenon, there is no individual negative cognitive disposition that’s driving that phenomenon as far as I can tell.
Mostly, in the case of climate, it seems to be driven by a craving for belonging in a political scene.
The point I was trying to make is that we click on and read negative news, and this skews our perceptions of what’s happening, and critically the negativity bias operates regardless of the actual reality of the problem, that is it doesn’t distinguish between the things that are very bad, just merely bad but solvable, and not bad at all.
In essence, I’m positing a selection effect, where we keep hearing more about the bad things, and hear less or none about the good things, so we are biased to believe that our world is more negative than it actually is.
And to connect it to the first comment, the reason you keep noticing precursors to existentially risky technology but not precursors existentially safe technology, or why this is happening:
Is essentially an aspect of negativity bias because your information sources emphasize the negative over the positive news, no matter what reality looks like.
The link where I got this idea is below:
https://archive.is/lc0aY
Some biotech contributes to existential risk but others doesn’t. A lot of vaccine technology doesn’t increase existential risk but reduces it because of reduced danger from viruses. Phage therapy is the same for reducing the risk from infectious bacteria.
LessWrong itself is a form of technology that’s intended to lead to existential risk reduction by facillitating a knowledge community to exist that otherwise wouldn’t.
The general idea of CFAR is that social technology they developed like double crux helps people to think more clearly and thus reduce existential risk.
Observation from playing Network Wars: The concept of good or bad luck is actually crucial for assessing one’s own performance in games with output randomness (most games irl). You literally can’t tell what you’re doing well in any individual match without that, it a sensitivity that lets you see through the noise and learn more informative lessons from each experience.
Rationality is basically just comparing alternatives and picking the best one, right?
Yes, but the hardest part of that turns out to be generating high quality model alternatives under limited compute lmao
Yes, but doing that correctly requires a lot of preparation.
Well, yes, but nobody’s going to get paid (in status) by putting it that way. It needs to be much more obfuscated and have a lot of math that requires mental contortions to apply to actual human choices.
More seriously, yes, but most of those words need multiple books worth of exploration to fully understand. “basically just”: what are the limits, edge cases, and applicability of exceptions. “comparing”: on what dimensions, and how to handle uncertainty that often overwhelms the simple calculations. “alternatives”: there are a near-infinite number, how to discover and filter the ones worth thinking about. “picking”: what does this even mean? how does choice work? “best”: what does THIS even mean? How would one know if an alternative is better than another?
Oh, and I guess “right?”: what would “wrong” look like, and how would you know which it is?
Yeah I guess the “just” was in jest, we all know how complicated this gets when you’re serious about it.
I considered adding a paragraph about how and why people fail to do this, how this definition characterizes ingroup and outgroup, and could probably write an entire post about it.
Not quite, in my opinion. In practice, humans tend to be wrong in predictable ways (what we call a “bias”) and so picking the best option isn’t easy.
What we call “rationality” tends to be the techniques / thought patterns that make us more likely to pick the best option when comparing alternatives.
Idea: Screen burn correction app that figures out how to exactly negate your screen’s issues by pretty much looking at itself in a mirror through the selfie cam, trying to display pure white, remembering the imperfections it sees, then tinting everything with the negation of that from then on.
Nobody seems to have made this yet. I think there might be things for tinting your screen in general, but it doesn’t know the specific quirks of your screenburn. Most of the apps for screen burn recommend that you just burn every color over the entire screen that isn’t damaged yet, so that they all get to be equally damaged, which seems like a really bad thing to be recommending.
As I get closer to posting my proposal to build a social network that operates on curators recommended via webs of trust, it is becoming easier for me to question existing collaborative filtering processes.
And, damn, scores on posts are pretty much meaningless if you don’t know how many people have seen the post, how many tried to read it, how many read all of it, and what the up/down ratio is. If you’re missing one of those pieces of information, then there exists an explanation for a low score that has no relationship to the post’s quality, and you can’t use the score to make a decision as to whether to give it a chance.
Hmm. It appears to me that Qualia are whatever observations affect indexical claims, and anything that affects indexical claims is a qualia, and this is probably significant
Yes, this seems straightforwardly true, although I don’t think it’s especially significant unless I’m failing to think of some relevant context about why you think indexical claims matter so much (but then I don’t spend a lot of time thinking very hard about semantics in a formal context, so maybe I’m just failing to grasp what all is encompassed by “indexical”).
It’s important because demystifying qualia would win esteem in a very large philosophical arena, heh. More seriously though, it seems like it would have to strike at something close to the heart of the meaning of agency.
I don’t think so, here is a counter-example:
Alice and Bob start talking in a room. Alice has an identical twin, Alex. Bob doesn’t know about the twin and thinks he’s talking to Alex. Bob asks: “How are you today?”. Before Alice responds, Alex walks in.
Bob’s observation of Alex will surprise him, and he’ll quickly figure out that something’s going on. But more importantly: Bob’s observation of Alex alters the indexical ‘you’ in “How are you today?” (at least compared to Bob’s intent, and it might change for Alice if she realises Bob was mistaken, too).
I don’t think this is anything close to describing qualia. The experience of surprise can be a quale, the feeling of discovering something can be a quale (eureka moments), the experience of the colour blue is a quale, but the observation of Alex is not.
Do you agree with this? (It’s from https://plato.stanford.edu/entries/indexicals/)
Btw, ‘qualia’ is the plural form of ‘quale’
That’s a well constructed example I think, but no that seems to be a completely different sense of “indexical”. The concept of indexical uncertainty we’re interested in is… I think… uncertainty about which kind of body or position in the universe your seat of consciousness is in, given that there could be more than one. The Sleeping Beauty problem is the most widely known example. The mirror chamber was another example.
I’m not sure I understand yet, but does the following line up with how you’re using the word?
Indexical uncertainty is uncertainty around the exact matter (or temporal location of such matter) that is directly facilitating, and required by, a mind. (this could be your mind or another person’s mind)
Notes:
“exact” might be too strong a word
I added “or temporal location of such matter” to cover the sleeping beauty case (which, btw, I’m apparently a halfer or double halfer according to wikipedia’s classifications, but haven’t thought much about it)
Edit/PS: I think my counter-example with Alice, Alex, and Bob still works with this definition.
I can see how this might result from confusing consciousness qua phenomenonality with consciousness qua personal identity.
I think I’m saying those are going to to turn out to be the same thing, though I’m not sure exactly where that intuition is coming from yet. Could be wrong.
Why would that be the case?
Decision theoretic things that I’m not sure whether are demonic, or just real and inescapable and legitimate, and I genuinely don’t fucking know which, yet:
extortion/threats/building torturizers to gain bargaining power
(or complying to with extortionary threats)
assigning bargaining power in proportion to an agent’s strength or wealth, as opposed to in proportion to its phenomenological measure.
(arguably wrong if you extend a rawlsian veil back beyond even your awareness of which observer you are or what your utility function is, which seems mathematically elegant and morally neat.)
I’m pretty sure extortion is fine because under an advanced cooperative bargaining process the threats are never actually carried out (not even to the point of needing to build the torturizer and leave it in a warehouse), and it seems to be neater as a result of not requiring us to make assumptions about what constitutes action or inaction.
Subjectivity-weighting seems much more thorny.
Can you expand on what you mean by “demonic”? Is it a shorthand for “indicative of broken cognition, because it’s both cruel and unnecessary”, or something else? I THINK what you’re wondering about is whether these techniques/behaviors are ever actually optimal when dealing with misaligned agents who you nonetheless consider to be moral patients. Is that close?
I think that both questions are related to uncertainty about the other agent(s). Bargaining implies costly changes to future behaviors (of both parties). Which makes signaling of capability and willingness important. Bargainers need to signal that they will change something in a meaningful way based on whatever agreement/concession is reached. In repeated interaction (which is almost all of them), actual follow-through is the strongest signal.
So, actual torture is the strongest signal of willingness and ability to torture. Building a torturizer shows capability, but only hints at willingness. Having materials that could build a torturizer or an orgasmatron is pretty weak, but not zero. Likewise with strength and wealth—it’s shows capability of benefit/reduced-harm from cooperation, which is an important prerequisite.
I don’t think you can assert that threats are never carried out, unless you somehow have perfect mutual knowledge (and then, it’s not bargaining, it’s just optimization). Thomas Schelling won a Nobel for his work in bargaining under uncertainty, and I think most of those calculations are valid, no matter how adavnced and rational the involved agents are, when their knowledge is incomplete and they’re misaligned in their goals.
Since acausal trade issues are basically spiritual, when the trade is bad I seek a word that means “spiritually bad.” You can read it as just “bad” if you want.
Probable crux: Cognitive transparency is actually easy for advanced agencies. It’s difficult for a human to prove to a distant human that they have the means to build and deploy a torturizer without actually doing it. It wouldn’t be difficult for brains that were designed to be capable of proving the state of their beliefs, and AGI participating in a community with other AGI would want to be capable of that. (The contemporary analog is trusted computing. The number of coordination problems it could solve for us, today, if it were fully applied, is actually depressing.)
There would still be uncertainties as a result of mutual comprehensibility issues, but they could turn out to be of negligible importance, especially once nobody’s lying any more.
Obviously it is not clear why “acausal trade issues” would be “spiritual” or what you mean by those terms.
So what?
Ah, sorry—I missed the acausal assumption in the post. I generally ignore such those explorations, as I don’t think “decision” is the right word without causality and conditional probability.
I think you’re right that cognitive transparency is a crux. I strongly doubt it’s possible to be mutual, or possible between agents near each other in cognitive power. It may be possible for a hyperintelligence to understand/predict a human-level intelligence, but in that case the human is so outclassed that “trade” is the wrong word, and “manipulation” or “slavery” (or maybe “absorption”) is a better model.
You don’t have to be able to simulate something to trust it for this or that. EG, the specification of alphazero is much simpler than the final weights, and knowing its training process, without knowing its weights, you can still trust that it will never, say, take a bribe to throw a match. Even if it comprehended bribery, we know from its spec info that it’s solely interested in winning whatever match it’s currently playing, and no sum would be enough.
To generalize, if we know something’s utility function, and if we know it had a robust design, even if we know nothing else about its history, we know what it’ll do.
A promise-keeping capacity is a property utility functions can have.
Yeah, definitely cruxy. It may be a property that utility functions could have, but it’s not a property that any necessarily do have. Moreover, we have zero examples of robust-designed agents with known utility functions, so it’s extremely unclear whether that will become the norm, let alone the universal assumption.
Trying to figure out why there’s so much in common between Jung’s concept of synchronicity, and acausal trade (in fact, jung seems to have coined the term acausal). Is it:
1) Scott Alexander (known to be a psychologist), or someone, drawing on the language of the paranormal, to accentuate the weird parts of acausal trade/LDT decisionmaking, which is useful to accentuate if you’re trying to communicate the novelty (though troublesome if you’re looking for mundane examples of acausal trade in human social behavior, which we’re pretty sure exist, given how much people talk about having faith in people, and how much emphasis is placed on knowing peoples’ character).
2) A coincidence
3) A synchronicity
4) Jung was actually partly trying to study LDT principles and just happened to have been fooled by some statistical magic tricks into believing in paranormal serendipity as well, and now people have convinced themselves that it was all about the paranormal shit, because 🎵 the discourse only keeps the parts it hates 🎵
An argument that the reason most “sasquatch” samples turn out to have human DNA is that sasquatch/wildman phenotype (real) is actually not very many mutations away from sapiens, because it’s mostly just a result of re-enabling a bunch of traits that were disabled under sapiens self-domestication/neotenization https://www.againsttheinternet.com/post/60-revolutionary-biology-pt-2-the-development-and-evolution-of-sasquatch
I’m wondering if the “Zana just had african DNA” finding might have been a result of measurement or interpretation error: We don’t know the sasquatch markers are, so even if they diverged from sapiens like 100,000 years ago (long enough for actual evolution to occur), as long as there hadn’t been interbreeding with any other ethnicities, aren’t there are tests that totally would have just said “sapiens with no asian or caucasian background therefore african”?
If so, that could be investigated.
Fun factoid: it is claimed that some South American apes used fire before human came and kill them: https://evolbiol.ru/document/915
My opinion is that the St Petersberg game isn’t paradoxical, it is very valuable, you should play it, it’s counterintuitive to you because you can’t actually imagine a quantity that comes in linear proportion to utility, you have never encountered one, none seems to exist.
Money, for instance, is definitely not linearly proportionate to utility, the more you get the less it’s worth to you, and at its extremes, it can command no more resources than what the market offers, and if you get enough of it, the market will notice and it will all become valueless.
Every resource that exists has sub-linear utility returns in the extremes.
(Hmm. What about land? Seems linear, to an extent)
Things that healthy people don’t have innate dispositions towards: Optimism, Pessimism, Agreeability, Disagreeability, Patience, Impatience.
Whether you are those things should completely depend on the situation you’re in. If it doesn’t, you may be engaging in magical thinking about how the world works. Things are not guaranteed to go well, nor poorly. People are not fully trustworthy, nor are they consistently malignant. Some things are worth nurturing, others aren’t. It’s all situational.
An analytic account of Depression: When the agent has noticed that strategies that seemed fruitful before have stopped working, and doesn’t have any better strategies in mind.
I imagine you’ll often see this type of depression behavior in algorithmic trading strategies, as soon as they start consistently losing enough money to notice that something must have changed about the trading environment, maybe more sophisticated strategies have found a way to dutch book them. Those strategies will then be retired, and the trader or their agency will have to search for new ones.
Chronic depression in humans (designed agencies wouldn’t tend to have such an obvious bug) kinda feels like when the behavior of searching for new strategies has itself has been caught within depression’s scope as an invalidated strategy.
Searching anew is what’s supposed to happen (I had a post called “Good Gloom” that drew an analogy to turning all of the lights out in a town so that you’ll be able to see the light pollution of a new town, and set out for it), but if the depression is blocking that too, you stop moving.
Wild Speculative Civics: What if we found ways of reliably detecting when tragedies of the commons have occurred, then artificially increased their costs (charging enormous fines) to anyone who might have participated in creating them, until it’s not even individually rational to contribute to them any more?
That sounds like punishing any usage of common resources which is likely undesireable.
Good policy to manage individual commons requires to think through how their usage is best managed. Elinor Ostrom did a lot of research into what works for setting up good systems.
Sounds like auctioning the usage of the common.
I can imagine a few technical problems, like determining what level of usage is optimal (you don’t want people to overfish the lake, but you don’t know exactly how many fish are there), or the costs of policing. But it would be possible to propose a few dozen situations where this strategy could be used, and address these issues individually; and then perhaps only use the strategy in some of them. Or perhaps by examining individual specific cases, we would discover a common pattern why this doesn’t work.
Until you learn FDT, you cannot see the difference between faith and idealism, nor the difference between pragmatism and cynicism. The tension between idealism and pragmatism genuinely cannot be managed gracefully without FDT, it defines their narrow synthesis.
More should be written about this, because cynicism, idealism afflicts many.
Have you ever seen someone express stern (but valid, actionable) criticisms, conveyed with actual anger, towards an organization, and then been hired, to implement their reforms?
If that has never happened, is there a reasonable explanation for that or is it just, as it appears, almost all orgs are run by and infested with narcissism? (a culture of undervaluing criticism and not protecting critics)
I think that’s happened with a friend of mine who went into social work. They are in a field where that makes sense.