I often have the experience of being in the middle of a discussion and wanting to reference some simple but important idea / point, but there doesn’t exist any such thing. Often my reaction is “if only there was time to write an LW post that I can then link to in the future”. So far I’ve just been letting these ideas be forgotten, because it would be Yet Another Thing To Keep Track Of. I’m now going to experiment with making subcomments here simply collecting the ideas; perhaps other people will write posts about them at some point, if they’re even understandable.
Let’s say you’re trying to develop some novel true knowledge about some domain. For example, maybe you want to figure out what the effect of a maximum wage law would be, or whether AI takeoff will be continuous or discontinuous. How likely is it that your answer to the question is actually true?
(I’m assuming here that you can’t defer to other people on this claim; nobody else in the world has tried to seriously tackle the question, though they may have tackled somewhat related things, or developed more basic knowledge in the domain that you can leverage.)
First, you might think that the probability of your claims being true is linear in the number of insights you have, with some soft minimum needed before you really have any hope of being better than random (e.g. for maximum wage, you probably have ~no hope of doing better than random without Econ 101 knowledge), and some soft maximum where you almost certainly have the truth. This suggests that P(true) is a logistic function of the number of insights.
Further, you might expect that for every doubling of time you spend, you get a constant number of new insights (the logarithmic returns are because you have diminishing marginal returns on time, since you are always picking the low-hanging fruit first). So then P(true) is logistic in terms of log(time spent). And in particular, there is some soft minimum of time spent before you have much hope of doing better than random.
This soft minimum on time is going to depend on a bunch of things—how “hard” or “complex” or “high-dimensional” the domain is, how smart / knowledgeable you are, how much empirical data you have, etc. But mostly my point is that these soft minimums exist.
A common pattern in my experience on LessWrong is that people will take some domain that I think is hard / complex / high-dimensional, and will then make a claim about it based on some pretty simple argument. In this case my response is usually “idk, that argument seems directionally right, but who knows, I could see there being things that make it wrong”, without being able to point to any such thing (because I also have spent barely any time thinking about the domain). Perhaps a better way of saying it would be “I think you need to have thought about this for more time than you have before I expect you to do better than random”.
Sometimes people say “look at these past accidents; in these cases there were giant bureaucracies that didn’t care about safety at all, therefore we should be pessimistic about about AI safety”. I think this is backwards, and that you should actually conclude the reverse: this is evidence that problems tend to be easy, and therefore we should be optimistic about AI safety.
It’s easiest to see with a Bayesian treatment. Let’s say we start completely uncertain about what fraction of people will care about problems, i.e. uniform distribution over [0, 100]%. In what worlds do I expect to see accidents where giant bureaucracies don’t care about safety? Almost all of them—even if 90% of people care about safety, there will still be some cases where people didn’t care and accidents happened; and of course we’d hear about them if so (and not hear about the cases where accidents didn’t happen). You can get a strong update against 99.9999% and higher, but by the time you’re at 90% the update seems pretty weak. Given how much selection there is, I think even the update against 99% is relatively weak. So really you just don’t learn much about how careful people will be by looking at our accident track record (unless you can also quantify the denominator of how many “potential accidents” there could have been).
However, it feels pretty notable to me that the vast majority of accidents I hear about in detail are ones where it seems like there were a bunch of obvious mistakes and the accidents would have been prevented had there been a decision-maker who cared (enough) about safety. And unlike the previous paragraph, I do expect to hear about accidents that we couldn’t have prevented, so I don’t have to worry about selection bias. So it seems like I should conclude that usually problems are pretty easy, and “all we have to do” is make sure people care. (One counterargument is that problems look obvious only in hindsight; at the time the obvious mistakes may not have been obvious.)
Examples of accidents that fit this pattern: the Challenger crash, the Boeing 737-MAX issues, everything in Engineering a Safer World, though admittedly the latter category suffers from some selection bias.
You’ve heard of crucial considerations, but have you heard of red herring considerations?
These are considerations that intuitively sound like they could matter a whole lot, but actually no matter how the consideration turns out it doesn’t affect anything decision-relevant.
To solve a problem quickly, it’s important to identify red herring considerations before wasting a bunch of time on them. Sometimes you can even start outlining solutions that turn a bunch of seemingly-crucial considerations into red herring considerations.
For example, it might seem like “what is the right system of ethics” is a crucial consideration for AI alignment (after all, you need to know ethics to write down a utility function), but once you decide to instead aim to design algorithms that allow you to build AI systems for any task you have in mind, that turns into a red herring consideration.
Here’s an example where I argue that, for a specific question, anthropics is a red herring consideration (thus avoiding the question of whether to use SSA or SIA).
“Burden of proof” is a bad framing for epistemics. It is not incumbent on others to provide exactly the sequence of arguments to make you believe their claim; your job is to figure out whether the claim is true or not. Whether the other person has given good arguments for the claim does not usually have much bearing on whether the claim is true or not.
Similarly, don’t say “I haven’t seen this justified, so I don’t believe it”; say “I don’t believe it, and I haven’t seen it justified” (unless you are specifically relying on absence of evidence being evidence of absence, which you usually should not be, in the contexts that I see people doing this).
I’m not 100% sure this needs to be much longer. It might actually be good to just make this a top-level post so you can link to it when you want, and maybe specifically note that if people have specific confusions/complaints/arguments that they don’t think the post addresses, you’ll update the post to address those as they come up?
(Maybe caveating the whole post under “this is not currently well argued, but I wanted to get the ball rolling on having some kind of link”)
That said, my main counterargument is: “Sometimes people are trying to change the status quo of norms/laws/etc. It’s not necessarily possible to review every single claim anyone makes, and it is reasonable to filter your attention to ‘claims that have been reasonably well argued.’”
I think ‘burden of proof’ isn’t quite the right frame but there is something there that still seems important. I think the bad thing comes from distinguishing epistemics vs Overton-norm-fighting, which are in fact separate.
maybe specifically note that if people have specific confusions/complaints/arguments that they don’t think the post addresses, you’ll update the post to address those as they come up?
I don’t really want this responsibility, which is part of why I’m doing all of these on the shortform. I’m happy for you to copy it into a top-level post of your own if you want.
Sometimes people are trying to change the status quo of norms/laws/etc. It’s not necessarily possible to review every single claim anyone makes, and it is reasonable to filter your attention to ’claims that have been reasonably well argued.
I agree this makes sense, but then say “I’m not looking into this because it hasn’t been well argued (and my time/attention is limited)”, rather than “I don’t believe this because it hasn’t been well argued”.
In that example, X is “AI will not take over the world”, so Y makes X more likely. So if someone comes to me and says “If we use <technique>, then AI will be safe”, I might respond, “well, if we were using your technique, and we assume that AI does not have the ability to take over the world during training, it seems like the AI might still take over the world at deployment because <reason>”.
I don’t think this is a great example, it just happens to be the one I was using at the time, and I wanted to write it down. I’m explicitly trying for this to be a low-effort thing, so I’m not going to try to write more examples now.
EDIT: Actually, the double descent comment below has a similar structure, where X = “double descent occurs because we first fix bad errors and then regularize”, and Y = “we’re using an MLP / CNN with relu activations and vanilla gradient descent”.
In fact, the AUP power comment does this too, where X = “we can penalize power by penalizing the ability to gain reward”, and Y = “the environment is deterministic, has a true noop action, and has a state-based reward”.
Maybe another way to say this is:
I endorse applying the “X proves too much” argument even to impossible scenarios, as long as the assumptions underlying the impossible scenarios have nothing to do with X. (Note this is not the case in formal logic, where if you start with an impossible scenario you can prove anything, and so you can never apply an “X proves too much” argument to an impossible scenario.)
When you make an argument about a person or group of people, often a useful thought process is “can I apply this argument to myself or a group that includes me? If this isn’t a type error, but I disagree with the conclusion, what’s the difference between me and them that makes the argument apply to them but not me? How convinced I am that they actually differ from me on this axis?”
“Minimize AI risk” is not the same thing as “maximize the chance that we are maximally confident that the AI is safe”. (Somewhat related comment thread.)
An incentive for property X (for humans) usually functions via selection, not via behavior change. A couple of consequences:
In small populations, even strong incentives for X may not get you much more of X, since there isn’t a large enough population for there to be much deviation on X to select on.
It’s pretty pointless to tell individual people to “buck the incentives”, even if they are principled people who try to avoid doing bad things, if they take your advice they probably just get selected against.
Let’s say we’re talking about something complicated. Assume that any proposition about the complicated thing can be reformulated as a series of conjunctions.
Suppose Alice thinks P with 90% confidence (and therefore not-P with 10% confidence). Here’s a fully general counterargument that Alice is wrong:
Decompose P into a series of conjunctions Q1, Q2, … Qn, with n > 10. (You can first decompose not-P into R1 and R2, then decompose R1 further, and decompose R2 further, etc.)
Ask Alice to estimate P(Qk | Q1, Q2, … Q{k-1}) for all k.
At least one of these must be over 99% (if we have n = 11 and they were all 99%, then probability of P would be (0.99 ^ 11) = 89.5% which contradicts the original 90%).
Argue that Alice can’t possibly have enough knowledge to place under 1% on the negation of the statement.
----
What’s the upshot? When two people disagree on a complicated claim, decomposing the question is only a good move when both people think that is the right way to carve up the question. Most of the disagreement is likely in how to carve up the claim in the first place.
In general, evaluate the credibility of experts on the decisions they make or recommend, not on the beliefs they espouse. The selection in our world is based much more on outcomes of decisions than on calibration of beliefs, so you should expect experts to be way better on the former than on the latter.
By “selection”, I mean both selection pressures generated by humans, e.g. which doctors gain the most reputation, and selection pressures generated by nature, e.g. most people know how to catch a ball even though most people would get conceptual physics questions wrong.
Similarly, trust decisions / recommendations given by experts more than the beliefs and justifications for those recommendations.
1. Observe the world and form some gears-y model of underlying low-level factors, and then make predictions by “rolling out” that model
2. Observe relatively stable high-level features of the world, predict that those will continue as is, and make inferences about low-level factors conditioned on those predictions.
I expect that most intellectual progress is accomplished by people with lots of detailed knowledge and expertise in an area doing option 1.
However, I expect that in the absence of detailed expertise, you will do much better at predicting the world by using option 2.
I think many people on LW tend to use option 1 almost always and my “deference” to option 2 in the absence of expertise is what leads to disagreements like How good is humanity at coordination?
Conversely, I think many of the most prominent EAs who are skeptical of AI risk are using option 2 in a situation where I can use option 1 (and I think they can defer to people who can use option 1).
Yeah, that sounds about right to me. I think in terms of this framework my claim is primarily “for reasonably complex systems, if you try to do 2 without expertise, you will fail, but you may not realize you have failed”.
I’m also noticing I mean something slightly different by “expertise” than is typically meant. My intended meaning of “expertise” is more like “you have lots of data and observations about the system”, e.g. I think LW self-help stuff is reasonably likely to work (for the LW audience) because people have lots of detailed knowledge and observations about themselves and their friends.
Yeah, I think so? I have a vague sense that there are slight differences but I certainly haven’t explained them here.
EDIT: Also, I think a major point I would want to make if I wrote this post is that you will almost certainly be quite wrong if you use option 1 without expertise, in a way that other people without expertise won’t be able to identify, because there are far more ways the world can be than you (or others) will have thought about when making your gears-y model.
Sounds like you probably disagree with the (exaggeratedly stated) point made here then, yeah?
(My own take is the cop-out-like, “it depends”. I think how much you ought to defer to experts varies a lot based on what the topic is, what the specific question is, details of your own personal characteristics, how much thought you’ve put into it, etc.)
Sounds like you probably disagree with the (exaggeratedly stated) point made here then, yeah?
Correct.
My own take is the cop-out-like, “it depends”. I think how much you ought to defer to experts varies a lot based on what the topic is, what the specific question is, details of your own personal characteristics, how much thought you’ve put into it, etc.
I didn’t say you should defer to experts, just that if you try to build gears-y models you’ll be wrong. It’s totally possible that there’s no way to get to reliably correct answers and you instead want decisions that are good regardless of what the answer is.
It’s totally possible that there’s no way to get to reliably correct answers and you instead want decisions that are good regardless of what the answer is.
Intellectual progress requires points with nuance. However, on online discussion forums (including LW, AIAF, EA Forum), people seem to frequently lose sight of the nuanced point being made—rather than thinking of a comment thread as “this is trying to ascertain whether X is true”, they seem to instead read the comments, perform some sort of inference over what the author must believe if that comment were written in isolation, and then respond to that model of beliefs. This makes it hard to have nuance without adding a ton of clarification and qualifiers everywhere.
I find that similar dynamics happen in group conversations, and to some extent even in one-on-one conversations (though much less so).
The simple response to the unilateralist curse under the standard setting is to aggregate opinions amongst the people in the reference class, and then do the majority vote.
A particular flawed response is to look for N opinions that say “intervening is net negative” and intervene iff you cannot find that many opinions. This sacrifices value and induces a new unilateralist curse on people who think the intervention is negative. (Example.)
However, the hardest thing about the unilateralist curse is figuring out how to define the reference class in the first place.
I didn’t get it… is the problem with the “look for N opinions” response that you aren’t computing the denominator (|”intervening is positive”| + |”intervening is negative”|)?
Yes, that’s the problem. In this situation, if N << population / 2, you are likely to not intervene even when the intervention is net positive; if N >> population / 2, you are likely to intervene even when the intervention is net negative.
(This is under the standard model of a one-shot decision where each participant gets a noisy observation of the true value with the noise being iid Gaussians with mean zero.)
Under the standard setting, the optimizer’s curse only changes your naive estimate of the EV of the action you choose. It does not change the naive decision you make. So, it is not valid to use the optimizer’s curse as a critique of people who use EV calculations to make decisions, but it is valid to use it as a critique of people who make claims about the EV calculations of their most preferred outcome (if they don’t already account for it).
At any given time, is there anything especially wrong about using citation count (weighted by the weightings of other paper’s citation count) as a rough proxy for “what are the most important papers, and/or best authors, weighted?”
My sense is the thing that’s bad about this is that it creates an easy goodhart metric. I can imagine worlds where it’s already so thoroughly goodharted that it doesn’t signal anything anymore. If that’s the case, can you get around that by grounding it out in some number of trusted authors, and purging obviously fraudulent authors from the system?
I’m asking from the lens of “I’d like to have some kind barometer for which scientific papers (or, also, LW posts) are the best. And this just… actually seems pretty good, at least if you were only using it as a one-time-check.”
It depends what you mean by “rough proxy”, and whether you’re applying it to scientific papers (where Goodhart has been out in force for decades, so a one-time check is off the table) or to LessWrong posts (where citation-count has never been something people cared about). Most things have zero citations, and this is indeed a negative quality signal. But after you get to stuff that’s cited at all, citation count is mainly determined by the type and SEO of a paper, rather than its quality. Eg this paper. Citations also don’t distinguish building upon something from criticizing it. That’s much worse in the Goodhart arena than the one-time arena, but still pretty bad in the one-shot case.
In a given (sub)field, the highest-cited papers tend to be those which introduced or substantially improved on a key idea/result/concept; so they’re important in that sense. If you’re looking for the best introduction though that will often be a textbook, and there might be important caveats or limitations in a later and less-cited paper.
I’ve also had a problem where a few highly cited papers propose $approach, many papers apply or puport to extend it, and then eventually someone does a well-powered study checking whether $approach actually works. Either way that’s an important paper, but they tend to be under-cited either because either the results are “obvious” (and usually a small effect) or the field of $approach studies shrinks considerably.
It’s an extremely goodhartable metric but perhaps the best we have for papers; for authors I tend to ask “does this person have good taste in problems (important+tractable), and are their methods appropriate to the task?”.
On October 26, 2020, I submitted a security vulnerability report to the Facebook bug bounty program. The submission was rejected as a duplicate. As of today (April 14), it is still not fixed. I just resubmitted, since it seems to have fallen through the cracks or something. However, I consider all my responsible disclosure responsibilities to be discharged.
Once an Oculus Quest or Oculus Quest 2 is logged in to a Facebook account, its login can’t be revoked. There is login-token revocation UI in Facebook’s Settings>Security and Login menu, but changing the account password and revoking the login there does not work.
One practical impact of this is that if your Facebook account is ever compromised, and the attacker uses this vulnerability, they have permanent access.
The other practical impact is that if someone has unsupervised access to your unlocked Quest headset, and they use the built-in web browser to go to facebook.com, they have full access to your Facebook account, including Messenger, without having to do anything special at all. This means that if you’ve ever made a confidentiality agreement regarding something you discussed on Facebook Messenger, you probably can’t lend your headset to anyone, ever.
Additionally, the lock-screen on the Oculus Quest 2 does not have a strict enough rate limit; it gives unlimited tries at 2/minute, so trying all lock-screen combinations takes approximately 35 days. This can be done without network access, and can be automated with some effort. So if someone steals a *locked* Oculus Quest 2, they can also use that to break into your Facebook account. There is almost certainly a much faster way to do this involving disassembling the device, but this is bad enough.
Is your logic that releasing this heinous volun into the public is more likely to pressure FB to do something about this? Because if so, I’m not sure that LW is a forum with enough public spotlight to generate pressure. OTOH, I imagine some percentage of readers here aren’t well-aligned but are looking for informational edge, in which case it’s possible this does more harm than good?
I’m not super-confident in this model—eg, it also seems entirely possible to me that lots of FB security engineers read the site and one or more will be shouting ZOMG! any moment over this..
I’m posting here (cross-posted with my FB wall and Twitter) mostly to vent about it, and to warn people that sharing VR headsets has infosec implications they may not have been aware of. I don’t think this comment will have much effect on Facebook’s actions.
Although with a sufficiently complex (or specific) compiler or processor, any arbitrary stream of information can be made into a quine.
You can embed information in either of instructions/substrate, or compiler/processor. Quines usually seem limited to describing instructions/substrate, but that’s not the only place information can be drawn from. I’ve kinda come to think of it as 2-term operation (ex: “this bit of quine code is a quine wrt python”).
(More physical quines: Ink scribbles on a sheet of paper are a quine wrt a copy-machine. At about the far-end of “replicative complexity gets stored in the processor,” you have the “activated button” of a button-making machine (which activates with a button, and outputs buttons in an active state). I think the “activated button” here is either a quine, or almost a quine.)
The cool thing about life is that it it is both a quine, and its own processor. (And also, sorta its own power source.)
I find it simpler to call systems like this “living” (at least while active/functional), since they’re meaningfully more than just quines.
Viruses are definitely quines, though. Viruses and plasmids are non-living quines that compile and run on living biological systems.
Physics: Feels close. Hm… biological life as a self-compiler on a physics substrate?
DNA or gametes seem really close to a “quine” for this: plug it into the right part of an active compiler, and it outputs many instances of its own code + a customized compiler. Although it crashes/gets rejected if the compiler is too different (ex: plant & animal have different regulatory markers & different sugar-related protein modifications).
I don’t have a fixed word for the “custom compiler” thing yet (“optimized compiler”? “coopted compiler”? “spawner”? “Q-spawner”?). I have seen analogous stuff in other places, and I’m tempted to call it a somewhat common pattern, though. (ex: vertically-transmitted CS compiler corruption, or viruses producing tumor micro-environments that are more favorable to replicating the virus)
An unusual mathematical-leaning definition of a living thing.
(Or, to be more precise… a potential living immortal? A replicon? Whatever.)
A self-replicating physical entity with...
3 Core Components:
Quine: Contains the code to produce itself
Parser: A parser for that code
Code includes instructions for the parser; ideally, compressed instructions
Power: Actively running (probably on some substrate)
Not actively running is death, although for some the death is temporary.
Access to resources may be a subcomponent of this?
Additional components:
Substrate: A material that can be converted into more self
Possibly a multi-step process
Access to resources is probably a component of this
Translator: Converts quine into power, or vice-versa
Not always necessary; sometimes a quine is held as power, not substrate
Parser and Code: the information can actually be stored on either; you can extract the correct complex signal from complete randomness using an arbitrarily-complex parser chosen for that purpose. There are analogies that can be drawn in the other direction, too: a fairly dumb parser can make fairly complicated things, given enough instructions. (Something something Turing Machines)
Ideally, though, they’re well-paired and the compression method is something sensible, to reduce complexity.
A quine by itself has only a partial life; it is just its own code, it requires a parser to replicate.
(If you allow for arbitrarily complex parsers, any code could be a quine, if you were willing to search for the right parser.)
Compilers are… parsers? (Or translators?)
It is possible for what was “code” information to be embedded into the parser. I think this is part of what happens when you completely integrate parts in IFS.
Examples
Example replicons: A bacterium, a cell, a clonal organism, a pair of opposite-sexed humans (but not a single), self-contained repeating Game of Life automata, the eventual goal of RepRap (a 3D printer fully producing a copy of itself)
Viruses: Sometimes quines, sometimes pre-quine and translator
The distinctive thing about Lisp is that its core is a language defined by writing an interpreter in itself. It wasn’t originally intended as a programming language in the ordinary sense. It was meant to be a formal model of computation, an alternative to the Turing machine. If you want to write an interpreter for a language in itself, what’s the minimum set of predefined operators you need? The Lisp that John McCarthy invented, or more accurately discovered, is an answer to that question.
(Complex multi-step life-cycle in the T5 case, though?. The quine produced a bare-bones quine-compiling “spawner” when it interacts with the wild-type, and then replicates on that. Almost analogous to CS viruses that infect the compiler, and any future compiler compiled by that compiler.)
Process is Art: Is art that demonstrates how to craft the art, a quine on a human compiler?
Some connection to: Compartmentalization (Black Box Testing, unit tests, separation of software components) and the “swappability” of keeping things general and non-embedded, with clearly-marked and limited interface-surfaces (APIs). Generates pressure against having “arbitrary compilers” in practice.
The coronavirus response has been so bad I no longer doubt many Sci-Fi premises. Before I often said to myself “you have tech that can do X and you still have problem Y. Ridiculous!”. But apparently, we can make a coronavirus vaccine in two days. But we still had over a year of lockdowns and millions of deaths. One in six hundred people in the USA have died of the virus but we blocked a vaccine because one in a million people who take it MIGHT develop treatable blood clots.
My standards for ‘realistic’ dysfunction have gotten a lot lower.
The heads of Government and the FDA don’t work like you do. Who knows the incentives that they have? It’s entirely possible that for them this is just a political play(moving chess pieces) that make sense for them while the well-being of the people take secondary place to a particular political move.
This wouldn’t be the first thing that this happens in any kind of government agency, but, at any rate, it’s too early to be skeptical. We need to see how this unfolds, may be the pausing don’t last as much.
I remember Yudkowsky asking for a realistic explanation for why the Empire in Star Wars is stuck in an equilibrium where it builds destroyable gigantic weapons.
I once read a comment somewhere that Paul Graham is not a rationalist, though he does share some traits, like writing a lot of self-improvement advice. From what I can tell Paul himself considers himself a builder; a builder of code and companies. But there is some overlap with rationalists, Paul Graham mostly builds information systems. (He is somewhat disdainful of hardware, which I consider the real engineering, but I am a physicist.) Rationalists are focussed on improving their own intelligence and other forms of intelligence. So both spend a great deal of time building and improving intelligent information systems, as well as improving their own mind, but for different reasons. For one the goal is merely to build, and self-improvement is a method, for the other self-improvement is the goal and building is a method. Well, and for some the goal is to build a self-improving intelligence (that doesn’t wipe us out).
Builders and rationalists. Experimentalists and theoretical empiricists. I suppose they work well together.
This is a really good comment. If you care to know more about his thinking, he has a book called, “hackers and painters” which I think sums up very well his views. But yes, it’s a redistribution of wealth and power from strong people and bureaucrats to what he calls “nerds” as in people who know technology deeply and actually build things.
The idea of instrumental rationality touches at the edges of builders, and you need to if you ever so desire to act in the world.
Note that for hardware, the problems are that you need a minimum instruction set in order to make a computer work. So long as you at least implement the minimum instruction set, and for all supported instructions perform the instructions (which are all in similar classes of functions) bit for bit correctly, done. It’s ever more difficult to make a faster computer for a similar manufacturing cost and power consumption because the physics keep getting harder.
But it is in some ways a “solved problem”. Whether a given instance of a computer is ‘better’ is a measurable parameter, the hardware people try things, the systems and software engineers adopt the next chip if it meets their definition of ‘better’.
So yeah if we want to see new forms of applications that aren’t in the same class as what we have already seen—that’s a software and math problem.
I’m not sure I follow. Whether it’s the evolving configuration of atoms or bits, both can lead to new applications. The main difference to me seems that today it is typically harder to configure atoms than bits, but perhaps that’s just by our own design of the atoms underlying the bits? If some desired information system would require a specific atomic configuration, then you’d be hardware constrained again.
Let’s say that in order to build AGI we find out you actually need super power efficient computronium, and silicon can’t do that, you need carbon. Now it’s no longer a solved hardware problem, you are going to have to invest massively in carbon based computing. Paul and the rationalists are stuck waiting for the hardware engineers.
I am saying below a certain level of abstraction it becomes a solved problem in that you precisely have defined what correctness is and have fully represented your system. And you can trivially check any output and validate it versus a model.
The reason software fails constantly is we don’t have a good definition that can be checked by computer of what correctness means. Software Unit tests help but are not nearly as reliable as tests for silicon correctness. Moreover software just ends up being absurdly more complex than hardware and ai systems are worse.
Part of it is “unique complexity”. A big hardware system is millions of copies of the same repeating element. And locality matters—an element cannot affect another one far away unless a wire connects them. A big software system is millions of copies of often duplicated and nested and invisibly coupled code.
Years after I first thought of it, I continue to think that this chain reaction is the core of what it means for something to be an agent, AND why agency is such a big deal, the sort of thing we should expect to arise and outcompete non-agents. Here’s a diagram:
Roughly, plans are necessary for generalizing to new situations, for being competitive in contests for which there hasn’t been time for natural selection to do lots of optimization of policies. But plans are only as good as the knowledge they are based on. And knowledge doesn’t come a priori; it needs to be learned from data. And, crucially, data is of varying quality, because it’s almost always irrelevant/unimportant. High-quality data, the kind that gives you useful knowledge, is hard to come by. Indeed, you may need to make a plan for how to get it. (Or more generally, being better at making plans makes you better at getting higher-quality data, which makes you more knowledgeable, which makes your plans better.)
I think it’s probably even simpler than that: feedback loops are the minimum viable agent, i.e. a thermostat is the simplest kind of agent possible. Stuff like knowledge and planning are elaborations on the simple theme of the negative feedback circuit.
I disagree; I think we go astray by counting things like thermostats as agents. I’m proposing that this particular feedback loop I diagrammed is really important, a much more interesting phenomenon to study than the more general category of feedback loop that includes thermostats.
An aircraft carrier costs $13 billion. An anti-ship cruise missile costs $2 million. Few surface warships survived the first day of the Singularity War.
A cruise missile is a complex machine, guided by sensitive electronics. Semiconductor fabricators are even more complex machines. Few semiconductor factories survived the nuclear retaliation.
A B-52 Stratofortress is a simpler machine.
Robert (Bob) Manchester’s bomber flew west from Boeing Field. The crew disassembled their landing gear and dropped it in the Pacific Ocean. The staticy voice of Mark Campbell, Leader of the Human Resistance, radioed into Robert’s headset. Robert could barely hear it over the noise of the engines. He turned the volume up. It would damage his hearing but that didn’t matter anymore. The attack wouldn’t save America. Nothing could at this point. But the attack might buy time to launch a few extra von Neumann probes.
The squadron flew over miles after miles of automated factories. What was once called Tianjin was now just Sector 153. The first few flak cannons felt perfunctory. The anti-air fire increased as they drew closer to enemy network hub. Robert dropped the bombs. The pilot, Peter Garcia, looked for a target to kamikaze.
They drew closer to the ground. Robert Manchester looked out the window. He wondered why the Eurasian AI had chosen to structure its industry around such humanoid robots.
In response to my earlier post about Myers-Briggs (where I suggested a more detailed notation for more nuanced communication about personality types), it was pointed out that there is some correlation between the four traits being measured, and this makes the system communicate less information on average than it otherwise would (The traditional notation would communicate 4 bits, my version would communicate ~9.2 if there was no correlation).
I do object to the characterization that it all measures “the same thing”, since none of the traits perfectly predicts the others, and all 16 of the traditional configurations have people they describe (though some are more common than others); but I do think it makes sense to try to disentangle things—if the I / E scale is correlated with the J / P scale, we can subtract some amount of J points for more introverted people, and add J points for the extroverts, so that an introvert needs to be more “judgmental” to be considered “J” than an equally judgmental extrovert, with the goal being that 50% of extroverts will be J, 50% P, and have a similar 50-50 split for introverts.
By adjusting for these correlations across all pairs, we can more finely detect and communicate the underlying traits that cause “Judgement” and “Perception” that aren’t just a result of a person being more extroverted (a disposition that rewards those who are best able to use their intuition) or introverted (which often leads to pursuits that require careful thinking distanced from our whims).
I have a potential category of questions that could fit on Metaculus and work as an “AGI fire alarm.” The questions are of the format “After an AI system achieves task x, how many years will it take for world output to double?”
I don’t mind jumping through a few extra hoops in order to access a website idiosyncratically. But sometimes the process feels overly sectarian.
I was trying out the Tencent cloud without using Tor when I got a CAPTCHA. Sure, whatever. They verified my email. That’s normal. Then they wanted to verify my phone number. Okay. (Using phone numbers to verify accounts is standard practice for Chinese companies.) Then they required me to verify my credit card with a nominal $1 charge. I can understand their wanting to take extra care when it comes to processing international transactions. Then they required me to send a photo of my driver’s licence. Fine. Then they required 24 hours to process my application. Okay. Then they rejected my application. I wonder if that’s what the Internet feels like everyday to non-Americans.
I often anonymize my traffic with Tor. Sometimes I’ll end up on the French or German Google, which helps remind me that the Internet I see everyday is not the Internet everyone else sees.
Other people use Tor too, which is necessary to anonymize my traffic. Some Tor users aren’t really people. They’re bots. By accessing access the Internet from the same Tor exit relays as these bots, websites often suspect me of being a bot.
I encounter many pages like this.
This is a Russian CAPTCHA.
Prove you’re human by typing “вчepaшний garden”. Maybe I should write some OCR software to make proving my humanity less inconvenient.
Another time I had to identify which Chinese characters were written incorrectly.
The most frustrating CAPTCHAs require me to annotate images for self-driving cars. I do not mind annotating images of self-driving cars. I do mind, after having spent several minutes annotating images of self-driving cards, getting rejected based off of a traffic analysis of my IP address.
I do mind, after having spent several minutes annotating images of self-driving cards
I think it’s worst when you have edge cases like the Google Captcha that shows 16 tiles and you have to choose which tiles contain the item they are looking for and some of the tails contain it only a little bit on the edge.
I think you can steelman Ben Goertzel-style worries about near-term amoral applications of AI being bad “formative influences” on AGI, but mostly under a continuous takeoff model of the world. If AGI is a continuous development of earlier systems, then maybe it shares some datasets and learned models with earlier AI projects, and definitely it shares the broader ecosystems of tools, dataset-gathering methodologies, model-evaluating paradigms, and institutional knowledge on the part of the developers. If the ecosystem in which this thing “grows up” is one that has previously been optimized for marketing, or military applications, or what have you, this is going to have ramifications in how the first AGI projects are designed and what they get exposed to. The more continuous you think the development is going to be, the more this can be intervened on by trying to make sure that AI is pro-social even in the short term.
Random thought: if you have a big enough compost pile, would it spontaneously break into flames due to the heat generated by the bioprocesses that occur therein? If so, at what size would it burst into flames? Surely it could happen before it reached the size of the sun, even ignoring gravitational effects.
(Just pondering out loud, not really asking unless someone really wants to answer)
For a value of “break into flames” that matches damp and poorly-oxygenated fuel, yep! This case in Australia is illustrative; you tend to get a lot of nasty smoke rather than a nice campfire vibe.
You’d have to mismanage a household-scale compost pile very badly before it spontaneously combusts, but it’s a known and common failure mode for commercial-scale operations above a few tons. Specific details about when depend a great deal on the composition of the pile; with nitrate filmstock it was possible with as little as a few grams.
The child-in-a-pond thought experiment is weird, because people use it in ways it clearly doesn’t work for (especially in arguing for effective altruism).
For example, it observes you would be altruistic in a near situation with the drowning child, and then assumes that you ought to care about people far away as much as people near you. People usually don’t really argue against this second step, but very much could. But the thought experiment makes no justification for that extension of the circle of moral concern, it just assumes it.
Similarly, it says nothing about how effectively you ought to use your resources, only that you probably ought to be more altruistic in a stranger-encompassing way.
But not only does this thought experiment not argue for the things people usually use it for, it’s also not good for arguing that you ought to be more altruistic!
Underlying it is a theme that plays a role in many thought experiments in ethics: they appeal to game-theoretic intuition for useful social strategies, but say nothing of what these strategies are useful for.
Here, if people catch you letting a child drown in a pond while standing idly, you’re probably going to be excluded from many communities or even punished. And this schema occurs very often! Unwilling organ donors, trolley problems, and violinists.
Bottom line: Don’t use the drowning child argument to argue for effective altruism.
I don’t know, I think it’s a pretty decent argument. I agree it sometimes gets overused, but I do think given it’s assumptions “you care about people far away as much as people closeby” and “there are lots of people far away you can help much more than people close by” and “here is a situation where you would help someone closeby, so you might also want to help the people far away in the same way” are all part of a totally valid logical chain of inference that seems useful to have in discussions on ethics.
Like, you don’t need to take it to an extreme, but it seems locally valid and totally fine to use, even if not all the assumptions that make it locally valid are always fully explicated.
Right, my gripe with the argument is that these first two assumptions are almost always unstated, and most of the time when people use the argument, they “trick” people into agreeing with assumption one.
(for the record, I think the first premise is true)
On self-reflection, I just plain don’t care about people far away as much as those near to me. Parts of me think I should, but other parts aren’t swayed. The fact that a lot of the motivating stories for EA don’t address this at all is one of the reasons I don’t listen very closely to EA advice.
I am (somewhat) an altruist. And I strive to be effective at everything I undertake. But I’m not an EA, and I don’t really understand those who are.
Yep, that’s fine. I am not a moral prescriptivist who tells you what you have to care about.
I do think that you are probably going to change your mind on this at some point in the next millennium if we ever get to live that long, and I do have a bunch of arguments that feel relevant, but I don’t think it’s completely implausible you really don’t care.
I do think that not caring about how people are far away is pretty common, and building EA on that assumption seems fine. Not all clubs and institutions need to be justifiable to everyone.
Last month, I wrote a post here titled “Even Inflationary Currencies Should Have Fixed Total Supply”, which wasn’t well-received. One problem was that the point I argued for wasn’t exactly the same as what the title stated: I supported both currencies with fixed total supply, and currencies that instead choose to scale supply proportional to the amount of value in the currency’s ecosystem, and many people got confused and put off by the disparity between the title and my actual thesis; indeed, one of the most common critiques in the comments was a reiteration of a point I had already made in the original post.
Zvi helpfully pointed out another effect that nominal inflation has that serves as part of the reason inflation is implemented the way it is, that I wasn’t previously aware of, namely that nominal inflation induces people to accept worsening prices they psychologically would otherwise resist. While I feel intentionally invoking this effect flirts with the boundary of dishonesty, I do recognize the power and practical benefits of this effect.
All that said, I do stand by the core of my original thesis: nominal inflation is a source of much confusion for normal people, and makes the information provided by price signals less easily legible over long spans of time, which is problematic. Even if the day-to-day currency continues to nominally inflate like things are now, it would be stupid not to coordinate around a standard stable unit of value (like [Year XXXX] Dollars, except without having to explicitly name a specific year as the basis of reference; and maybe don’t call it dollars, to make it clear that the unit isn’t fluidly under the control of some organization)
You set a schedule of times you want to be productive, and a frequency, and then it rings you at random (but with that frequency) to bug you with questions like:
--Are you “in the zone” right now? [Y] [N]
--(if no) What are you doing? [text box] [common answer] [ common answer] [...]
The point is to cheaply collect data about when you are most productive and what your main time-wasters are, while also giving you gentle nudges to stop procrastinating/browsing/daydream/doomscrolling/working-sluggishly, take a deep breath, reconsider your priorities for the day, and start afresh.
Probably wouldn’t work for most people but it feels like it might for me.
This format of data collection is called “experience sampling”. I suspect there might be already made solutions.
Would you pay for such an app? If so, how much?
Also looks like your crux is actually becoming more productive (i.e. experience sampling is a just a mean to reach that). Perhaps just understanding your motivation better would help (basically http://mindingourway.com/guilt/)?
More surprised than perhaps I should be that people take up tags right away after creating them. I created the IFS tag just a few days ago after noticing it didn’t exist but wanted to link it and I added the first ~5 posts that came up if I searched for “internal family systems”. It now has quite a few more posts tagged with it that I didn’t add. Super cool to see the system working in real time!
Anna writes about bucket errors . Attempted summary: sometimes two facts are mentally tracked by only one variable; in that case, correctly updating the belief about one fact can also incorrectly update the belief about the other fact, so it is sometimes epistemic to flinch away from the truth of the first fact (until you can create more variables to track the facts separately).
There’s a conjugate error: two actions are bound together in one “lever”.
For example, I want to clean my messy room. But somehow it feels pointless / tiring, even before I’ve started. If I just started cleaning anyway, I’d get bogged down in some corner, trying to make a bunch of decisions about where exactly to put lots of futzy random objects, tiring myself out and leaving my room still annoyingly cluttered. It’s not that there’s a necessary connection between cleaning my room and futzing around inefficiently; it’s that the only lever I have right now that activates the “clean room” action also activates the “futz interminably” action.
What I want instead is to create a lever that activates “clean room” but not “futz”, e.g. by explicitly noting the possibility to just put futzy stuff in a box and not deal with it more. When I do that, I feel motivated to clean my messy room. I think this explains some “akrasia”.
The general pattern: I want to do X to acheive some goal, but the only way (that I know how right now) to do X is if I also do Y, and doing Y in this situation would be bad. Flinching away from action toward a goal is often about protecting your goals.
Good insight, but I’m not sure if it’s an error, or just a feature of the fact that reality is generally entangled. Most actions do, in fact, have multiple consequences on different axes.
One often ends up pulling multiple levers, to try to amplify the effects you like and dampen the ones you don’t.
Use Authy, not Google authenticator. GA not supporting any sort of backups is a huge problem.
I often have the experience of being in the middle of a discussion and wanting to reference some simple but important idea / point, but there doesn’t exist any such thing. Often my reaction is “if only there was time to write an LW post that I can then link to in the future”. So far I’ve just been letting these ideas be forgotten, because it would be Yet Another Thing To Keep Track Of. I’m now going to experiment with making subcomments here simply collecting the ideas; perhaps other people will write posts about them at some point, if they’re even understandable.
Let’s say you’re trying to develop some novel true knowledge about some domain. For example, maybe you want to figure out what the effect of a maximum wage law would be, or whether AI takeoff will be continuous or discontinuous. How likely is it that your answer to the question is actually true?
(I’m assuming here that you can’t defer to other people on this claim; nobody else in the world has tried to seriously tackle the question, though they may have tackled somewhat related things, or developed more basic knowledge in the domain that you can leverage.)
First, you might think that the probability of your claims being true is linear in the number of insights you have, with some soft minimum needed before you really have any hope of being better than random (e.g. for maximum wage, you probably have ~no hope of doing better than random without Econ 101 knowledge), and some soft maximum where you almost certainly have the truth. This suggests that P(true) is a logistic function of the number of insights.
Further, you might expect that for every doubling of time you spend, you get a constant number of new insights (the logarithmic returns are because you have diminishing marginal returns on time, since you are always picking the low-hanging fruit first). So then P(true) is logistic in terms of log(time spent). And in particular, there is some soft minimum of time spent before you have much hope of doing better than random.
This soft minimum on time is going to depend on a bunch of things—how “hard” or “complex” or “high-dimensional” the domain is, how smart / knowledgeable you are, how much empirical data you have, etc. But mostly my point is that these soft minimums exist.
A common pattern in my experience on LessWrong is that people will take some domain that I think is hard / complex / high-dimensional, and will then make a claim about it based on some pretty simple argument. In this case my response is usually “idk, that argument seems directionally right, but who knows, I could see there being things that make it wrong”, without being able to point to any such thing (because I also have spent barely any time thinking about the domain). Perhaps a better way of saying it would be “I think you need to have thought about this for more time than you have before I expect you to do better than random”.
Sometimes people say “look at these past accidents; in these cases there were giant bureaucracies that didn’t care about safety at all, therefore we should be pessimistic about about AI safety”. I think this is backwards, and that you should actually conclude the reverse: this is evidence that problems tend to be easy, and therefore we should be optimistic about AI safety.
This is not just one man’s modus ponens—the key issue is the selection effect.
It’s easiest to see with a Bayesian treatment. Let’s say we start completely uncertain about what fraction of people will care about problems, i.e. uniform distribution over [0, 100]%. In what worlds do I expect to see accidents where giant bureaucracies don’t care about safety? Almost all of them—even if 90% of people care about safety, there will still be some cases where people didn’t care and accidents happened; and of course we’d hear about them if so (and not hear about the cases where accidents didn’t happen). You can get a strong update against 99.9999% and higher, but by the time you’re at 90% the update seems pretty weak. Given how much selection there is, I think even the update against 99% is relatively weak. So really you just don’t learn much about how careful people will be by looking at our accident track record (unless you can also quantify the denominator of how many “potential accidents” there could have been).
However, it feels pretty notable to me that the vast majority of accidents I hear about in detail are ones where it seems like there were a bunch of obvious mistakes and the accidents would have been prevented had there been a decision-maker who cared (enough) about safety. And unlike the previous paragraph, I do expect to hear about accidents that we couldn’t have prevented, so I don’t have to worry about selection bias. So it seems like I should conclude that usually problems are pretty easy, and “all we have to do” is make sure people care. (One counterargument is that problems look obvious only in hindsight; at the time the obvious mistakes may not have been obvious.)
Examples of accidents that fit this pattern: the Challenger crash, the Boeing 737-MAX issues, everything in Engineering a Safer World, though admittedly the latter category suffers from some selection bias.
You’ve heard of crucial considerations, but have you heard of red herring considerations?
These are considerations that intuitively sound like they could matter a whole lot, but actually no matter how the consideration turns out it doesn’t affect anything decision-relevant.
To solve a problem quickly, it’s important to identify red herring considerations before wasting a bunch of time on them. Sometimes you can even start outlining solutions that turn a bunch of seemingly-crucial considerations into red herring considerations.
For example, it might seem like “what is the right system of ethics” is a crucial consideration for AI alignment (after all, you need to know ethics to write down a utility function), but once you decide to instead aim to design algorithms that allow you to build AI systems for any task you have in mind, that turns into a red herring consideration.
Here’s an example where I argue that, for a specific question, anthropics is a red herring consideration (thus avoiding the question of whether to use SSA or SIA).
Alternate names: sham considerations? insignificant considerations?
“Burden of proof” is a bad framing for epistemics. It is not incumbent on others to provide exactly the sequence of arguments to make you believe their claim; your job is to figure out whether the claim is true or not. Whether the other person has given good arguments for the claim does not usually have much bearing on whether the claim is true or not.
Similarly, don’t say “I haven’t seen this justified, so I don’t believe it”; say “I don’t believe it, and I haven’t seen it justified” (unless you are specifically relying on absence of evidence being evidence of absence, which you usually should not be, in the contexts that I see people doing this).
I’m not 100% sure this needs to be much longer. It might actually be good to just make this a top-level post so you can link to it when you want, and maybe specifically note that if people have specific confusions/complaints/arguments that they don’t think the post addresses, you’ll update the post to address those as they come up?
(Maybe caveating the whole post under “this is not currently well argued, but I wanted to get the ball rolling on having some kind of link”)
That said, my main counterargument is: “Sometimes people are trying to change the status quo of norms/laws/etc. It’s not necessarily possible to review every single claim anyone makes, and it is reasonable to filter your attention to ‘claims that have been reasonably well argued.’”
I think ‘burden of proof’ isn’t quite the right frame but there is something there that still seems important. I think the bad thing comes from distinguishing epistemics vs Overton-norm-fighting, which are in fact separate.
I don’t really want this responsibility, which is part of why I’m doing all of these on the shortform. I’m happy for you to copy it into a top-level post of your own if you want.
I agree this makes sense, but then say “I’m not looking into this because it hasn’t been well argued (and my time/attention is limited)”, rather than “I don’t believe this because it hasn’t been well argued”.
An argument form that I like:
I think this should be convincing even if Y is false, unless you can explain why your argument for X does not work under assumption Y.
An example: any AI safety story (X) should also work if you assume that the AI does not have the ability to take over the world during training (Y).
Trying to follow this. Doesn’t the Y (AI not taking over the world during training) make it less likely that X(AI will take over the world at all)?
Which seems to contradict the argument structure. Perhaps you can give a few more examples to make more clear the structure?
In that example, X is “AI will not take over the world”, so Y makes X more likely. So if someone comes to me and says “If we use <technique>, then AI will be safe”, I might respond, “well, if we were using your technique, and we assume that AI does not have the ability to take over the world during training, it seems like the AI might still take over the world at deployment because <reason>”.
I don’t think this is a great example, it just happens to be the one I was using at the time, and I wanted to write it down. I’m explicitly trying for this to be a low-effort thing, so I’m not going to try to write more examples now.
EDIT: Actually, the double descent comment below has a similar structure, where X = “double descent occurs because we first fix bad errors and then regularize”, and Y = “we’re using an MLP / CNN with relu activations and vanilla gradient descent”.
In fact, the AUP power comment does this too, where X = “we can penalize power by penalizing the ability to gain reward”, and Y = “the environment is deterministic, has a true noop action, and has a state-based reward”.
Maybe another way to say this is:
I endorse applying the “X proves too much” argument even to impossible scenarios, as long as the assumptions underlying the impossible scenarios have nothing to do with X. (Note this is not the case in formal logic, where if you start with an impossible scenario you can prove anything, and so you can never apply an “X proves too much” argument to an impossible scenario.)
When you make an argument about a person or group of people, often a useful thought process is “can I apply this argument to myself or a group that includes me? If this isn’t a type error, but I disagree with the conclusion, what’s the difference between me and them that makes the argument apply to them but not me? How convinced I am that they actually differ from me on this axis?”
“Minimize AI risk” is not the same thing as “maximize the chance that we are maximally confident that the AI is safe”. (Somewhat related comment thread.)
An incentive for property X (for humans) usually functions via selection, not via behavior change. A couple of consequences:
In small populations, even strong incentives for X may not get you much more of X, since there isn’t a large enough population for there to be much deviation on X to select on.
It’s pretty pointless to tell individual people to “buck the incentives”, even if they are principled people who try to avoid doing bad things, if they take your advice they probably just get selected against.
Let’s say we’re talking about something complicated. Assume that any proposition about the complicated thing can be reformulated as a series of conjunctions.
Suppose Alice thinks P with 90% confidence (and therefore not-P with 10% confidence). Here’s a fully general counterargument that Alice is wrong:
Decompose P into a series of conjunctions Q1, Q2, … Qn, with n > 10. (You can first decompose not-P into R1 and R2, then decompose R1 further, and decompose R2 further, etc.)
Ask Alice to estimate P(Qk | Q1, Q2, … Q{k-1}) for all k.
At least one of these must be over 99% (if we have n = 11 and they were all 99%, then probability of P would be (0.99 ^ 11) = 89.5% which contradicts the original 90%).
Argue that Alice can’t possibly have enough knowledge to place under 1% on the negation of the statement.
----
What’s the upshot? When two people disagree on a complicated claim, decomposing the question is only a good move when both people think that is the right way to carve up the question. Most of the disagreement is likely in how to carve up the claim in the first place.
In general, evaluate the credibility of experts on the decisions they make or recommend, not on the beliefs they espouse. The selection in our world is based much more on outcomes of decisions than on calibration of beliefs, so you should expect experts to be way better on the former than on the latter.
By “selection”, I mean both selection pressures generated by humans, e.g. which doctors gain the most reputation, and selection pressures generated by nature, e.g. most people know how to catch a ball even though most people would get conceptual physics questions wrong.
Similarly, trust decisions / recommendations given by experts more than the beliefs and justifications for those recommendations.
I like this experiment! Keep ’em coming.
Consider two methods of thinking:
1. Observe the world and form some gears-y model of underlying low-level factors, and then make predictions by “rolling out” that model
2. Observe relatively stable high-level features of the world, predict that those will continue as is, and make inferences about low-level factors conditioned on those predictions.
I expect that most intellectual progress is accomplished by people with lots of detailed knowledge and expertise in an area doing option 1.
However, I expect that in the absence of detailed expertise, you will do much better at predicting the world by using option 2.
I think many people on LW tend to use option 1 almost always and my “deference” to option 2 in the absence of expertise is what leads to disagreements like How good is humanity at coordination?
Conversely, I think many of the most prominent EAs who are skeptical of AI risk are using option 2 in a situation where I can use option 1 (and I think they can defer to people who can use option 1).
I recently interviewed someone who has a lot of experience predicting systems, and they had 4 steps similar to your two above.
Observe the world and see if it’s sufficient to other systems to predict based on intuitionistic analogies.
If there’s not a good analogy, Understand the first principles, then try to reason about the equilibria of that.
If that doesn’t work, Assume the world will stay in a stable state, and try to reason from that.
If that doesn’t work, figure out the worst case scenario and plan from there.
I think 1 and 2 are what you do with expertise, and 3 and 4 are what you do without expertise.
Yeah, that sounds about right to me. I think in terms of this framework my claim is primarily “for reasonably complex systems, if you try to do 2 without expertise, you will fail, but you may not realize you have failed”.
I’m also noticing I mean something slightly different by “expertise” than is typically meant. My intended meaning of “expertise” is more like “you have lots of data and observations about the system”, e.g. I think LW self-help stuff is reasonably likely to work (for the LW audience) because people have lots of detailed knowledge and observations about themselves and their friends.
Options 1 & 2 sound to me a lot like inside view and outside view. Fair?
Yeah, I think so? I have a vague sense that there are slight differences but I certainly haven’t explained them here.
EDIT: Also, I think a major point I would want to make if I wrote this post is that you will almost certainly be quite wrong if you use option 1 without expertise, in a way that other people without expertise won’t be able to identify, because there are far more ways the world can be than you (or others) will have thought about when making your gears-y model.
Sounds like you probably disagree with the (exaggeratedly stated) point made here then, yeah?
(My own take is the cop-out-like, “it depends”. I think how much you ought to defer to experts varies a lot based on what the topic is, what the specific question is, details of your own personal characteristics, how much thought you’ve put into it, etc.)
Correct.
I didn’t say you should defer to experts, just that if you try to build gears-y models you’ll be wrong. It’s totally possible that there’s no way to get to reliably correct answers and you instead want decisions that are good regardless of what the answer is.
Good point!
Intellectual progress requires points with nuance. However, on online discussion forums (including LW, AIAF, EA Forum), people seem to frequently lose sight of the nuanced point being made—rather than thinking of a comment thread as “this is trying to ascertain whether X is true”, they seem to instead read the comments, perform some sort of inference over what the author must believe if that comment were written in isolation, and then respond to that model of beliefs. This makes it hard to have nuance without adding a ton of clarification and qualifiers everywhere.
I find that similar dynamics happen in group conversations, and to some extent even in one-on-one conversations (though much less so).
The simple response to the unilateralist curse under the standard setting is to aggregate opinions amongst the people in the reference class, and then do the majority vote.
A particular flawed response is to look for N opinions that say “intervening is net negative” and intervene iff you cannot find that many opinions. This sacrifices value and induces a new unilateralist curse on people who think the intervention is negative. (Example.)
However, the hardest thing about the unilateralist curse is figuring out how to define the reference class in the first place.
I didn’t get it… is the problem with the “look for N opinions” response that you aren’t computing the denominator (|”intervening is positive”| + |”intervening is negative”|)?
Yes, that’s the problem. In this situation, if N << population / 2, you are likely to not intervene even when the intervention is net positive; if N >> population / 2, you are likely to intervene even when the intervention is net negative.
(This is under the standard model of a one-shot decision where each participant gets a noisy observation of the true value with the noise being iid Gaussians with mean zero.)
Under the standard setting, the optimizer’s curse only changes your naive estimate of the EV of the action you choose. It does not change the naive decision you make. So, it is not valid to use the optimizer’s curse as a critique of people who use EV calculations to make decisions, but it is valid to use it as a critique of people who make claims about the EV calculations of their most preferred outcome (if they don’t already account for it).
At any given time, is there anything especially wrong about using citation count (weighted by the weightings of other paper’s citation count) as a rough proxy for “what are the most important papers, and/or best authors, weighted?”
My sense is the thing that’s bad about this is that it creates an easy goodhart metric. I can imagine worlds where it’s already so thoroughly goodharted that it doesn’t signal anything anymore. If that’s the case, can you get around that by grounding it out in some number of trusted authors, and purging obviously fraudulent authors from the system?
I’m asking from the lens of “I’d like to have some kind barometer for which scientific papers (or, also, LW posts) are the best. And this just… actually seems pretty good, at least if you were only using it as a one-time-check.”
It depends what you mean by “rough proxy”, and whether you’re applying it to scientific papers (where Goodhart has been out in force for decades, so a one-time check is off the table) or to LessWrong posts (where citation-count has never been something people cared about). Most things have zero citations, and this is indeed a negative quality signal. But after you get to stuff that’s cited at all, citation count is mainly determined by the type and SEO of a paper, rather than its quality. Eg this paper. Citations also don’t distinguish building upon something from criticizing it. That’s much worse in the Goodhart arena than the one-time arena, but still pretty bad in the one-shot case.
Nod. “positive vs disagreement citation” is an important angle I wasn’t thinking about.
Important for what? Best for what?
In a given (sub)field, the highest-cited papers tend to be those which introduced or substantially improved on a key idea/result/concept; so they’re important in that sense. If you’re looking for the best introduction though that will often be a textbook, and there might be important caveats or limitations in a later and less-cited paper.
I’ve also had a problem where a few highly cited papers propose $approach, many papers apply or puport to extend it, and then eventually someone does a well-powered study checking whether $approach actually works. Either way that’s an important paper, but they tend to be under-cited either because either the results are “obvious” (and usually a small effect) or the field of $approach studies shrinks considerably.
It’s an extremely goodhartable metric but perhaps the best we have for papers; for authors I tend to ask “does this person have good taste in problems (important+tractable), and are their methods appropriate to the task?”.
On October 26, 2020, I submitted a security vulnerability report to the Facebook bug bounty program. The submission was rejected as a duplicate. As of today (April 14), it is still not fixed. I just resubmitted, since it seems to have fallen through the cracks or something. However, I consider all my responsible disclosure responsibilities to be discharged.
Once an Oculus Quest or Oculus Quest 2 is logged in to a Facebook account, its login can’t be revoked. There is login-token revocation UI in Facebook’s Settings>Security and Login menu, but changing the account password and revoking the login there does not work.
One practical impact of this is that if your Facebook account is ever compromised, and the attacker uses this vulnerability, they have permanent access.
The other practical impact is that if someone has unsupervised access to your unlocked Quest headset, and they use the built-in web browser to go to facebook.com, they have full access to your Facebook account, including Messenger, without having to do anything special at all. This means that if you’ve ever made a confidentiality agreement regarding something you discussed on Facebook Messenger, you probably can’t lend your headset to anyone, ever.
Additionally, the lock-screen on the Oculus Quest 2 does not have a strict enough rate limit; it gives unlimited tries at 2/minute, so trying all lock-screen combinations takes approximately 35 days. This can be done without network access, and can be automated with some effort. So if someone steals a *locked* Oculus Quest 2, they can also use that to break into your Facebook account. There is almost certainly a much faster way to do this involving disassembling the device, but this is bad enough.
Is your logic that releasing this heinous volun into the public is more likely to pressure FB to do something about this? Because if so, I’m not sure that LW is a forum with enough public spotlight to generate pressure. OTOH, I imagine some percentage of readers here aren’t well-aligned but are looking for informational edge, in which case it’s possible this does more harm than good?
I’m not super-confident in this model—eg, it also seems entirely possible to me that lots of FB security engineers read the site and one or more will be shouting ZOMG! any moment over this..
I’m posting here (cross-posted with my FB wall and Twitter) mostly to vent about it, and to warn people that sharing VR headsets has infosec implications they may not have been aware of. I don’t think this comment will have much effect on Facebook’s actions.
Life is quined matter.
DNA is a quine, when processed by DNA Replicase.
Although with a sufficiently complex (or specific) compiler or processor, any arbitrary stream of information can be made into a quine.
You can embed information in either of instructions/substrate, or compiler/processor. Quines usually seem limited to describing instructions/substrate, but that’s not the only place information can be drawn from. I’ve kinda come to think of it as 2-term operation (ex: “this bit of quine code is a quine wrt python”).
(More physical quines: Ink scribbles on a sheet of paper are a quine wrt a copy-machine. At about the far-end of “replicative complexity gets stored in the processor,” you have the “activated button” of a button-making machine (which activates with a button, and outputs buttons in an active state). I think the “activated button” here is either a quine, or almost a quine.)
The cool thing about life is that it it is both a quine, and its own processor. (And also, sorta its own power source.)
I find it simpler to call systems like this “living” (at least while active/functional), since they’re meaningfully more than just quines.
Viruses are definitely quines, though. Viruses and plasmids are non-living quines that compile and run on living biological systems.
Isn’t life then a quine running on physics itself as a substrate?
I hadn’t considered thinking of quines as two-place, but that’s obvious in retrospect.
It’s sorta non-obvious. I kinda poked at this for hours, at some point? It took a while for me to settle on a model I liked for this.
Here’s the full notes for what I came up with.
Physics: Feels close. Hm… biological life as a self-compiler on a physics substrate?
DNA or gametes seem really close to a “quine” for this: plug it into the right part of an active compiler, and it outputs many instances of its own code + a customized compiler. Although it crashes/gets rejected if the compiler is too different (ex: plant & animal have different regulatory markers & different sugar-related protein modifications).
I don’t have a fixed word for the “custom compiler” thing yet (“optimized compiler”? “coopted compiler”? “spawner”? “Q-spawner”?). I have seen analogous stuff in other places, and I’m tempted to call it a somewhat common pattern, though. (ex: vertically-transmitted CS compiler corruption, or viruses producing tumor micro-environments that are more favorable to replicating the virus)
Live Parsers and Quines
An unusual mathematical-leaning definition of a living thing.
(Or, to be more precise… a potential living immortal? A replicon? Whatever.)
A self-replicating physical entity with...
3 Core Components:
Quine: Contains the code to produce itself
Parser: A parser for that code
Code includes instructions for the parser; ideally, compressed instructions
Power: Actively running (probably on some substrate)
Not actively running is death, although for some the death is temporary.
Access to resources may be a subcomponent of this?
Additional components:
Substrate: A material that can be converted into more self
Possibly a multi-step process
Access to resources is probably a component of this
Translator: Converts quine into power, or vice-versa
Not always necessary; sometimes a quine is held as power, not substrate
Parser and Code: the information can actually be stored on either; you can extract the correct complex signal from complete randomness using an arbitrarily-complex parser chosen for that purpose. There are analogies that can be drawn in the other direction, too: a fairly dumb parser can make fairly complicated things, given enough instructions. (Something something Turing Machines)
Ideally, though, they’re well-paired and the compression method is something sensible, to reduce complexity.
A quine by itself has only a partial life; it is just its own code, it requires a parser to replicate.
(If you allow for arbitrarily complex parsers, any code could be a quine, if you were willing to search for the right parser.)
Compilers are… parsers? (Or translators?)
It is possible for what was “code” information to be embedded into the parser. I think this is part of what happens when you completely integrate parts in IFS.
Examples
Example replicons: A bacterium, a cell, a clonal organism, a pair of opposite-sexed humans (but not a single), self-contained repeating Game of Life automata, the eventual goal of RepRap (a 3D printer fully producing a copy of itself)
Viruses: Sometimes quines, sometimes pre-quine and translator
-- What I Worked On, Paul Graham
Related: Self-Reference, Post-Irony, Hofstadter’s Strange Loop, Turing Machine, Automata (less-so), deconstruction (less-so)
Is Tiddlywiki a quine?
Is a cryonics’d human a quine? (wrt future technology)
The definition of parasite used in Nothing in evolution makes sense except in the light of parasites is literally quines.
(Complex multi-step life-cycle in the T5 case, though?. The quine produced a bare-bones quine-compiling “spawner” when it interacts with the wild-type, and then replicates on that. Almost analogous to CS viruses that infect the compiler, and any future compiler compiled by that compiler.)
Process is Art: Is art that demonstrates how to craft the art, a quine on a human compiler?
Some connection to: Compartmentalization (Black Box Testing, unit tests, separation of software components) and the “swappability” of keeping things general and non-embedded, with clearly-marked and limited interface-surfaces (APIs). Generates pressure against having “arbitrary compilers” in practice.
The coronavirus response has been so bad I no longer doubt many Sci-Fi premises. Before I often said to myself “you have tech that can do X and you still have problem Y. Ridiculous!”. But apparently, we can make a coronavirus vaccine in two days. But we still had over a year of lockdowns and millions of deaths. One in six hundred people in the USA have died of the virus but we blocked a vaccine because one in a million people who take it MIGHT develop treatable blood clots.
My standards for ‘realistic’ dysfunction have gotten a lot lower.
On the flipside: WTF Star Trek?
The heads of Government and the FDA don’t work like you do. Who knows the incentives that they have? It’s entirely possible that for them this is just a political play(moving chess pieces) that make sense for them while the well-being of the people take secondary place to a particular political move.
This wouldn’t be the first thing that this happens in any kind of government agency, but, at any rate, it’s too early to be skeptical. We need to see how this unfolds, may be the pausing don’t last as much.
I remember Yudkowsky asking for a realistic explanation for why the Empire in Star Wars is stuck in an equilibrium where it builds destroyable gigantic weapons.
I once read a comment somewhere that Paul Graham is not a rationalist, though he does share some traits, like writing a lot of self-improvement advice. From what I can tell Paul himself considers himself a builder; a builder of code and companies. But there is some overlap with rationalists, Paul Graham mostly builds information systems. (He is somewhat disdainful of hardware, which I consider the real engineering, but I am a physicist.) Rationalists are focussed on improving their own intelligence and other forms of intelligence. So both spend a great deal of time building and improving intelligent information systems, as well as improving their own mind, but for different reasons. For one the goal is merely to build, and self-improvement is a method, for the other self-improvement is the goal and building is a method. Well, and for some the goal is to build a self-improving intelligence (that doesn’t wipe us out).
Builders and rationalists. Experimentalists and theoretical empiricists. I suppose they work well together.
This is a really good comment. If you care to know more about his thinking, he has a book called, “hackers and painters” which I think sums up very well his views. But yes, it’s a redistribution of wealth and power from strong people and bureaucrats to what he calls “nerds” as in people who know technology deeply and actually build things.
The idea of instrumental rationality touches at the edges of builders, and you need to if you ever so desire to act in the world.
Note that for hardware, the problems are that you need a minimum instruction set in order to make a computer work. So long as you at least implement the minimum instruction set, and for all supported instructions perform the instructions (which are all in similar classes of functions) bit for bit correctly, done. It’s ever more difficult to make a faster computer for a similar manufacturing cost and power consumption because the physics keep getting harder.
But it is in some ways a “solved problem”. Whether a given instance of a computer is ‘better’ is a measurable parameter, the hardware people try things, the systems and software engineers adopt the next chip if it meets their definition of ‘better’.
So yeah if we want to see new forms of applications that aren’t in the same class as what we have already seen—that’s a software and math problem.
I’m not sure I follow. Whether it’s the evolving configuration of atoms or bits, both can lead to new applications. The main difference to me seems that today it is typically harder to configure atoms than bits, but perhaps that’s just by our own design of the atoms underlying the bits? If some desired information system would require a specific atomic configuration, then you’d be hardware constrained again.
Let’s say that in order to build AGI we find out you actually need super power efficient computronium, and silicon can’t do that, you need carbon. Now it’s no longer a solved hardware problem, you are going to have to invest massively in carbon based computing. Paul and the rationalists are stuck waiting for the hardware engineers.
I am saying below a certain level of abstraction it becomes a solved problem in that you precisely have defined what correctness is and have fully represented your system. And you can trivially check any output and validate it versus a model.
The reason software fails constantly is we don’t have a good definition that can be checked by computer of what correctness means. Software Unit tests help but are not nearly as reliable as tests for silicon correctness. Moreover software just ends up being absurdly more complex than hardware and ai systems are worse.
Part of it is “unique complexity”. A big hardware system is millions of copies of the same repeating element. And locality matters—an element cannot affect another one far away unless a wire connects them. A big software system is millions of copies of often duplicated and nested and invisibly coupled code.
Years after I first thought of it, I continue to think that this chain reaction is the core of what it means for something to be an agent, AND why agency is such a big deal, the sort of thing we should expect to arise and outcompete non-agents. Here’s a diagram:
Roughly, plans are necessary for generalizing to new situations, for being competitive in contests for which there hasn’t been time for natural selection to do lots of optimization of policies. But plans are only as good as the knowledge they are based on. And knowledge doesn’t come a priori; it needs to be learned from data. And, crucially, data is of varying quality, because it’s almost always irrelevant/unimportant. High-quality data, the kind that gives you useful knowledge, is hard to come by. Indeed, you may need to make a plan for how to get it. (Or more generally, being better at making plans makes you better at getting higher-quality data, which makes you more knowledgeable, which makes your plans better.)
Seems similar to the OODA loop
Yep! I prefer my terminology but it’s basically the same concept I think.
I think it’s probably even simpler than that: feedback loops are the minimum viable agent, i.e. a thermostat is the simplest kind of agent possible. Stuff like knowledge and planning are elaborations on the simple theme of the negative feedback circuit.
I disagree; I think we go astray by counting things like thermostats as agents. I’m proposing that this particular feedback loop I diagrammed is really important, a much more interesting phenomenon to study than the more general category of feedback loop that includes thermostats.
An aircraft carrier costs $13 billion. An anti-ship cruise missile costs $2 million. Few surface warships survived the first day of the Singularity War.
A cruise missile is a complex machine, guided by sensitive electronics. Semiconductor fabricators are even more complex machines. Few semiconductor factories survived the nuclear retaliation.
A B-52 Stratofortress is a simpler machine.
Robert (Bob) Manchester’s bomber flew west from Boeing Field. The crew disassembled their landing gear and dropped it in the Pacific Ocean. The staticy voice of Mark Campbell, Leader of the Human Resistance, radioed into Robert’s headset. Robert could barely hear it over the noise of the engines. He turned the volume up. It would damage his hearing but that didn’t matter anymore. The attack wouldn’t save America. Nothing could at this point. But the attack might buy time to launch a few extra von Neumann probes.
The squadron flew over miles after miles of automated factories. What was once called Tianjin was now just Sector 153. The first few flak cannons felt perfunctory. The anti-air fire increased as they drew closer to enemy network hub. Robert dropped the bombs. The pilot, Peter Garcia, looked for a target to kamikaze.
They drew closer to the ground. Robert Manchester looked out the window. He wondered why the Eurasian AI had chosen to structure its industry around such humanoid robots.
In response to my earlier post about Myers-Briggs (where I suggested a more detailed notation for more nuanced communication about personality types), it was pointed out that there is some correlation between the four traits being measured, and this makes the system communicate less information on average than it otherwise would (The traditional notation would communicate 4 bits, my version would communicate ~9.2 if there was no correlation).
I do object to the characterization that it all measures “the same thing”, since none of the traits perfectly predicts the others, and all 16 of the traditional configurations have people they describe (though some are more common than others); but I do think it makes sense to try to disentangle things—if the I / E scale is correlated with the J / P scale, we can subtract some amount of J points for more introverted people, and add J points for the extroverts, so that an introvert needs to be more “judgmental” to be considered “J” than an equally judgmental extrovert, with the goal being that 50% of extroverts will be J, 50% P, and have a similar 50-50 split for introverts.
By adjusting for these correlations across all pairs, we can more finely detect and communicate the underlying traits that cause “Judgement” and “Perception” that aren’t just a result of a person being more extroverted (a disposition that rewards those who are best able to use their intuition) or introverted (which often leads to pursuits that require careful thinking distanced from our whims).
I have a potential category of questions that could fit on Metaculus and work as an “AGI fire alarm.” The questions are of the format “After an AI system achieves task x, how many years will it take for world output to double?”
I don’t mind jumping through a few extra hoops in order to access a website idiosyncratically. But sometimes the process feels overly sectarian.
I was trying out the Tencent cloud without using Tor when I got a CAPTCHA. Sure, whatever. They verified my email. That’s normal. Then they wanted to verify my phone number. Okay. (Using phone numbers to verify accounts is standard practice for Chinese companies.) Then they required me to verify my credit card with a nominal $1 charge. I can understand their wanting to take extra care when it comes to processing international transactions. Then they required me to send a photo of my driver’s licence. Fine. Then they required 24 hours to process my application. Okay. Then they rejected my application. I wonder if that’s what the Internet feels like everyday to non-Americans.
I often anonymize my traffic with Tor. Sometimes I’ll end up on the French or German Google, which helps remind me that the Internet I see everyday is not the Internet everyone else sees.
Other people use Tor too, which is necessary to anonymize my traffic. Some Tor users aren’t really people. They’re bots. By accessing access the Internet from the same Tor exit relays as these bots, websites often suspect me of being a bot.
I encounter many pages like this.
This is a Russian CAPTCHA.
Prove you’re human by typing “вчepaшний garden”. Maybe I should write some OCR software to make proving my humanity less inconvenient.
Another time I had to identify which Chinese characters were written incorrectly.
The most frustrating CAPTCHAs require me to annotate images for self-driving cars. I do not mind annotating images of self-driving cars. I do mind, after having spent several minutes annotating images of self-driving cards, getting rejected based off of a traffic analysis of my IP address.
I think it’s worst when you have edge cases like the Google Captcha that shows 16 tiles and you have to choose which tiles contain the item they are looking for and some of the tails contain it only a little bit on the edge.
I think you can steelman Ben Goertzel-style worries about near-term amoral applications of AI being bad “formative influences” on AGI, but mostly under a continuous takeoff model of the world. If AGI is a continuous development of earlier systems, then maybe it shares some datasets and learned models with earlier AI projects, and definitely it shares the broader ecosystems of tools, dataset-gathering methodologies, model-evaluating paradigms, and institutional knowledge on the part of the developers. If the ecosystem in which this thing “grows up” is one that has previously been optimized for marketing, or military applications, or what have you, this is going to have ramifications in how the first AGI projects are designed and what they get exposed to. The more continuous you think the development is going to be, the more this can be intervened on by trying to make sure that AI is pro-social even in the short term.
Random thought: if you have a big enough compost pile, would it spontaneously break into flames due to the heat generated by the bioprocesses that occur therein? If so, at what size would it burst into flames? Surely it could happen before it reached the size of the sun, even ignoring gravitational effects.
(Just pondering out loud, not really asking unless someone really wants to answer)
For a value of “break into flames” that matches damp and poorly-oxygenated fuel, yep! This case in Australia is illustrative; you tend to get a lot of nasty smoke rather than a nice campfire vibe.
You’d have to mismanage a household-scale compost pile very badly before it spontaneously combusts, but it’s a known and common failure mode for commercial-scale operations above a few tons. Specific details about when depend a great deal on the composition of the pile; with nitrate filmstock it was possible with as little as a few grams.
The child-in-a-pond thought experiment is weird, because people use it in ways it clearly doesn’t work for (especially in arguing for effective altruism).
For example, it observes you would be altruistic in a near situation with the drowning child, and then assumes that you ought to care about people far away as much as people near you. People usually don’t really argue against this second step, but very much could. But the thought experiment makes no justification for that extension of the circle of moral concern, it just assumes it.
Similarly, it says nothing about how effectively you ought to use your resources, only that you probably ought to be more altruistic in a stranger-encompassing way.
But not only does this thought experiment not argue for the things people usually use it for, it’s also not good for arguing that you ought to be more altruistic!
Underlying it is a theme that plays a role in many thought experiments in ethics: they appeal to game-theoretic intuition for useful social strategies, but say nothing of what these strategies are useful for.
Here, if people catch you letting a child drown in a pond while standing idly, you’re probably going to be excluded from many communities or even punished. And this schema occurs very often! Unwilling organ donors, trolley problems, and violinists.
Bottom line: Don’t use the drowning child argument to argue for effective altruism.
I don’t know, I think it’s a pretty decent argument. I agree it sometimes gets overused, but I do think given it’s assumptions “you care about people far away as much as people closeby” and “there are lots of people far away you can help much more than people close by” and “here is a situation where you would help someone closeby, so you might also want to help the people far away in the same way” are all part of a totally valid logical chain of inference that seems useful to have in discussions on ethics.
Like, you don’t need to take it to an extreme, but it seems locally valid and totally fine to use, even if not all the assumptions that make it locally valid are always fully explicated.
Right, my gripe with the argument is that these first two assumptions are almost always unstated, and most of the time when people use the argument, they “trick” people into agreeing with assumption one.
(for the record, I think the first premise is true)
On self-reflection, I just plain don’t care about people far away as much as those near to me. Parts of me think I should, but other parts aren’t swayed. The fact that a lot of the motivating stories for EA don’t address this at all is one of the reasons I don’t listen very closely to EA advice.
I am (somewhat) an altruist. And I strive to be effective at everything I undertake. But I’m not an EA, and I don’t really understand those who are.
Yep, that’s fine. I am not a moral prescriptivist who tells you what you have to care about.
I do think that you are probably going to change your mind on this at some point in the next millennium if we ever get to live that long, and I do have a bunch of arguments that feel relevant, but I don’t think it’s completely implausible you really don’t care.
I do think that not caring about how people are far away is pretty common, and building EA on that assumption seems fine. Not all clubs and institutions need to be justifiable to everyone.
Last month, I wrote a post here titled “Even Inflationary Currencies Should Have Fixed Total Supply”, which wasn’t well-received. One problem was that the point I argued for wasn’t exactly the same as what the title stated: I supported both currencies with fixed total supply, and currencies that instead choose to scale supply proportional to the amount of value in the currency’s ecosystem, and many people got confused and put off by the disparity between the title and my actual thesis; indeed, one of the most common critiques in the comments was a reiteration of a point I had already made in the original post.
Zvi helpfully pointed out another effect that nominal inflation has that serves as part of the reason inflation is implemented the way it is, that I wasn’t previously aware of, namely that nominal inflation induces people to accept worsening prices they psychologically would otherwise resist. While I feel intentionally invoking this effect flirts with the boundary of dishonesty, I do recognize the power and practical benefits of this effect.
All that said, I do stand by the core of my original thesis: nominal inflation is a source of much confusion for normal people, and makes the information provided by price signals less easily legible over long spans of time, which is problematic. Even if the day-to-day currency continues to nominally inflate like things are now, it would be stupid not to coordinate around a standard stable unit of value (like [Year XXXX] Dollars, except without having to explicitly name a specific year as the basis of reference; and maybe don’t call it dollars, to make it clear that the unit isn’t fluidly under the control of some organization)
Productivity app idea:
You set a schedule of times you want to be productive, and a frequency, and then it rings you at random (but with that frequency) to bug you with questions like:
--Are you “in the zone” right now? [Y] [N]
--(if no) What are you doing? [text box] [common answer] [ common answer] [...]
The point is to cheaply collect data about when you are most productive and what your main time-wasters are, while also giving you gentle nudges to stop procrastinating/browsing/daydream/doomscrolling/working-sluggishly, take a deep breath, reconsider your priorities for the day, and start afresh.
Probably wouldn’t work for most people but it feels like it might for me.
“Are you in the zone right now?”
″… Well, I was.”
I’m betting that a little buzz on my phone which I can dismiss with a tap won’t kill my focus. We’ll see.
This is basically a souped up version of TagTime (by the Beeminder folks) so you might be able to start with their implementation.
I’ve been thinking about a similar idea.
This format of data collection is called “experience sampling”. I suspect there might be already made solutions.
Would you pay for such an app? If so, how much?
Also looks like your crux is actually becoming more productive (i.e. experience sampling is a just a mean to reach that). Perhaps just understanding your motivation better would help (basically http://mindingourway.com/guilt/)?
More surprised than perhaps I should be that people take up tags right away after creating them. I created the IFS tag just a few days ago after noticing it didn’t exist but wanted to link it and I added the first ~5 posts that came up if I searched for “internal family systems”. It now has quite a few more posts tagged with it that I didn’t add. Super cool to see the system working in real time!
__Levers error__.
Anna writes about bucket errors . Attempted summary: sometimes two facts are mentally tracked by only one variable; in that case, correctly updating the belief about one fact can also incorrectly update the belief about the other fact, so it is sometimes epistemic to flinch away from the truth of the first fact (until you can create more variables to track the facts separately).
There’s a conjugate error: two actions are bound together in one “lever”.
For example, I want to clean my messy room. But somehow it feels pointless / tiring, even before I’ve started. If I just started cleaning anyway, I’d get bogged down in some corner, trying to make a bunch of decisions about where exactly to put lots of futzy random objects, tiring myself out and leaving my room still annoyingly cluttered. It’s not that there’s a necessary connection between cleaning my room and futzing around inefficiently; it’s that the only lever I have right now that activates the “clean room” action also activates the “futz interminably” action.
What I want instead is to create a lever that activates “clean room” but not “futz”, e.g. by explicitly noting the possibility to just put futzy stuff in a box and not deal with it more. When I do that, I feel motivated to clean my messy room. I think this explains some “akrasia”.
The general pattern: I want to do X to acheive some goal, but the only way (that I know how right now) to do X is if I also do Y, and doing Y in this situation would be bad. Flinching away from action toward a goal is often about protecting your goals.
Good insight, but I’m not sure if it’s an error, or just a feature of the fact that reality is generally entangled. Most actions do, in fact, have multiple consequences on different axes.
One often ends up pulling multiple levers, to try to amplify the effects you like and dampen the ones you don’t.