Was on Vivek Hebbar’s team at MIRI, now working with Adrià Garriga-Alonso onvarious empirical alignment projects.
I’m looking for projects in interpretability, activation engineering, and control/oversight; DM me if you’re interested in working with me.
Eliezer Yudkowsky wrote in 2016:
At an early singularity summit, Jürgen Schmidhuber, who did some of the pioneering work on self-modifying agents that preserve their own utility functions with his Gödel machine, also solved the friendly AI problem. Yes, he came up with the one true utility function that is all you need to program into AGIs!
(For God’s sake, don’t try doing this yourselves. Everyone does it. They all come up with different utility functions. It’s always horrible.)
His one true utility function was “increasing the compression of environmental data.” Because science increases the compression of environmental data: if you understand science better, you can better compress what you see in the environment. Art, according to him, also involves compressing the environment better. I went up in Q&A and said, “Yes, science does let you compress the environment better, but you know what really maxes out your utility function? Building something that encrypts streams of 1s and 0s using a cryptographic key, and then reveals the cryptographic key to you.”
At first it seemed to me that EY refutes the entire idea that “increasing the compression of environmental data” is intrinsically valuable. This surprised me because my intuition says it is intrinsically valuable, though less so than other things I value.
But EY’s larger point was just that it’s highly nontrivial for people to imagine the global maximum of a function. In this specific case, building a machine that encrypts random data seems like a failure of embedded agency rather than a flaw in the idea behind the utility function. What’s going on here?
Clause #3 was chosen to heighten to reward/punishment for especially good or especially bad content. We’re inclined to think that single 100-karma post is worth more than four 25-karma posts and the exponentiation reflects this. (For comparison: 25^1.2 is 47.6, 100^1.2 is 251.2. So in our metric, one 100-karma post was worth about 30% more than four 25-karma posts).
Is the idea behind this that a high-quality post can provide more than a single strong-upvote of value per person, and that total karma is a proxy for this excess value?
Can’t we tell when the marginal utility of a mask at a certain hospital is high (e.g. by observing that they are totally out of masks and plan to reuse any donated ones) and donate at that point?
Assumption: we shouldn’t expect to be able to make strong quantitative predictions unless we also expect to be able to get rich playing the markets.
I’m confused about your distinction between quantitative and qualitative. The way I understand “quantitative”, there isn’t profit to be made off of every such prediction—for example, if copper alloys become widely used in hospitals and consumer products for its antimicrobial properties, the impact on copper prices would be tiny, and the companies making these products are privately traded.
The most efficient form of practice is generally to address one’s weaknesses. Why, then, don’t chess/Go players train by playing against engines optimized for this? I can imagine three types of engines:
Trained to play more human-like sound moves (soundness as measured by stronger engines like Stockfish, AlphaZero).
Trained to play less human-like sound moves.
Trained to win against (real or simulated) humans while making unsound moves.
The first tool would simply be an opponent when humans are inconvenient or not available. The second and third tools would highlight weaknesses in one’s game more efficiently than playing against humans or computers. I’m confused about why I can’t find any attempts at engines of type 1 that apply modern deep learning techniques, or any attempts whatsoever at engines of type 2 or 3.
It was a shock on arriving at the New York Times in 2004, as the paper’s movie editor, to realize that its editorial dynamic was essentially the reverse. By and large, talented reporters scrambled to match stories with what internally was often called “the narrative.” We were occasionally asked to map a narrative for our various beats a year in advance, square the plan with editors, then generate stories that fit the pre-designated line.
For me, an article linking to this one was the fifth Google result for “new york times narrative driven”.
Minor nitpick, but in section IV I think it’s unlikely that hemispherectomy and brain shrinkage can stack linearly while preserving a human-like consciousness, because successful hemispherectomy requires the preserved half to rewire some structures to the functions normally carried out by the removed half (source) whereas halving the amount of tissue greatly reduces the resources available for that. The situation reminds me of neural net compression. We can prune or use quantization, compressing the net by some factor, but the techniques don’t stack perfectly because they eliminate some of the same redundancies.
Slightly more relevant is the evolutionary argument that any easy change to the brain that decreases its power consumption or volume must give up something very evolutionarily valuable, since brains use a huge amount of energy and increase deaths from childbirth. That is, the architecture of meat brains isn’t too inefficient. While this doesn’t refute the idea of transferring consciousness gradually, it makes me skeptical that we can do so with general-purpose hardware economically.
I think it’s a good idea to babble here (list possible effects even if there’s a <10% subjective chance they happen) because if we restrict ourselves to ideas that we’re pretty sure will verify, we’ll miss most of the good ones.
Do you have examples of such stagnant predictions? That would indeed be strong evidence against his reliability if the accumulated evidence clearly points in one direction relative to his expectations.
Cost, range, and number of electric cars are rather artificial metrics—I think measuring the cost, range, and number of cars in total would be much more in the spirit of the post, which haven’t seen any surprising departures from trends. As for batteries, a ~15% decrease in cost per doubling in cumulative number of lithium-ion batteries produced combined with roughly exponential growth in the market has meant a smooth trend over the past few years: see this source.
I believe it’s also easily checked that there’s no significant discontinuity in cost to orbit. Starship could be promising, and Musk’s goal is an extremely ambitious $2M/1000t to LEO, but even that is only 30 years or so on the current trend line.
I use LW on both desktop and mobile and sometimes get confused between the gestures, so I think adding double-click as an option on desktop would make it slightly easier to use.
Stack Exchange question votes are asymmetric, such that an upvote gives +10 rep and a downvote −2. I discuss the benefits of that system here. While LW is different from SE (do users who produce high-quality content value LW karma as much as SE rep?), it might be worth doubling karma awarded to votes on questions.
It also occurs to me that it makes strong-upvoting too easy for my taste, nearly as easy as regular voting.
I suggest “Problem-Solving Tactics” as a tag, with https://www.lesswrong.com/posts/JcpwEKbmNHdwhpq5n/problem-relaxation-as-a-tactic as an example. “Rationality” is far too general for that post.
I can’t think of other posts, but the idea is a class of general tactics that can be applied to difficult technical problems, such as Polya’s heuristics for solving math problems: generalization, specialization, using special cases, pattern-matching to related problems, etc. There are others:
Use fresh eyes
Prove a stronger statement than needed
When there are many candidate strategies, use manual depth-first search and keep track efficiently
The tag wiki for the “techniques” tag pointed out by habryka says “set of actions (including “mental actions”) for improving one’s thinking so as to form accurate beliefs and/or make better decisions”, which I think is slightly different, but the tag wiki could be edited to encompass problem-solving tactics.
Would you mind sharing your thoughts?
Thanks for the link on the “dark night”. This passage seems to be the clearest definition:
While the manifestations of spiritual crisis are highly individual, and no two spiritual crises are exactly the same, there are some common features that appear for most people. These include a loss of sense of identity, radically changing personal values, and the occurrence of mystical and spiritual experiences.
It’s plausible to me that this represents the philosophical/spiritual side of the rationalist uncanny valley for many people but is less related to practical/epistemic issues. This gives me a little hope that the “dark night” aspects can be tackled using established techniques from spiritual traditions (Anna Solomon’s advice is quite different). Though that also seems dangerous, and I wouldn’t want to try it myself unless desperate—I believe in myself enough to try to get out using standard techniques.
At risk of digressing, I think I used the word “nihilism” as combining the flavor of moral anti-realism best explained as option 4-5 of this post with the intuition that any values meaningful enough to optimize towards must be reasonably robust to changes in initial conditions of reflection in an individual, and possibly to different humans and non-humans using the same meta-level reflection process. My conclusion is that I should use my moral intuition, but then dramatically pare down the list of things I think I value for the sake of robustness. Given the counterarguments, it doesn’t get all the weight, though I take it seriously. While Meaningness claims to roundly refute nihlism, it seems to use the word slightly differently and focus on other ways people turn to nihilism.
A doctor I know ordered masks claiming N95, KN95, or similar certification from 7 different Chinese companies and fit-tested them (a process involving putting the mask on and spraying ~0.3μm particles of a bitter-tasting compound into the air); 6 models failed for all people tested while the last one failed for some people (indicating adequate filtration but poor fit).
What do you think about the possibility that there is an undetected outbreak of ~1000 cases in Indonesia? There’s indirect evidence based on people traveling from Indonesia to countries with better testing (e.g. Singapore). This post and the Metaculus post I linked both seem convincing, but it’s unlikely that they’re both true.