moridinamael
In my own writing I am very conscious of whether I’m writing from a place of inspiration.
All my most successful posts came to me as a vibrant and compelling idea that very quickly took shape in my mind and ended up being finished and posted quickly. What made them clear and living in my mind is what made them readable and engaging to readers, my job was mainly to stay out of my own way, to translate that lightning bolt of thought into writing.There’s a symmetry there: it was easy to write because the idea was so clear to me in my own mind, and this clarity is also what makes it enjoyable to read. If you don’t quite know exactly what you’re trying to say, that problem isn’t going to be overcome by more “effort” at the prose level.
Unfortunately you can’t force inspiration, or at least I haven’t figured out how to do it. I have a lot of drafts that never go posted because that inspiration/clarity wasn’t there.
Good work.
The hardest part of moderation is the need to take action in cases where someone is consistently doing something that imposes a disproportionate burden on the community and the moderators, but which is difficult to explain to a third party unambiguously.
Moderators have to be empowered to make such decisions, even if they can’t perfectly justify them. The alternative is a moderation structure captured by proceduralism, which is predictably exploitable by bad actors.
That said — this is Less Wrong, so there will always be a nitpick — I do think people need to grow a thicker skin. I have so many friends who have valuable things to say, but never post on LW due to a feeling of intimidation. The cure for this is, IMO, not moderating the level of meanness of the commentariat, but encouraging people to learn to regulate their emotions in response to criticism. However, at the margins, clipping off the most uncharitable commenters is doubtless valuable.
Sorry, that’s what I get for replying from the Notification interface.
I’m not sure if I understand your question. I am using the initial quotes from Stoic/Buddhist texts as examples of perverse thinking that I don’t endorse.
As to (1), I was following The Mind Illuminated, for what it’s worth. And I am a big fan of emotional integration. Spiritual practices can help with that, but I think they can also get in the way, and it’s really hard to know in advance which direction you’re going.
I think we are basically on the same page with (2).
As for (3) I think it’s a matter of degree, requiring the kind of nuance that doesn’t fit on a bumper sticker. If you feel so much persistent guilt that it’s causing daily suffering, then that’s probably something you need to sort out. I was intentional in adding the phrase “for a bit” in “It’s okay to feel bad for a bit,” because I don’t actually think it’s okay to feel persistently bad forever! Those are definitely two different situations. If you have ongoing intrusive negative emotions, that sounds adjacent to trauma, and that can be sorted out with some work.
I always appreciate your insights and opinions on this general topic.
At the time, I was following the instructions in The Mind Illuminated very closely. I will grant that this may have been user error/skill issue, but given that The Mind Illuminated is often put forth as a remarkably accessible and lucid map through the stages of vipassana, and given that I still went this badly wrong, you have to wonder if maybe the path itself is perhaps too dangerous to be worth it.
The outcome I reached may have been predictable, given that the ultimate reason I was meditating at the time was to get some relief from the the ongoing suffering of a chronic migraine condition. In that specific sense, I was seeking detachment.
In the end I am left wondering if I would have been better off if I had taken up mountain biking instead of meditation, given that it turned out that the path to integrating my emotions led through action more than reflection.
This post resonated with me when it came out, and I think its thesis only seems more credible with time. Anthropic’s seminal “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet” (the Golden Gate Claude paper) seems right in line with these ideas. We can make scrutable the inscrutable as long as the inscrutable takes the form of something organized and regular and repeatable.
This article gets bonus points for me for being succinct and while still making its argument clearly.
My favorite Less Wrong posts are almost always the parables and the dialogues. I find it easier to process and remember information that is conveyed in this way. They’re also simply more fun to read.
This post was originally written as an entry for the FTX Future Fund prize, which, at the time of writing the original draft, was a $1,000,000 prize, which I did not win, partly because it wasn’t selected as the winner and partly because FTX imploded and the prize money vanished. (There is a lesson about the importance of proper calibration of the extrema of probability estimates somewhere in there.) In any case, I did not actually think I would win, because I was basically making fun of the contest organizers by pointing out that the whole ethos behind their prize specification was wrong. At the time, there was a live debate around timelines, and a lot of discussions about the bio-anchors paper, which itself made in microcosm the same mistakes that I was pointing at.
Technically, the very-first-draft of this post was an extremely long and detailed argument for short AGI timelines that I co-wrote with my brother, but I realized while writing it that the presumption that long and short timelines should be in some sense averaged together to get a better estimate was pervasive in the zeitgeist and needed to be addressed on its own.
I am happy with this post because it started a conversation that I thought needed to be had. My whole shtick these days is that our community has seemingly tried to skip over decision theory basics in favor of esoterica, to our collective detriment, and I feel like writing this post explicitly helped with that.
I am happy to have seen this post referenced favorably elsewhere. I think I wrote it about as well as I could have, given that I was going for the specific Less Wrong Parable stylistic thing and not trying to write literary fiction.
There is also a weird accident-of-history situation where all of the optimizers we’ve had for the last century are really single-objective optimizers at their core. The consequence of this has been that people have gotten in the habit of casting their optimization problems (mathematical, engineering, economic) in terms of a single-valued objective function, which is usually a simple weighted sum of the values of the objectives that they really care about.
To unpack my language choices briefly: when designing a vase, you care about its weight, its material cost, its strength, its radius, its height, possibly 50 other things including corrosion resistance and details of manufacturing complexity. To “optimize” the vase design, historically, you needed to come up with a function that smeared away the detail of the problem into one number, something like the “utility” of the vase design.
This is sort of terrible, if you think about it. You sacrifice resolution to make the problem easier to solve, but there’s a serious risk that you end up throwing away what you might have considered to be the global optimum when you do this. You also baked in something like a guess as to what the tradeoffs should be at the Pareto frontier prior to actually knowing what the solution would look like. You know you want the strongest, lightest, cheapest, largest, most beautiful vase, but you can’t have all those things at once, and you don’t really know how those factors trade off against each other until you’re able to hold the result in your hands and compare it to different “optimal” vases from slightly different manifolds. Of course, you can only do that if you accept that you are significantly uncertain about your preferences, meaning the design and optimization process should partly be viewed as an experiment aimed at uncovering your actual preferences regarding these design tradeoffs, which are a priori unknown.
The vase example is both a real example and also a metaphor for how considering humans as agents under the VNM paradigm is basically the same but possibly a million times worse. If you acknowledge the (true) assertion that you can’t really optimize a vase until you have a bunch of differently-optimal vases to examine in order to understand what you actually prefer and what tradeoffs you’re actually willing to make, you have to acknowledge that a human life, which is exponentially more complex, definitely cannot be usefully treated with such a tool.
As a final comment, there is almost a motte-bailey thing happening where Rationalists will say that, obviously, the VNM axioms describe the optimal framework in which to make decisions, and then proceed to never ever actually use the VNM axioms to make decisions.
This relates to my favorite question of economics: are graduate students poor or rich? This post suggests an answer I hadn’t thought of before: it depends on the attitudes of the graduate advisor, and almost nothing else.
Just in case people aren’t aware of this, drilling wells the “old fashioned way” is a very advanced technology. Typically a mechanically complex diamond-tipped tungsten carbide drill bit grinds its way down, while a fluid with precisely calibrated density and reactivity is circulated down the center of the drill string and back up the annulus between the drill string and edges of the hole, sweeping the drill cuttings up the borehole to the surface. A well 4 miles long and 8 inches wide has a volume of over 200,000L, meaning that’s the volume of rock that has to be mechanically removed from the hole during drilling. So that’s the volume of rock you would have to “blow” out of the hole with compressed air. You can see why using a circulating liquid with a reasonably high viscosity is more efficient for this purpose.
The other important thing about drilling fluid is that its density is calibrated to push statically against the walls of the hole as it is being drilled, preventing it from collapsing inward and preventing existing subsurface fluids from gushing into the wellbore. If you tried to drill a hole with no drilling fluid, it would probably collapse, and if it didn’t collapse, it would fill with high pressure groundwater and/or oil and/or explosive natural gas, which would possibly gush straight to the surface and literally blow up your surface facilities. These are all things that would almost inevitably happen if you tried to drill a hole using microwaves and compressed air.
tl;dr, drilling with microwaves might sense if you’re in space drilling into an asteroid, but makes so no sense for this application.
Talking to Golden Gate Claude reminds me of my relationship with my sense of self. My awareness of being Me is constantly hovering and injecting itself into every context. Is this what “self is an illusion” really means? I just need to unclamp my sense of self from its maximum value?
I think it is also good to consider that it’s the good-but-not-great hardware that has the best price-performance at any given point in time. The newest and best chips will always have a price premium. The chips one generation ago will be comparatively much cheaper per unit of performance. This has been generally true since I’ve started recording this kind of information.
As I think I mentioned in another comment, I didn’t mention Moore’s law at all because it has relatively little to do with the price-performance trend. It certainly is easy to end up with a superexponential trend when you have an (economic) exponential trend inside a (technological) exponential trend, but as other commenters point out, the economic term itself is probably superexponential, meaning we shouldn’t be surprised to see price-performance to fall more quickly than exponential even without exponential progress in chip speed.
One way of viewing planning is as an outer-loop on decision theory.
My approach to the general problem of planning skills was to start with decision theory and build up. In my Guild of the Rose Decision Theory courses was to spend time focusing on slowly building the most fundamental skills of decision theory. This included practicing manipulation of probabilities and utilities via decision trees, and practicing all these steps in a variety of both real and synthetic scenarios, to build an intuition regarding the nuances of how to set up decision problems on paper. The ultimate goal was to get the practitioners to the point where they usually don’t need to draw up a decision tree on paper, but rather to leverage those intuitions to quickly solve decision problems mentally, and/or recognize when a decision problem is actually tricky enough to merit breaking out the spreadsheet or Guesstimate project.
In my experience, even long-time rationalists are so incredibly bad at basic decision theory that trying to skip the step of learning to correctly set up a basic decision tree might actually be counterproductive. So my inclination is to focus on really mastering this art before attempting planning.
Another way of viewing planning is that planning is search.
For computationally bounded agents like us, search involves a natural tradeoff of breadth versus depth. Breadth is essentially idea generation, depth is idea selection and refinement. The tricky think about planning, in general, is that if 100x solutions exist, then those solutions are going to be found by spending the majority of the time on breadth-search, i.e. blue sky brainstorming for ways that the plan could look wildly different from the default approach, but that most situations don’t admit 100x plans. Most things in life, especially in our technological civilization, are already sort of optimized, because there is some existing refined solution that has already accommodated the relevant tradeoffs. I could get to work faster if I flew there in a helicopter, but considering in costs, the Pareto optimum is still driving my car on the freeway. Most things look like this. Well-considered Pareto solutions to real-world problems tend to look boring!
Therefor, if you spend a lot of time looking for 100x solutions, you will waste a lot of time, because these solutions usually won’t exist. Then, after failing to find a truly galaxy-brain solution, you will spend some amount of time refining the probably-already-obvious plan, realize that there are a lot of unknown-unknowns, and that the best way to get clarity on these is to just start working. Then you will realize that you would have been better off if you had just started working immediately and not bothered with “planning” at all, and you will either be Enlightened or depressed.
It gives me no pleasure to say this! Ten years ago I was all fired up on the idea that rationalists would Win and take over the world by finding these clever HPJEV-esque lateral thinking solutions. I have since realized that one creative rationalist is usually no match for tens of thousands of smart people exploring the manifold through natural breadth-first and then refining on the best solutions organically.
I am not actually completely blackpilled on the idea of scenario planning. Clearly there are situations for which scenario planning is appropriate. Massive capital allocations and long-term research programs might be two good examples. Even for these types of problems, it’s worth remembering that the manifold probably only admits to marginal optimizations, not 100x optimizations, so you shouldn’t spend too much time looking for them.
Well, there’s your problem!
Hardware Precision TFLOPS Price ($) TFLOPS/$ Nvidia GeForce RTX 4090 FP8 82.58 $1,600 0.05161 AMD RX 7600 FP8 21.5 $270 0.07963 TPU v5e INT8 393 $4730* 0.08309 H100 FP16 1979 $30,603 0.06467 H100 FP8 3958 $30,603 0.12933 * Estimated, sources suggest $3000-6000 From my notes. Your statement about RTX 4090 leading the pack in flops per dollar does not seem correct based on these sources, perhaps you have a better source for your numbers than I do.
I did not realize that H100 had >3.9 PFLOPS at 8-bit precision until you prompted me to look, so I appreciate that nudge. That does put the H100 above the TPU v5e in terms of FLOPS/$. Prior to that addition, you can see why I said TPU v5e was taking the lead. Note that the sticker price for TPU v5e is estimated, partly from a variety of sources, partly from my own estimate calculated from the lock-in hourly usage rates.
Note that FP8 and INT8 are both 8-bit computations and are in a certain sense comparable if not necessarily equivalent.
Could you lay that out for me, a little bit more politely? I’m curious.
Does Roodman’s model concern price-performance or raw performance improvement? I can’t find the reference and figured you might know. In either case, price-performance only depends on Moore’s law-like considerations in the numerator, while the denominator (price) is a a function of economics, which is going to change very rapidly as returns to capital spent on chips used for AI begins to grow.
As I remarked in other comments on this post, this is a plot of price-performance. The denominator is price, which can become cheap very fast. Potentially, as the demand for AI inference ramps up over the coming decade, the price of chips falls fast enough to drive this curve without chip speed growing nearly as fast. It is primarily an economic argument, not a purely technological argument.
For the purposes of forecasting, and understanding what the coming decade will look like, I think we care more about price-performance than raw chip speed. This is particularly true in a regime where both training and inference of large models benefit from massive parallelism. This means you can scale by buying new chips, and from a business or consumer perspective you benefit if those chips get cheaper and/or if they get faster at the same price.
I probably qualify as one of the people you’re describing.
My reasoning is that we are in the fortunate position of having AI that we can probably ask to do our alignment homework for us. Prior to two or three years ago it seemed implausible that we would get an AI that would:
* care about humans a lot, both by revealed preferences and according to all available interpretability evidence
* be quite smart, smarter than us in many ways, but not yet terrifyingly/dangerously smart
But we have such an AI. Arguably we have more than one such. This is good! We lucked out!
Eliezer has been saying for some time that one of his proposed solutions to the Alignment Problem is to shut down all AI research and to genetically engineer a generation of Von Neumanns to do the hard math and philosophy. This path seems unlikely to happen. However, we almost have a generation of Von Neumanns in our datacenters. I say almost because they are definitely not there yet, but I think, based on an informed awareness of LLM development capabilities and plausible mid-term trajectories, that we will soon have access to arbitrarily many copies of brilliant-but-not-superintelligent friendly AIs who care about human wellbeing, and will be more than adequate partners in the development of AI Alignment theory.
I can foresee many objections and critiques of this perspective. On the highest possible level, I acknowledge that using AI to do our AI Alignment homework carries risks. But I think these risks are clearly more favorable to us than the risks we all thought we would be facing in the late part of the early part of the Singularity. For example, what we don’t have is a broadly generally-capable version of AlphaZero. We have something that landed in just the right part of the intelligence space where it can help us quite a lot and probably not kill us all.