Researcher #3ainstitute, interdisciplinary, interested in everything; ongoing PhD in CS (learning/testing/verification), open sourcerer, more at https://zhd.dev
Zac Hatfield Dodds(Zac Hatfield-Dodds)
Karma: 74
Some factors which I think are both important and missing from your model:
Risk. You probably cannot convince me that, in a liquid market, your outperforming trading strategy does not round to “picking up pennies in front of a steamroller”.
Availability of capital. If you have to lock up $10K for a year per 20-hours-of-research deal, you’re probably more constrained by money than time.
Opportunity costs. If you have sufficient quant and business skills to make money trading, you can probably make more working somewhere and investing the proceeds in index funds.
Transaction costs, taxes, etc.
On the other hand, there’s some suggestive evidence that seed-stage returns have a power-law distribution with - implying that the best strategy is to filter out the obvious duds and then invest in literally everything else.
Worldwide demand should be easily big enough to justify [subcontracting manufacturing]
If it was legal to sell vaccines for the market price, or anywhere near their actual value, of course. Thanks to monopsony purchasers (i.e. irrationally cheap governments), we instead see massive underproduction.
The hypothesis that Australia succeeded because it was using good epistemics to make decisions is not holding up well in the endgame.
From Australia, this hypothesis was only ever plausible if you looked at high-level outcomes rather than the actual decision-making.
We got basically one thing right: pursue local elimination. Without going into details, this only happened because the Victorian state government unilaterally held their hard lockdown all the way back to nothing-for-two-weeks, ending our winter second wave. Doing so created both a status quo and (having paid higher-than-if-faster costs) a very strong constituency for elimination.
Victoria remains the only area with non-neglibible masking. Nationwide, we continue to make expensive and obvious mistakes about handwashing, distancing, quarantine, and appear to be bungling our vaccine rollout.
Zero active cases and zero local transmission covers a multitude of sins. I attribute the result as much to good luck as epistemic skill, and am very glad that COVID is not such a hard problem that we can’t afford mistakes.
Important for what? Best for what?
In a given (sub)field, the highest-cited papers tend to be those which introduced or substantially improved on a key idea/result/concept; so they’re important in that sense. If you’re looking for the best introduction though that will often be a textbook, and there might be important caveats or limitations in a later and less-cited paper.
I’ve also had a problem where a few highly cited papers propose $approach, many papers apply or puport to extend it, and then eventually someone does a well-powered study checking whether $approach actually works. Either way that’s an important paper, but they tend to be under-cited either because either the results are “obvious” (and usually a small effect) or the field of $approach studies shrinks considerably.
It’s an extremely goodhartable metric but perhaps the best we have for papers; for authors I tend to ask “does this person have good taste in problems (important+tractable), and are their methods appropriate to the task?”.
Excellent question!
I’m not personally concerned about what Bostrom called ‘risks of irrationality and error’ or ‘risks to valuable states and activities’. There are costs of rationality though, where knowing just a little can expose you to harms that you’re not yet equipped to handle (classic examples: scope sensitivity, demandingness, death). This rounds to common sense - ‘be sensitive about when/whether/how to discuss upsetting topics’.
Mostly though, I’m inclined to keep quiet about data, idea, and attention hazards where my teenage self might have wanted to share interesting ideas like the antibiotic-gradient trick, at least without some benefit beyond having a fun discussion. Threat models for election security, yes—there’s a clear public interest in everyone understanding the tradeoffs involved in paper vs electronic ballots, or remote vs polling-place voting. Ideas for asymmetric warfare, not so much.
At a more concrete level, I’ve spent the last ~14 months holding strong and unusual views on most pandemic-related matters, though I don’t think any of them would raise eyebrows on LessWrong. A minority are probably now mainstream, the others—unfortunately—remain weird.
Taking information hazards seriously.
This can range from the benign (is it a good idea to post very weird beliefs here?) to the more worrying (plausible attacks on $insert_important_system_here), and upwards.
youvegottobekiddingme
The entire project description is full of “we will”, “we aim to”, “we are creating”… without visible evidence that the project has actually made a novel technical thing, I tend to assume that it’s just a cash grab using a pile of buzzwords.
For a value of “break into flames” that matches damp and poorly-oxygenated fuel, yep! This case in Australia is illustrative; you tend to get a lot of nasty smoke rather than a nice campfire vibe.
You’d have to mismanage a household-scale compost pile very badly before it spontaneously combusts, but it’s a known and common failure mode for commercial-scale operations above a few tons. Specific details about when depend a great deal on the composition of the pile; with nitrate filmstock it was possible with as little as a few grams.
It’s at least syntatically-valid word salad composed of relevant words, which is a substantial advance—and per Gwern, I’m very cautious about generalising from “the first few results from this prompt are bad” to “GPT can’t X”.
Testing in full generality is certainly AGI-complete (and a nice ingredient for recursive self-improvement!), but I think you’re overestimating the difficulty of pattern-matching your way to decent tests. Chess used to be considered AGI-complete too; I’d guess testing is more like poetry+arithmetic in that if you can handle context, style, and some details it comes out pretty nicely.
I expect GPT-4 to be substantially better at this ‘out of the box’ due to
the usual combination of larger, better at generalising, scaling laws, etc.
super-linear performance gains on arithmetic-like tasks due to generalisation, with spillover to code-related tasks
the extra github (and blog, etc) data is probably pretty helpful given steady adoption since ~2015 or so
Example outputs from Ghostwriter vs GPT-3:
$ hypothesis write gzip.compress import gzip from hypothesis import given, strategies as st @given(compresslevel=st.just(9), data=st.nothing()) def test_roundtrip_compress_decompress(compresslevel, data): value0 = gzip.compress(data=data, compresslevel=compresslevel) value1 = gzip.decompress(data=value0) assert data == value1, (data, value1)while GPT-3 tends to produce examples like (first four that I generated just now):
@given(st.bytes(st.uuid4())) def test(x): expected = x result = bytes(gzip(x)) assert bytes(result) == expected @given(st.bytes()) def test_round_trip(xs): compressed, uncompressed = gzip_and_unzip(xs) assert is_equal(compressed, uncompressed) @given(st.bytes("foobar")) def test(xs): assert gzip(xs) == xs @given(st.bytes()) def test(xs): zipped_xs = gzip(xs) uncompressed_xs = zlib.decompress(zipped_xs) assert zipped_xs == uncompressed_xsSo it’s clearly ‘getting the right idea’, even without any fine-tuning at all, but not there yet. It’s also a lot worse at this without a natural-language description of the test we want in the prompt:
This document was written by a very smart AI programmer, who is skilled in testing Python, friendly, and helpful. Let's start by testing the two properties of sorting a list: the output is a permutation of the input, and the elements of the output are ordered: @given(st.lists(st.integers())) def test(xs): result = sorted(xs) assert set(result) == set(xs) assert is_in_order(result) Our second test demonstrates a round-trip test. Given any string of bytes, if you use gzip to compress and then decompress, the result will be equal to the input:Of course translating natural language into Python code is pretty cool too!
Reason about code.
Specifically, I’ve been trying to get GPT-3 to outperform the Hypothesis Ghostwriter in automatic generation of tests and specifications, without any success. I expect that GPT-4 will also underperform; but that it could outperform if fine-tuned on the problem.
If I knew where to get training data I’d like to try this with GPT-3 for that matter; I’m much more attached to the user experience of ”
hypothesis write mypackagegenerates good tests” than any particular implementation (modulo installation and other managable ops issues for novice users).
Considered as an epistemology, I don’t think you’re missing anything.
To reconstruct Popperian falsification from Bayes, see that if you observe something that some hypothesis gave probability ~0 (“impossible”), that hypothesis is almost certainly false—it’s been “falsified” by the evidence. With a large enough hypothesis space you can recover Bayes from Popper - that’s Solomonoff Induction—but you’d never want to in practice.
For more about science—as institution, culture, discipline, human activity, etc. - and ideal Bayesian rationality, see the Science and Rationality sequence. I was going to single out particular essays, but honestly the whole sequence is probably relevant!
With the capital I have on hand as a PhD student, there’s just no way that running something like Vitalik’s pipeline to make money on prediction markets will have a higher excess return-on-hours-worked over holding ETH than my next-best option (which I currently think is a business I’m starting).
If I was starting with a larger capital pool, or equivalently a lower hourly rate, I can see how it would be attractive though.
I don’t use Polymarket because, relative to a material investment,
The base rate of crypto institutions losing everything through hacks, fraud, etc. is way too high
On a related note, “USDC” / “Tether” does not inspire confidence
Conversely, volatility—“it would have been a shame if I earned $50,000 betting against Trump but simultaneously lost $500,000 missing out on ETH price changes”
More generally, I haven’t yet seen a prediction market where the “easy money” looks more attractive on a risk-and-work-adjusted basis than working on HypoFuzz. Perhaps others have similar opportunity costs?
Don’t forget inventions: in the long run, changing the set of available goods and services has been even more important (!) than improving their distribution. Notable post-WWII examples include high-yield cereal varieties, smallpox and polio vaccines, everything made with semiconductors and all the services they enable...
An alternative framing of this project (for global utilitarians) is “the project of explaining our moral intuitions and other ethical considerations” ala rule utilitiarianism. I don’t see this as a purely empirical project though; there’s also a lot of conceptual work which philosophy is well equipped for. That said,
those inclined to stop at the basic set seem more likely to sit down in the armchair for a bit, answer their central questions with reference to the basic set, then get up and leave — in search of further, empirical information and opportunity relevant to promoting their basic values in practice
rings very true. I enrolled in a double degree in arts/science, including philosophy, but by the end of second year writing another essay exploring how _____ arises from utilitarianism was a lot less enticing than a semester studying {cybernetics, climate, supercomputing, remote sensing, …} and so I graduated with a degree in interdisciplinary sciences instead.
It seems to me that your post is missing something: what specifically do you want people to learn?
I am an educator—currently teaching CS research and Python for a general audience—and I find it’s easy to get people to learn… and surprisingly hard to have everyone learn the same specific thing (to a degree they can use, not just repeat on a test <=2 weeks later). Circumstances to discover and apply the ideas for themselves are best, followed by a variety of communications strategies like the videos, diagrams, and docs you mention.
For code-quality I think you’re asking “how do I help colleagues with different expertise work with this code”, and to me that calls for thinking about communication rather than education. Education includes communication, but it’s larger, slower, harder, and fortunately not necessary here :-)
I don’t think we’re actually disagreeing much about outcomes (which I agree have been great!), or even that Australia has competently executed at least enough of the important things to get right. Of the five items you mention I’d include borders, quarantine, snap-lockdowns, and testing as part of the local elimination policy; we haven’t done them perfectly but we have done them well enough.
I understand “using good epistemics to make decisions” to require that your decisions should be made based on a coherent understanding and cost-benefit analysis of the situation, even if both might change over time. “Merely” getting good outcomes doesn’t count!
For example, we still encourage pointless handwashing and distancing while iffy on masks or ventilation—and because we got to zero transmission in other ways, that’s OK. Similarly, it’s true that Australia’s slow vaccine rollout hasn’t cost many lives so far and I hope that neither winter nor variants change that. The cost-in-expectation of an unlikely outbreak should still drive faster vaccination efforts IMO, especially from resources that can’t be used elsewhere.