I think it would be helpful for helping others calibrate, though obviously it’s fairly personal.
Xodarap
Karma: 755
Possibly too sensitive, but could you share how the photos performed on Photfeeler? Particularly what percentile attractiveness?
Sure, I think everyone agrees that marginal returns to labor diminish with the number of employees. John’s claim though was that returns are non-positive, and that seems empirically false.
We have Wildeford’s Third Law: “Most >10 year forecasts are technically also AI forecasts”.
We need a law like “Most statements about the value of EA are technically also AI forecasts”.
Yep that’s fair, there is some subjectivity here. I was hoping that the charges from SDNY would have a specific amount that Sam was alleged to have defrauded, but they don’t seem to.
Regarding $4B missing: adding in Anthropic gets another $4B on the EA side of the ledger, and founders pledge another $1B. The value produced by Anthropic is questionable, and maybe negative of course, but I think by the strict definition of “donated or built in terms of successful companies” EA comes out ahead.
(And OpenAI gets another $80B, so if you count that then I think even the most aggressive definition of how much FTX defrauded is smaller. But obviously OAI’s EA credentials are dubious.)
EA has defrauded much more money than we’ve ever donated or built in terms of successful companies
FTX is missing $1.8B. OpenPhil has donated $2.8B.
I do think it’s at the top of frauds in the last decade, though that’s a narrower category.
Nikola went from a peak market cap of $66B to ~$1B today, vs. FTX which went from ~$32B to [some unknown but non-negative number].
I also think the Forex scandal counts as bigger (as one reference point, banks paid >$10B in fines), although I’m not exactly sure how one should define the “size” of fraud.[1]
I wouldn’t be surprised if there’s some precise category in which FTX is the top, but my guess is that you have to define that category fairly precisely.
- ^
Wikipedia says “the monetary losses caused by manipulation of the forex market were estimated to represent $11.5 billion per year for Britain’s 20.7 million pension holders alone” which, if anywhere close to true, would make this way bigger than FTX, but I think the methodology behind that number is just guessing that market manipulation made foreign-exchange x% less efficient, and then multiplying through by x%, which isn’t a terrible methodology but also isn’t super rigorous.
- ^
Oh yeah, just because it’s a reference point that doesn’t mean that we should copy them
I think almost any large organization/company would have gone through a much more comprehensive fault-analysis and would have made many measurable improvements.
I claim YCombinator is a counter example.
(The existence of one counterexample obviously doesn’t disagree with the “almost any” claim.)
IMO the EA community has had a reckoning, a post-mortem, an update, etc. far more than most social or political movements would (and do) in response to similar misbehavior from a prominent member
As a reference point: fraud seems fairly common in ycombinator backed companies, but I can’t find any sort of postmortem, even about major things like uBiome where the founders are literally fugitives from the FBI.
It seems like you could tell a fairly compelling story that YC pushing founders to pursue risky strategies and flout rules is upstream of this level of fraudulent behavior, though I haven’t investigated closely.
My guess is that they just kind of accept that their advice to founders is just going to backfire 1-2% of the time.
Thanks for the questions!
I feel a little confused about this myself; it’s possible I’m doing something wrong. (The code I’m using is the `get_prob` function in the linked notebook; someone with LLM experience can probably say if that’s broken without understanding the context.) My best guess is that human intuition has a hard time conceptualizing just how many possibilities exist; e.g. “Female”, “female”, “F”, “f” etc. are all separate tokens which might realistically be continuations.
I haven’t noticed anything; my guess is that there probably is some effect but it would be hard to predict ex ante. The weights used to look up information about “Ben” are also the weights used to look up information about “the Eiffel Tower”, so messing with the former will also mess with the latter, though I don’t really understand how.
A thing I would really like to do here is better understand “superposition”. A really cool finding would be something like: messing with the “gender” dimension of “Ben” is the same as messing with the “architected by” dimension of “the Eiffel Tower” because the model “repurposes” the gender dimension when talking about landmarks since landmarks don’t have genders. But much more research would be required here to find something like that.
My guess is that this is just randomness. It would be interesting to force the random seed to be the same before and after modification and see how much it actually changes.
Thanks! I mentioned anthropic in the post, but would similarly find it interesting if someone did a write up about cohere. It could be that OAI is not representative for reasons I don’t understand.
Yep, revenue multiples are a heuristic for expectations of future growth, which is what I care about
This is true, but I’m not aware of any investments on $0 revenue at the $10B scale. Would love to hear of counterexamples if you know of any![1]
- ^
Instagram is the closest I can think of, but that was ~20x smaller and an acquisition, not an investment
I tried playing the game Nate suggested with myself. I think it updated me a bit more towards Holden’s view, though I’m very confident that if I did it with someone who was more expert than I am both the attacker and the defender would be more competent, and possibly the attacker would win.
Attacker: okay, let’s start with a classic: Alignment strategy of “kill all the capabilities researchers so networks are easier to interpret.”
Defender: Arguments that this is a bad idea will obviously be in any training set that a human level AI researcher would be trained on. E.g. posts from this site.Attacker: sure. But those arguments don’t address what an AI would be considering after many cycles of reflection. For example: it might observe that Alice endorses things like war where people are killed “for the greater good”, and a consistent extrapolation from these beliefs is that murder is acceptable.
Defender: still pretty sure that the training corpus would include stuff about the state having a monopoly on violence, and any faithful attempt to replicate Alice would just clearly not have her murdering people? Like a next token prediction that has her writing a serial killer manifesto would get a ton of loss.Attacker: it’s true that you probably wouldn’t predict that Alice would actively murder people, but you would predict that she would be okay with allowing people to die through her own inaction (standard “child drowning in a pond” thought experiment stuff). And just like genocides are bureaucratized such that no single individual feels responsible, the AI might come up with some system which doesn’t actually leave Alice feeling responsible for the capabilities researchers dying.
(Meta point: when does something stop being POUDA? Like what if Alice’s CEV actually is to do something wild (in the opinion of current-Alice)? I think for the sake of this exercise we should not assume that Alice actually would want to do something wild if she knew/reflected more, but this might be ignoring an important threat vector?)
Defender: I’m not sure exactly what this would look like, but I’m imagining something like “build a biological science company that has an opaque bureaucracy such that each person pursuing the goodwill somehow result in the collective creating a bio weapon that kills capabilities researchers” and this just seems really outside what you would expect Alice to do? I concede that there might not be anything in the training set which specifically prohibits this per se, but it just seems like a wild departure from Alice’s usual work of interpreting neurons.(Meta: is this Nate’s point? Iterating reflection will inevitably take us wildly outside the training distribution so our models of what an AI attempting to replicate Alice would do are wildly off? Or is this a point for Holden: the only way we can get POUDA is by doing something that seems really implausible?)
Yeah that’s correct on both counts (that does seem like an important distinction, and neither really match my experience, though the former is more similar).
I spent about a decade at a company that grew from 3,000 to 10,000 people; I would guess the layers of management were roughly the logarithm in base 7 of the number of people. Manager selection was honestly kind of a disorganized process, but it was basically: impress your direct manager enough that they suggest you for management, then impress your division manager enough that they sign off on this suggestion.
I’m currently somewhere much smaller, I report to the top layer and have two layers below me. Process is roughly the same.
I realized that I should have said that I found your spotify example the most compelling: the problems I see/saw are less “manager screws over the business to personally advance” but rather “helping the business would require manager to take a personal hit, and they didn’t want to do that.”
For what it’s worth, I think a naïve reading of this post would imply that moral mazes are more common than my experience indicates.
I’ve been in middle management at a few places, and in general people just do reasonable things because they are reasonable people, and they aren’t ruthlessly optimizing enough to be super political even if that’s the theoretical equilibrium of the game they are playing.[1]
- ^
This obviously doesn’t mean that they are ruthlessly optimizing for the company’s true goals though. They are just kind of casually doing things they think are good for the business, because playing politics is too much work.
- ^
FYI I think your first skepticism was mentioned in the safety from speed section; she concludes that section:
These [objections] all seem plausible. But also plausibly wrong. I don’t know of a decisive analysis of any of these considerations, and am not going to do one here. My impression is that they could basically all go either way.
She mentions your second skepticism near the top, but I don’t see anywhere she directly addresses it.
One of the authors has now posted about this here
Thanks for writing this up! I wonder how feasible it is to just do a cycle of bulking and cutting and then do one of body recomposition and compare the results. I expect that the results will be too close to tell a difference, which I guess just means that you should do whichever is easier.