StellaAthena

RAND re­port finds no effect of cur­rent LLMs on vi­a­bil­ity of bioter­ror­ism attacks

• StellaAthena 30 Nov 2021 15:32 UTC

  57 points

  on: Visi­ble Thoughts Pro­ject and Bounty Announcement

  Hi! Co-author of the linked “exploration” here. I have some reservations about the exact request (left as a separate comment) but I’m very excited about this idea in general. I’ve been advocating for direct spending on AI research as a place with a huge ROI for alignment research for a while and it’s very exciting to see this happening.

  I don’t have the time (or aptitude) to produce a really high quality dataset, but I (and EleutherAI in general) would be happy to help with training the models if that’s desired. We’d be happy to consult on model design or training set-up, or to simply train the models for you all. No compensation necessary, just excited to contribute to worthwhile alignment research.

• StellaAthena 30 Nov 2021 12:59 UTC

  51 points

  on: Visi­ble Thoughts Pro­ject and Bounty Announcement

  What is the purpose of requesting such extremely long submissions? This comes out to ~600 pages of text per submission, which is extremely far beyond anything that current technology could leverage. Current NLP systems are unable to reason about more than 2048 tokens at a time, and handle longer inputs by splitting them up. Even if we assume that great strides are made in long-range attention over the next year or two, it does not seem plausible to me to anticipate SOTA systems in the near future to be able to use this dataset to its fullest. There’s inherent value in a more diverse set of scenarios, given the strong propensity of language models to overfit on repeated data. While this isn’t strictly speaking talking about repeating data, I am under the strong impression that having more diverse short scripts is going to train a much better mode than less diverse long scripts, assuming that the short scripts are still at or beyond the maximum context length a language model can handle.

  For the same reasons it is challenging to leverage, I think that this will also be very challenging to produce. I think that changing the request to 100 different 6 page (10 step) or 10 different 60 page (100 step) stories would be a) much easier to produce and b) much more likely to actually help train an AI. It also allows you to pear down the per-submission payouts, assuaging some concerns in the comments about the winner-take-all and adversarial nature of the competition. If you offer $20 per 10-step story for 1,000 stories it greatly reduces the chances that someone will end up spending a ton of effort but be unable to get it in on time for the reward.

  To put the length of this in prospective, a feature length movie script is typically around 100-130 pages. The ask here is to write 1-2 novels, or 5-6 movie scripts. That’s a massive amount of writing, and not something anyone can complete quickly.

  What links here?

  • StellaAthena's comment on Visi­ble Thoughts Pro­ject and Bounty Announcement by So8res (30 Nov 2021 15:32 UTC; 57 points)
  • Jared Kaplan's comment on Visi­ble Thoughts Pro­ject and Bounty Announcement by So8res (1 Dec 2021 4:33 UTC; 23 points)

• StellaAthena 30 Aug 2023 3:40 UTC

  49 points

  24

  on: In­tro­duc­ing the Cen­ter for AI Policy (& we’re hiring!)

  CAIP is also advised by experts from other organizations and is supported by many volunteers.

  Who are the experts that advise you? Are claims like “our proposals will not impede the vast majority of AI developers” vetted by the developers you’re looking to avoid impacting?

• StellaAthena 17 May 2022 2:25 UTC

  33 points

  on: Is AI Progress Im­pos­si­ble To Pre­dict?

  Individual MMMLU tasks are extremely noisy. They’re so noisy that the paper actually specifically recommends that you don’t draw conclusions from performance on individual tasks and instead look at four high level topical categories. The individual tasks also have extremely large variances in their variance. Some of them are pretty easy for a college educated adult, while others have genuine experts scoring less than 80%.

  This is compounded by the fact that the sample sizes vary wildly. While many of the tasks have around 100 questions, while at the other extreme there is a task with 1534 questions. The aggregated topics however have the same number of questions per topic, because the task was explicitly designed for analysis along those lines.

  I don’t know the extent to which these issues plague the other evaluations, but I think more care needs to be taken before drawing conclusions with highly noisy data.

• StellaAthena 2 Dec 2021 5:52 UTC

  30 points

  in reply to: Eliezer Yudkowsky’s comment on: Visi­ble Thoughts Pro­ject and Bounty Announcement

  1: I expect that it’s easier for authors to write longer thoughtful things that make sense;

  I pretty strongly disagree. The key thing I think you are missing here is parallelism: you don’t want one person to write you 100 different 600 page stories, you one person to organize 100 people to write you one 600 page story each. And it’s a lot easier to scale if you set the barrier of entry lower. There are many more people who can write 60 page stories than 600 page stories, and it’s easier to find 1,000 people to write 60 pages each than it is to find 100 people to write 600 pages each. There’s also much less risk on both your side and theirs. If someone drops out half way through writing you lose 30 pages not 300.

  Based on this comment:

  I state: we’d be happy, nay, ecstatic, to get nice coherent complete shorter runs, thereby disproving my concern that short runs won’t be possible to complete, and to pay for them proportionally.

  I’m now under the impression that you’d be willing to pay out the 20k for 10 runs of 100 steps each (subject to reasonable quality control) and bringing that about was my main goal in commenting.

  The other major worry I have about this pitch is the experimental design. I’m still happy you’re doing this, but this doesn’t seem to be the best project crafting in my mind. Briefly my concerns are:

  1. This is a very topically specific ask of unclear generalization. I would prefer a more generic ask that is not directly connected to D&D.

  2. In my experience training large language models, the number of examples is more important than the length of examples. Training on 100 shorter sequences is better than training on 10 longer sequences if the total length is the same. In particular, I think “You would also expect scarier systems to have an easier time learning without overnarrowing from 100 big examples instead of 10,000 small examples.” is not clearly true and very plausibly false.

  3. Using this dataset in a meaningful fashion requires making a priori unrelated breakthroughs, making it overly inaccessible. I think that your comment “I don’t want to freeze into the dataset the weird limitations of our current technology, and make it be useful only for training dungeons that are weird the same way 2021 dungeons are weird,” is thinking about this the wrong way. The goal should be to maximize the time that we can effectively use this dataset, not be content with the fact that one day it will be useful.

  4. This is a pilot for the real thing you’re after, but the “pilot” is a multi-year million-dollar effort. That doesn’t seem like a very well designed pilot to me.

• StellaAthena 4 May 2023 23:48 UTC

  23 points

  2

  in reply to: M. Y. Zuo’s comment on: White House An­nounces “New Ac­tions to Pro­mote Re­spon­si­ble AI In­no­va­tion”

  Hi, I’m helping support the event. I think that some mistranslation happened by a non-AI person. The event is about having humans get together and do prompt hacking and similar on a variety of models side-by-side. ScaleAI built the app that’s orchestrating the routing of info, model querying, and human interaction. Scale’s platform isn’t doing the evaluation itself. That’s being done by users on-site and then by ML and security researchers analyzing the data after the fact.

• StellaAthena 2 Oct 2021 19:16 UTC

  23 points

  in reply to: Elizabeth’s comment on: The LessWrong Team is now Light­cone In­fras­truc­ture, come work with us!

  This response confuses me.

  1. Who is being punished here? I see people leaving feedback and discussing ideas, and have no idea who you are worried about.

  2. I strongly agree with AI_WAIFU, but don’t have a useful general strategy for non-profit funding. My opposition is based on a simple heuristic: wealthy orgs should not systematically underpay their employees. Making a thread saying that seems extremely not useful.

  Speaking to the general point, as AI_WAIFU points out, there is an extremely large amount of money apparently sitting around. The thread he links to implies that EA has about 5 million dollars per active member of the community and that cash is growing faster than membership. That’s an obscene amount of cash, and being stingy about pay doesn’t really make sense to me.

  Others in this thread have brought up the fact that many non-profit underpay, but that’s not because there’s some kind of virtue in underpaying (quite the opposite: it’s exploitive), it’s because they’re poor. EA is apparently are swimming in cash, so that comparison doesn’t make much sense here. Additionally, many non-profits compensate for underpaying with extremely generous benefits, which this post makes no mention of.

  “We pay less than you’re worth because we only want people who really care about the mission” is typically a lie HR tells people, not an actual thing people believe. Reading that it’s a thing that Lightcone believes worries me, as it makes me feel like you’re drinking your own Kool-Aide too hard.

  This also signals that you don’t care about your employees. Pay is the number one way orgs indicate that they care about their employees.

• StellaAthena 24 Nov 2021 18:43 UTC

  17 points

  in reply to: gwern’s comment on: Yud­kowsky and Chris­ti­ano dis­cuss “Take­off Speeds”

  For Sanh et al. (2021), we were able to negotiate access to preliminary numbers from the BIG Bench project and run the T0 models on it. However the authors of Sanh et al. and the authors of BIG Bench are different groups of people.

• StellaAthena 20 Aug 2015 8:49 UTC

  15 points

  on: 0 And 1 Are Not Probabilities

  This article is largely incoherent. The main justification is the abuse of an invalid transformations: y=x/​(1-x) is not the bijection that he asserts it is, because it’s not a function that maps [0,1] onto R. It’s a function that maps [0,1] onto [1,\intfy] as a subset of the topological closure of R. And that’s okay, but you can’t say “well I don’t like the topological closure of R, so I’ll just use R and claim that 1 is where the problem is.”

  Additionally, his discussion of log odds and such is perfectly fine, but ignores the fact that there are places where you do need to have an odds of 0:1, or a log odds of negative infinity. Probability theory stops working when you throw out 0 and 1, it’s as simple as that.

  Even if you don’t want to handle tautologies or contradictions, there are other ways to get P(X)=0 or 1. The probability that a real number chosen uniformly from the real interval [0,1] is 0. It has to be. It’s a provable fact under ZFC and to decide otherwise is to say that you’re more attached to the idea of 0 and 1 not being probabilities than you are to the fact that mathematics is consistent and if you really believe that, well, there’s absolutely nothing I have to say to you.

  This is one of those situations where EY just demonstrates he knows very little mathematics.

The View from 30,000 Feet: Pre­face to the Se­cond EleutherAI Retrospective

• StellaAthena 30 Aug 2023 5:52 UTC

  14 points

  0

  in reply to: Holly_Elmore’s comment on: In­tro­duc­ing the Cen­ter for AI Policy (& we’re hiring!)

  Nora didn’t say that this proposal is harmful. Nora said that if Zach’s explanation for the disconnect between their rhetoric and their stated policy goals is correct (namely that they don’t really know what they’re talking about) then their existence is likely net-harmful.

  That said, yes requiring everyone who wants to finetune LLaMA 2 get a license would be absurd and harmful. la3orn and gallabyres articulate some reasons why in this thread.

  Another reason is that it’s impossible to enforce, and passing laws or regulations and then not enforcing them is really bad for credibility.

  Another reason is that the history of AI is a history of people ignoring laws and ethics so long as it makes them money and they can afford to pay the fines. Unless this regulation comes with fines so harsh that they remove all possibility of making money off of models, OpenAI et al. won’t be getting licenses. They’ll just pay the fines while small scale and indie devs (who allegedly the OP is specifically hoping to not impact) screech their work to a halt and wait for the government to tell them it’s okay for them to continue to do their work.

  Also, such a regulation seems like it would be illegal in the US. While the government does have wide latitude to regulate commercial activities that impact multiple states, this is rather specifically a proposal that would regulate all activity (even models that never get released!). I’m unaware of any precedent for such an action, can you name one?

• StellaAthena 6 Jun 2023 22:23 UTC

  13 points

  7

  on: Terry Tao is host­ing an “AI to As­sist Math­e­mat­i­cal Rea­son­ing” workshop

  I don’t understand the community obsession with Tao and recruiting him to work on alignment. This is a thing I hear about multiple times a year with no explanation of why it would be desirable other than “he’s famous for being very smart.”

  I also don’t see why you’d think there’s be an opportunity to do this… it’s an online event, which heavily limits the ability to corner him in the hallway. It’s not even clear to me that you’d have an opportunity to speak with him… he’s moderating several discussions and panels, but any submitted questions to said events would go to the people actually in the discussions not the moderator.

  Can you elaborate on what you’re actually thinking this would look like?

• StellaAthena 21 Feb 2023 17:38 UTC

  13 points

  8

  on: Ba­sic facts about lan­guage mod­els dur­ing training

  This is really exciting work to see, and exactly the kind of thing I was hoping people would do when designing the Pythia model suite. It looks like you’re experimenting with the 5 smallest models, but haven’t done analysis on the 2.8B, 6.9B, or 12B models. Is that something you’re planning on adding, or no?

  I am really very surprised that the distributions don’t seem to match any standard parameterized distribution. I was fully ready to say “okay, let’s retrain some of the smaller Pythia models initialized using the distribution you think the weights come from” but apparently we can’t do that easily. I suppose we can do a MCMC sampler? In general, it seems like a natural follow-up to the contents of this post is to change the way we initialize things in models, retrain them, and see what happens (esp. with the loss curve). If that’s something you’d like to collaborate with EleutherAI about, I would be more than happy to arrange something :)

  In general, the reliability of the things you’re seeing across model scales is really cool. I agree that it seems to refute some of the theoretical assumptions of the NTK literature, but I wonder if perhaps it’s consistent with the Tensor Programs work by Greg Yang et al. that lead to muP.

  To clarify what’s going on with the Pythia models:

  1. This work appears to be using the initial model release, which has an inconsistent naming scheme. Some models were named based on total parameters, while others were named based on the number of learnable parameters. The former is what models are typically named based on, but the later is what people put on the x-axis of scaling laws plots. This is a nomenclature change only with no impact on results.

  2. Shortly after release, we renamed the models to be consistently named using the total number of parameters. The models studied in this post are currently named 70M, 160M, 410M, 1B, and 1.4B.

  3. When writing the paper for these models, we discovered a handful of inconsistencies in the suite’s hyperparameters. Specifically, the batch size and some all-reduce optimizations were inconsistent across training. We expect this to have no impact on the OP or 90% of experiments using the suite. That said, if we’re going to spend all this compute to design a suite for controlled scientific experiments, it should control for as many factors as possible. The current models will remain public and people are encouraged to compare results across them to further validate that various properties don’t impact the behavior that they’re finding.

• StellaAthena 4 Jan 2023 15:13 UTC

  LW: 12 AF: 6

  9

  AF

  on: Ba­sic Facts about Lan­guage Model Internals

  I’m not sure when you developed this work, but the LLM.int8 paper identifies outliers as an essential factor in achieving performance for models larger than 2.7B parameters (see Fig. 1 and Fig. 3 especially). There’s also some follow-up work here and here. Very curiously, the GLM-130B paper reports that they don’t see outlier features at all, or the negative effects of their lack of impact.

  I’ve spoken with Tim (LLM.int8 lead author) about this a bit and some people in EleutherAI, and I’m wondering if there’s some kind of explicit or implicit regularizing effect in the GLM model that prevents it from learning outlier features. If this is the case, one might expect to find different patterns in outliers in models with sufficiently different architecture, perhaps GPT-2 vs Pythia vs GLM vs T5

• StellaAthena 17 Feb 2022 18:09 UTC

  11 points

  AF

  on: Com­pute Trends Across Three eras of Ma­chine Learning

  The distinction between “large scale era” and the rest of DL looks rather suspicious to me. You don’t give a meaningful defense of which points you label “large scale era” in your plot and largely it looks like you took a handful of the most expensive models each year to give a different label to.

  On what basis can you conclude that Turing NLG, GPT-J, GShard, and Switch Transformers aren’t part of the “large scale era”? The fact that they weren’t literally the largest models trained that year?

  There’s also a lot of research that didn’t make your analysis, including work explicitly geared towards smaller models. What exclusion criteria did you use? I feel like if I was to perform the same analysis with a slightly different sample of papers I could come to wildly divergent conclusions.

• StellaAthena 11 Nov 2021 19:21 UTC

  LW: 10 AF: 3

  AF

  in reply to: Ben Pace’s comment on: Dis­cus­sion with Eliezer Yud­kowsky on AGI interventions

  If superintelligence is approximately multimodal GPT-17 plus reinforcement learning, then understanding how GPT-3-scale algorithms function is exceptionally important to understanding super-intelligence.

  Also, if superintelligence doesn’t happen then prosaic alignment is the only kind of alignment.

• StellaAthena 3 Oct 2021 21:22 UTC

  10 points

  in reply to: ChristianKl’s comment on: The LessWrong Team is now Light­cone In­fras­truc­ture, come work with us!

  Why is this problem better solved by systematically underpaying everyone as opposed to firing people who act “in favor of what advances their own power” or who promote infighting?

• StellaAthena 9 Sep 2021 22:55 UTC

  10 points

  in reply to: Davidmanheim’s comment on: Sam Alt­man Q&A Notes—Aftermath

  When was it stated that the talk was off the record? You seem to be the only person in this thread (myself included) who remembers that.

• StellaAthena 25 Jan 2024 21:38 UTC

  9 points

  0

  in reply to: habryka’s comment on: RAND re­port finds no effect of cur­rent LLMs on vi­a­bil­ity of bioter­ror­ism attacks

  Thanks! I like your title more :)