Dan Elton blog: https://moreisdifferent.substack.com/ website: http://www.moreisdifferent.com twitter: https://twitter.com/moreisdifferent
delton137
Karma: 267
I want to point out that there has been some very small amounts of progress in the last 10 years on the problem of moving from connectome to simulation rather than no progress.
First, there has been interesting work at the JHU Applied Physics Lab which extends what Busbice was trying to do when he tried to run as simulation of c elegans in a Lego Mindstorms robot (by the way, that work by Busbice was very much overhyped by Busbice and in the media, so it’s fitting that you didn’t mention it). They use a basic integrate and fire model to simulate the neurons (which is probably actually not very accurate here because c elegans neurons don’t actually seem to spike much and seem to rely on subthreshold activity more so than in other organisms). To assign weights to the different synapses they used what appears to be a very crude metric—the weight was determined in proportion to the total number of synapses the two neurons on either side of the synapse share. Despite the crudeness of their approach, the simulated worm did manage to reverse it’s direction when bumping into walls. I believe this work was a project that summer interns did and didn’t have a lot of funding, which makes it more impressive in my mind than it might seem at first glance.
Another line of work that seems worth pointing out is this 2018 work simulating “hexagonal cells” in the drosophilia which has been done at Janelia: “A Connectome Based Hexagonal Lattice Convolutional Network Model of the Drosophila Visual System”. They claim “Our work is the first demonstration, that knowledge of the connectome can enable in silico predictions of the functional properties of individual neurons in a circuit”. I skimmed this paper and found it a bit underwhelming since it appears the validation of the model was mostly in terms of summary statistics.
Finally, for anyone who wants to learn what happened with the OpenWorm project, the CarbonCopies Foundation did a workshop in June 2021 with Steven Larson. A recording of the 4 hour event is online. I was present for a bit of it at the time it aired, but my recollection is dim. I believe part of the issue they ran into was figuring out how to simulate the physiology of the neuron (ie all the non-neuronal cells). Some people in the OpenWorm open source community managed to build a 3D model (you can view it here). If I recall correctly, he mentioned there was some work to embed that model in a fluid dynamics simulation and “wire it” with a crude simulation of the nervous system, and they got it to wiggle in some way that looked plausible.
I think a bit too much mindshare is being spent on these sci-fi scenario discussions, although they are fun.
Honestly I have trouble following these arguments about deception evolving in RL. In particular I can’t quite wrap my head around how the agent ends up optimizing for something else (not a proxy objective, but a possibly totally orthogonal objective like “please my human masters so I can later do X”). In any case, it seems self awareness is required for the type of deception that you’re envisioning. Which brings up an interesting question—can a purely feed-forward network develop self-awareness during training? I don’t know about you, but I have trouble picturing it happening unless there is some sort of loop involved.
Here’s some updates on this:
We have two Facebook event pages created for this:
https://www.facebook.com/events/208338324360886 (35 RSVPs)
https://www.facebook.com/events/1028113637697900 (18 RSVPs)
This is great, but we need more people. It might be worth gently reminding people that it only takes a few minutes to set up a Twitter account.
We have some big Twitter influencers who have signaled they are on our side but haven’t yet used our hashtags. They should be our primary targets to get involved:Matthew Yglesias − 498k followers - ( https://twitter.com/mattyglesias ) (see https://twitter.com/mattyglesias/status/1360578252172128260 https://twitter.com/mattyglesias/status/1356935647580327937 )
Nate Silver − 3.6 M followers ( https://twitter.com/NateSilver538 ) - retweeted Matt Yglesias’s tweet and agreed J&J approval is too slow
Conor Friedersdorf − 74.8k followers (https://twitter.com/conor64 ) - staff writer at The Atlantic, many tweets on approving AZ
Alex Tabarrok − 44.4k followers ( https://twitter.com/ATabarrok )
Robert Zubrin − 13k followers ( https://twitter.com/robert_zubrin ) (see https://twitter.com/robert_zubrin/status/1359712941902299136?s=20 etc)
I have compiled a more complete list of Twitter accounts that are verified on our side (at least for #ApproveAstraZeneca) here:
https://twitter.com/i/lists/1356638243588956163
The key is that everyone tweet out unique content and commentary. So, post your own, original tweets and then comment on other people’s tweets. Twitter’s algorithms are pretty smart. Simply copy/pasting or retweeting stuff isn’t going to work. Everyone should seek to provide their own angle / frame on the issue. We can almost consider it like an evolutionary experiment to see which frames go the most viral. Here are some ideas, building off the list the OP provided:
Britain has already distributed N AstraZeneca vaccines with x affect. (ie show the very clear graphs and stats on drops in deaths and cases)
People should be free to take medicines to protect themselves (ie provide a libertarian and/or utilitarian argument for Free to Choose medicine / bodily autonomy, etc).
A rapidly evolving crisis requires ability to take decisive actions. FDA is not equipped to make such actions—so we need Biden to step in and longer term reforms so the FDA can take swift action in emergencies .
Oregon just legalized all hard drugs. Can the FDA legalize the AstraZeneca vaccine? (great thing to target at Libertarians. See this very popular tweet https://twitter.com/ATabarrok/status/1357517719852232705)
FDA doesn’t allow companies to pre-distribute before EUA. So it’s not surprising distribution has been painfully slow even though millions of Pfizer and Moderna doses were manufactured prior to EUA!
FDA should “measure time in lives lost” (see this tweet : https://twitter.com/sociologyWV/status/1356602925355855873 ). Ie “—the FDA scheduled the meeting to finalize J&J approval just 66,000 deaths from now.”
The fiasco with FDA not approving at-home COVID19 tests. For latest see https://thehill.com/.../538349-biden-raises-hopes-for-new… (Innova is shipping tests overseas because FDA won’t approve them. UK has approved the Innova at-home test, it seems: https://bmj.com/content/371/bmj.m4950)
Data shows fractional Moderna doses are just as effective (see a number of MR posts on this such as https://marginalrevolution.com/.../how-to-double-the...) But FDA won’t allow it and needs a lengthly process to be undertaken to get it approved.
#InvisibleGraveyard frame—picture of a graveyard and estimated stats about lives loss due to FDA delay.
Emphasize AZ likely to prevent death and hospitalization for the SA strain (provide quotes / limited data from trial).
Emphasize lower cost and no need for “deep freeze” ultra-low temp refrigeration for AZ.
Talk about the Novavax vaccine, which looks really great against the SA variant (see data table here: https://twitter.com/MonicaGandhi9/status/1360002608891527171/photo/1 ). We need more discussion / analysis of Novavax. The interim Phase III data looks really good and if it was approved for emergency use right now many lives could be saved. It’s a small company though so we need to investigate the manufacturing side (seems to be scant info on that), which might be the larger bottleneck.
The “one day sooner” frame.
Compelling personal stories about how people have been affected by FDA delay. For instance:
My loved one died waiting to get a vaccine.
My grandfather/ grandmother has been struggling for weeks to obtain a vaccine. AZ would be a literal life saver.
Stories about job loss… the sooner pandemic ends the sooner I can get back to work and pay my bills etc etc.
Finally, we will have about 4 high quality graphics done by a professional graphic designer and 4 lower quality graphics prepared (photos of experts with quotes). So far we have released 2, which you may have see floating around. We’re still figuring out the best way to release these graphics for maximal impact. We have a coordination discord here although it’s not really being used much : https://discord.gg/xHutNv8e
Hashtags we will focus on are #UnclogTheFDA and #ApproveAstraZeneca. I like #InvisibleGraveyard also and of course we are open to other ideas. Based on what I read, the simpler the hashtag, the better. It appears one word / singular noun type things trend the most. We might all want to mention keyword “FDA” in all our tweets also so maybe “FDA” will go trending.
Its funny because 90+% of articles on Salon.com are ‘godawful clickbait’ in my opinion—with this one being one of the exceptions.
I have mixed feelings on this. I have mentored ~5 undergraduates in the past 4 years and observed many others, and their research productivity varies enormously. How much of that is due to IQ vs other factors I really have no idea. My personal feeling was most of the variability was due to life factors like the social environment (family/friends) they were ensconced in and how much time that permitted them to focus on research.
My impression from TAing physics for life scientists for two years was that a large number felt they were intrinsically bad at math. That’s really bad! We need to be spreading more growth mindset ideas, not the idea that you’re limited by your IQ. Or at the very least, the idea that math doesn’t have to come naturally or be easy for you to be a scientist or engineer. I struggled with math my entire way through undergrad and my PhD. If the drive I developed as a child to become a scientist wasn’t so strong, I’m sure I would have dropped out.
My feeling is we are more bottlenecked on great engineers than sciences. [Also, the linear model (science → invention → engineering/innovation) is wrong!] Also, we should bring back inventors—that should be a thing again.
I think it would be awesome if some day 50% of people were engineers and inventors. People with middling IQ can still contribute a lot! Maybe not to theoretical physics, but to many other areas! We hear a lot of gushing things about scientific geniuses, especially on this site and I think we discount the importance of everyday engineers and also people like lab techs and support staff, which are increasingly important as science becomes more multidisciplinary and collaborative.
I’ve looked into these methods a lot, in 2020 (I’m not so much up to date on the latest literature). I wrote a review in my 2020 paper, “Self-explaining AI as an alternative to interpretable AI”.
There are a lot of issues with saliency mapping techniques, as you are aware (I saw you link to the “sanity checks” paper below). Funnily enough though, the super simple technique of occlusion mapping does seem to work very well, though! It’s kinda hilarious actually that there are so many complicated mathematical techniques for saliency mapping, but I have seen no good arguments as to why they are better than just occlusion mapping. I think this is a symptom of people optimizing for paper publishing and trying to impress reviewers with novelty and math rather than actually building stuff that is useful.You may find this interesting: “Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization”. What they show is that a very simple model-agnostic technique (finding the image that maximizes an output) allows people to make better predictions about how a CNN will behave than Olah’s activation maximization method, which produces images that can be hard to understand. This is exactly the sort of empirical testing I suggested in my Less Wrong post from Nov last year.
The comparison isn’t super fair because Olah’s techniques were designed for detailed mechanistic understanding, not allowing users to quickly be able to predict CNN behaviour. But it does show that simple techniques can have utility for helping users understand at a high level how an AI works.
For what it’s worth—I see value in votes being public by default. It can be very useful to see who upvoted or downvoted your comment. Of course then people will use the upvote feature just to indicate they read a post, but that’s OK (we are familiar with that system from Facebook, Twitter, etc).
I’m pretty apathetic about all the other proposals here. Reactions seem to me to be unnecessary distractions. [side note—emojiis are very ambiguous so it’s good you put words next to each one to explain what they are supposed to mean]. The way I would interpret reactions would be as a poll of people’s system 1 snap judgements. That is arguably useful/interesting information in many contexts but also distracting in other contexts.
I’m having trouble understanding the n-cut metric used in Filan’s work.
A more intuitive measure would be the sum of weights contained in edges that go between each subset of vertices divided by the total sum of weights in the graph as a whole. That’s not quite what n-cut measures though, if you look at the equation—it isn’t normalized that way.
It would be nice if there were some figures of examples of modular graphs with different n-cut values to provide an intuitive understanding of what n-cut = 9 means vs n-cut = 5.
Look at the latest paper (the earlier one seems to have some errors) and look at figure 6. If we just focus on the FASHION dataset results, the control networks had n-cut ~ 10.6 and the dropout networks had just slightly lower values, like 10.3 (if I’m understanding this correctly). The L1 regularized networks had slightly lower n-cut (around 10) and L2 regularized had n-cuts going down to 8. (note the results are sort of all over the map, though). It makes sense to me that dropout would lead to less clusterability—because you end up with multiple redundant sub-networks in different places all doing the same thing.
Anyway, my main question is if a decrease in n-cut from ~11 to 8 is significant. What about going from 8.5 to 5 like in figure 8?
It’s odd that L1 improves clusterability relative to L2 in figure 8 but not figure 6. I would intuitively think L1 would improve clusterability since it’s easier for weights to get suppressed to exactly zero
As a side note, I’m glad they changed the title—“Neural networks are surprisingly modular” is a rather unscientific name for a paper—scientific papers should stick to facts, not subjective impressions like how surprised you are. (other authors are guilty of using “surprising” in the title too, see here)
As far as Olah’s work, I think it’s the best I’ve seen for visualizing the internal workings of neural nets, but I’m also a bit worried his methods as they are right now are missing a lot, for instance the non-robust features described in this paper. Also, the change of basis issue, where random directions often give meaningful/interpretable visualizations looks like a serious issue to me (see the section “Interactions between neurons” here). But that’s a whole separate discussion..
I would modify the theory slightly by noting that the brain may become hypersensitive to sensations arising from the area that was originally damaged, even after it has healed. Sensations that are otherwise normal can then trigger pain. I went to the website about pain reprocessing therapy and stumbled upon an interview with Alan Gordon where he talked about this. I suspect that high level beliefs about tissue damage etc play a role here also in causing the brain to become hyper focused on sensations coming from a particular region and to interpret them as painful.
Something else that comes to mind here is the rubber hand illusion. Watch this video—and look at the flinches! Interesting, eh?
edit: (ok, the rubber hand illusion isn’t clearly related, but it’s interesting!)
Thanks for sharing! That’s a pretty sophisticated modeling function but it makes sense. I personally think Moore’s law (the FLOPS/$ version) will continue, but I know there’s a lot of skepticism about that.
Could you make another graph like Fig 4 but showing projected cost, using Moore’s law to estimate cost? The cost is going to be a lot, right?
Another point is that when you optimize relentlessly for one thing, you have might have trouble exploring the space adequately (get stuck at local maxima). That’s why RL agents/algorithms often take random actions when they are training (they call this “exploration” instead of “exploitation”). Maybe random actions can be thought of as a form of slack? Micro-slacks?
Look at Kenneth Stanley’s arguments about why objective functions are bad (video talk on it here). Basically he’s saying we need a lot more random exploration. Humans are similar—we have an open-ended drive to explore in addition to drives to optimize a utility function. Of course maybe you can argue the open-ended drive to explore is ultimately in the service of utility optimization, but you can argue the same about slack, too.
I don’t have much direct experience with transformers (I was part of some research with BERT once where we found it was really hard to use without adding hard-coded rules on top, but I have no experience with the modern GPT stuff). However, what you are saying makes a lot of sense to me based on my experience with CNNs and the attempts I’ve seen to explain/justify CNN behaviour with side channels (for instance this medical image classification system that also generates text as a side output).
See also my comment on Facebook.
As far as “playing the comments game”, I admit I am guilty of that. At a deeper level it comes from a desire to connect with like-minded people. I may even be doing it right now.
We like to think people post because they are genuinely intellectually engaged in the material we’ve written, but the truth is people post comments for a myriad of different reasons, including wanting to score comment ‘points’ or ‘karma’ or engage in a back-and-forth with a figure they admire. People like getting attention. [even shy nerdy people who are socially isolated or socially awkward, for which commenting on an internet blog may count as a significant social engagement] As you point out, the ‘comments game’ motivation isn’t necessarily bad in terms of the consequences—it gets debate and discussion going. Given the importance of the topics discussed on LW and elsewhere, even low quality discussion is better than no discussion, or shutting people out.
Obviously though, there is a tension in the ‘rational-sphere’, though between wanting to draw in lots of new people and wanting to maintain a sense of community, or people who are on the ‘same wavelength’. This tension is not at all unique to rationalism, and it typically leads to some type of fragmentation—people who want to ‘spread rationalism’ and grow the movement go one way and the people who want to maintain a sense of community and maintain purity go another. I’ve seen the same dynamic at work in the Libertarian party and in Christian churches. I think we have to accept both sides have good points.
But getting back to your post, it seems like you are more on the ‘we need to maintain a sense of community’ side. Personally I haven’t been very active in forums or online communities, but from what I have seen, maintaining a community online is possible , but it takes work—it requires considerable organization, active moderators and administrators, etc. Some platforms are more conducive to it than others. I can’t really comment on the viability of LW, since I’m kinda new here, but it seems to be a good place.
As a side note, I’m not sure how much ‘social trust’ is required for commenting. While I might be very hesitant to talk to someone at a cocktail party for fear of annoying them, or because I don’t trust them to take me seriously, I don’t feel that way about commenting, or if I do, it’s to a much lower extent. There is a difference—talking to someone in real life requires really interrupting them and taking their time, while writing a comment doesn’t really interrupt someone as they can always ignore it if they want to. What you said about more socially privileged people being more trusting or confident is definitely true though.
Since nobody else posted these:
Bay Area is Sat Dec 17th (Eventbrite) (Facebook)
South Florida (about an hour north of Miami) is Sat Dec 17th (Eventbrite) (Facebook)
The thing you are trying to study (“returns on cognitive reinvestment”) is probably one of the hardest things in the world to understand scientifically. It requires understanding both the capabilities of specific self-modifying agents and the complexity of the world. It depends what problem you are focusing on too—the shape of the curve may be very different for chess vs something like curing disease. Why? Because chess I can simulate on a computer, so throwing more compute at it leads to some returns. I can’t simulate human biology in a computer—we have to actually have people in labs doing complicated experiments just to understand one tiny bit of human biology.. so having more compute / cognitive power in any given agent isn’t necessarily going to speed things along.. you also need a way of manipulating things in labs (either humans or robots doing lots of experiments). Maybe in the future an AI could read massive numbers of scientific papers and synthesize them into new insights, but precisely what sort of “cognitive engine” is required to do that is also very controversial (could GPT-N do it?).
Are you familiar with the debate about Bloom et al and whether ideas are getting harder to find? (https://guzey.com/economics/bloom/ , https://www.cold-takes.com/why-it-matters-if-ideas-get-harder-to-find/). That’s relevant to predicting take-off.
The other post I always point people too is this one by Chollet.
I don’t necessarily agree with it but I found it stimulating and helpful for understanding some of the complexities here.
So basically, this is a really complex thing.. throwing some definitions and math at it isn’t going to be very useful, I’m sorry to say. Throwing math and definitions at stuff is easy. Modeling data by fitting functions is easy. Neither is very useful in terms of actually being able to predict in novel situations (ie extrapolation / generalization), which is what we need to predict AI take-off dynamics. Actually understanding things mechanistically and coming up with explanatory theories that can withstand criticism and repeated experimental tests is very hard. That’s why typically people break hard questions/problems down into easier sub-questions/problems.
I think what you’re saying makes a lot of sense. When assembling a good training data set, it’s all about diversity.
The Skeptics Guide to the Universe podcast interviewed Grant Richey about this. He notes that some of the headlines were misleading, because the study did find that when flossing is performed by a dental hygienist on children, it has positive effect. So, a better encapsulation of the recent review is that improper flossing doesn’t have any positive effect. On the other hand, its very unlikely to hurt you, unless you damage your gums in the process.
in case anyone wants a detailed review of the literature from before this study, Grant Richey did a blog post on it a few months ago: https://www.sciencebasedmedicine.org/may-the-floss-be-with-you/
hah… actually not a bad idea… too late now. BTW the recording will be available eventually if you’re interested.
That’s really cool, thanks for sharing!
(cross posting this comment from E. S. Yudkowksy’s Facebook with some edits / elaboration)
Has anyone tried fine-tuning a transformer on small datasets of increasing size to get a sense of how large a dataset would be needed to do this well? I suspect it might have to be very large.
Note this is similar to the “self explaining AI” idea I explored in early 2020, which I threw together a paper on (I am hesitant to link to it because it’s not that great of a paper and much of the discussion there is CNN specific, but here it is.). I can see how producing “thoughts” could help us trust/determine how much a model really understands what’s going on or how to make a good story.
However I also could see the “thoughts” output misleading people—people might mistake the model’s explanations as mapping onto the calculations going on inside the model to produce an output. The way GPT-3 works, I suspect, is very far from how humans think. GPT-3 is very bad at a lot of common sense and physics-based reasoning, for instance, yet based on the thoughts output the user might be misled into thinking the model understands common sense notions or physics since it’s spouting off a version of some stuff it got from it’s training data.
Any work along these lines would definitely need empirical testing / studies to show that the extra “thoughts” output is useful to end-users in some way (like predicting failure modes or helping debug failures).
Also, I’m unclear on what constitutes a “run”… roughly how long does the text have to be, in words, to have a chance at getting $20,000?