LessWrong developer, rationalist since the Overcoming Bias days. Connoisseur of jargon.
jimrandomh(Jim Babcock)
Karma: 16,665
Modafinil seems to require a prescription only in theory, not in practice. The overseas mail-order sellers don’t ask, and US pharmacies aren’t price-competitive with the mail-order sellers. Based on this research the risk of skipping the physician consult seems pretty negligible (provided you’re competent enough to check for interactions and monitor side effects yourself). I make sure my medical records reflect the fact that I use it, so that I wouldn’t eg lose access if I were in an ICU for some reason and out of a general policy of treating doctors like ally-consultants who are most useful if they have accurate information. But it would seem pretty wasteful to have an appointment just to generate a modafinil prescription and do nothing else.
I’m not a fan of gatekeeping, especially where low-risk no-abuse-potential drugs are concerned, but I’d encourage people to try the open and honest strategy first. Doctors are legally allowed to prescribe things off-label once they’re approved for any purpose, and their reaction to the idea will tell you a lot about whether they’re a doctor you want to talk to again if you need a consultation about something else in the future.
In the most common branch of this conversation, Alice is predictably going to tell Bob about it soon, and is speaking to Carol first in order to sort out details and gain courage. If Carol went and preemptively informed Bob, before Alice talked to Bob herself, this would be analogous to sharing an unfinished draft. This would be bad, but the badness really isn’t about secrecy.
The contents of an unfinished draft headed for publication aren’t secret, except in a narrow and time-limited sense. The problem is that the sharing undermines the impact of the later publication, causes people to associate the author with a lower quality product, and potentially misleads people about the author’s beliefs. Similarly, if Carol goes and preemptively tells Bob about Alice’s crush, then this is likely to give Bob a misleading negative impression of Alice.
It’s reasonable for Alice to ask Carol not to do that, and it’s okay for them to not have a detailed model of all of the above. But if Alice never tells Bob, and five years later Bob and Carol are looking back on the preceding years and asking if they could have gone differently? In that case, I think discarding the information seems like a pure harm.
There are a few legible categories in which secrecy serves a clear purpose, such as trade secrets. In those contexts, secrecy is fine. There are a few categories that have been societally and legally carved out as special cases where confidentiality is enforced—lawyers, priests, and therapists—because some people would only consult them if they could do so with the benefit confidentiality, and there being deterred from consulting them would have negative externalities.
Outside of these categories, secrecy is generally bad and transparency is generally good. A group of people in which everyone practices their secret-keeping and talks a lot about how to keeps secrets effectively is *suspicious*. This is particularly true if the example secrets are social and not technological. Being good at this sort of secret keeping makes it easier to shield bad actors and to get away with transgressions, and AFAICT doesn’t do much else. That makes it a signal of wanting to be able to do those things. This is true even if the secrets aren’t specifically about transgressions in particular, because all sorts of things can turn out to be clues later for reasons that weren’t easy to foresee.
A lot of people in the rationality community are trying to cosplay as therapists, and part of the cosplay is to import the confidentiality rules. This is done without an understanding of why those rules are there, and what other rules are there to mitigate the problems that confidentiality creates.
(An additional important caveat: A disclosure motivated by a desire to harm is bad, independent of whether the thing disclosed counts as a legitimate secret or not. Many things are bad to disclose if you’re socially attacking someone, but fine to disclose if you’re doing so because they were incidentally informative about something else.)
(This is a reply to the “Induction Bump” Phase Change video by Catherine Olsson and the rest of Anthropic. I’m writing it here instead of as a YouTube comment because YouTube comments aren’t a good place for discussion.)
(Epistemic status: Speculative musings I had while following along, which might be useful for inspiring future experiments, or surfacing my own misunderstandings, and possibly duplicating ideas found in prior work which I have not surveyed.)
The change-in-loss sample after the bump (at 19:10) surprised me. As you say, it seemed to get noticeably better at things that correspond to bigram rules (or bigrammish rules). I was expecting the improvement to be in something that, in some sense, requires metalearning; whereas (if I’m understanding correctly), a depth-1 network would be able to start learning these particular bigrams starting on the very first training step. If this does correspond to learning bigrams, what could explain the delay before learning these particular bigrams starts?
My speculative guess is that there’s a pattern where some learnable features have stronger gradients than others, and that training a multi-layer (but not a single layer) network proceeds in stages where first (a) multiple layer-1 nodes race to learn a function (loss falls rapidly), then (b) layer-2 nodes figure out which layer-1 nodes won the race and sever their connections to the ones that didn’t (loss falls slowly), and then (c) the freed-up layer-1 neurons move on to learning something else (loss falls rapidly again). Under this hypothesis, “the bump” corresponds to part of this sequence.
This seems to match what we see in the graph at 30:20. In that graph (which is training-step vs induction score for selected nodes in layer 1), some of the nodes go to 1, and some of the nodes start going to 1, then reverse direction shortly after the first set of nodes hits their asymptote. This is what you’d expect to see if the nodes that reverse direction were headed towards being duplicates of nodes that didn’t, but which lost the race and then got repurposed.
This seems like it might be a useful microcosm for studying the behavior of redundant nodes in general. (We know large networks have a lot of redundancy because SqueezeNets work). One major worry about neural-net transparency is that we might end up in a state where we can inspect neural nets and find things that are there, but that the methods that do that won’t be able to assert that things *aren’t* there. Concretely, this might look like finding nodes that track an unwanted concept, pinning or ablating those nodes, and then finding that the concept still presented due via duplicate nodes or via an off-basis aggregation of nodes that seem unrelated.
It seems worth noting that taking map-territory distinction as object is useful in contexts that aren’t about highminded philosophy or epistemology. Eg if I’m debugging some software, one angle to do that from is searching for the relevant difference between my mental model of how the software works, and how it actually works.
For the highly abstract epistemology stuff, my stance is: Most people should probably be more pluralistic. Thinking about map-territory correspondence is a frame in which you can figure out what’s true. Thinking about predictiveness is a frame in which you can figure out what’s true. Coherence, Bayesianism, and pragmatism are frames in which you can figure out what’s true.
A lot of philosophy seems like what it’s trying to do is to establish a pecking order between frames: to make one particular thing The Ultimate Source of Truth, and rebuild and justify the rest in terms of that. I think this is a mistake. What we actually do, when we’re not trying to be philosophical, and what actually works, is we use all the frames we’ve picked up, we use them to judge each other’s predictions, we kick out the ones that underperform too badly, and eventually wind up at a fixpoint where all our models of truth are making mostly the same predictions. This works better because lots of things are easy to think about in one ontology and hard to think about in another, and if we try to mash everything into the same ontology for philosophical reasons, the mashing process tends to force us into tortured analogies that yield unreliable conclusions.
I don’t think it was a software issue. I forget the document I was reading, but I vaguely remember hearing it reported that a hardware malfunction that caused the cameras to fail. If it was a “bug” that turned the camera off at all I would have considered that suspicious, because it would probably be the easiest way of disabling a prison camera.
What I would find convincing is if there was a specific diagnosed hardware issue, which couldn’t be triggered by software at all, like one of the chips had a bad solder joint or there was a burnt insect lying across two contacts. I haven’t heard anything like that, though.
I completely disagree in general with your sentiment about computer forensics. It is very difficult to cover up evidence of a breach 100% perfectly.
I think you’re really overstating the difficulty. Especially in the context of a denial-of-service attack on a camera system, which really is easy mode: there’s no need for persistence, the storage is being continuously rewritten so there’s not much free space to worry about, and an approximately-identical copy of the whole security system can probably be bought from the same camera vendor to use as a testbed.
Yes, it really is that bad. It was already that bad even before the pandemic and before things got politicized.
I recall once at a (2018) party, a friend told an anecdote about having a weird and serious side effect from a common, old psychiatric medication, which wasn’t a listed side effect. A psychiatrist at the party said they had a patient who had the same side effect from the same drug, and had looked into it at the time and thought that wasn’t supposed to happen, and asked if he could share the anecdote. I asked if he had ever tried reporting events like that, and how that worked. He said that there was theoretically a system for doing so, but he had never used it, nor had his colleagues, and it wasn’t a thing in practice.
Against that backdrop, you have a vaccine adverse event reporting system, and a society polarized into a group that wants to glorify them and a group that wants to demonize them. A the same time, side effect rates are (I think, based on super sparse data) very different between different age demographics, and between people who have had COVID recently and who haven’t, and between the different vaccines. On balance I think it’s still quite clear that getting vaccinated is better than getting COVID, and getting vaccinated and getting reduced-severity breakthrough COVID is better than getting full-severity COVID, but most institutions seem to be choosing “downplay side effects as much as possible” rather than “honestly talk about the balance of considerations”.
(That said, not all of the things you point at are actually that bad. Estimating the fraction of cases which are of a variant is easy, much easier than estimating the number of cases in the population as a whole, because all you have to do is set up a sequencer in a lab that was already doing PCR tests.Estimating the exposure-to-infectious time period is not as trivial, but the serial interval is a good substitute metric, and that can be easily estimated by asking positive cases when they think they were exposed, and taking the subset who seem confident in the exposure date.)
This is interesting in that it’s biologically plausible, in ways that backpropagating neural networks aren’t. But this doesn’t have state of the art performance, and I’m going to register the prediction that, when scaled and tuned, it will still fail to match the best existing scaled-and-tuned neural networks. I could be wrong about this, and the framing of how this algorithm works is different enough from the usual backpropagation that I’d be interested in the results of the experiment, but I’m not all that excited (or scared).
You report 98.9% accuract on PI-MNIST. For comparison, this 2015 paper on ladder networks claims (table 1) to have gotten 0.57% test error (99.43% accuracy) when using the full training set. I haven’t looked very hard for other papers (the permutation invariance is a weird search term), but I expect a thorough search of the literature to turn up significantly higher. I also have a preexisting belief that MNIST-in-particular has a large number of easy samples and a small number of hard samples, such that getting to ~98% is easy and then it starts getting progressively harder.
Some possibilities: Clapping hands, sparkles, smiling face with open hands, 100
I’m a bit worried about having too many yellow-circle-face icons, because they’re hard to tell apart when they’re small.
The idea behind this voting system is to act as a culture-shaping tool: the ballot you see when you hover over the vote buttons is meant to tell you what we think makes for good and bad comments. Ideally, this message comes across even if you aren’t voting much.
I’ve given some thought to the specific axes and reactions, but they should still be treated very much as a first draft. I’m very interested in comparing other people’s lists of axes and reactions, and in arguments about what should and shouldn’t be included. What makes for a good comment? What should people be paying attention to? What have you wanted to communicate to authors, which you wish you had a button for instead of having to write a whole comment?
A big uncertainty I have about this voting system is how much of a problem the extra complexity is. Is seeing the extra score components on comments distracting? Does having a bunch of extra axes to vote on make voting too time consuming or overwhelming? Feedback on this is appreciated.
And of course, now that we have a setup in place where we can try out alternative voting systems, we’re interested in any original ideas people have for different ways of voting on comments that we haven’t thought of.
Open Thread—Jan 2022 [Vote Experiment!]
The cameras in front of his cell were torn apart by the FBI for analysis and apparently, there was no evidence of tampering.
This isn’t the evidence you think it is. Pretty much every publicly discovered piece of high-end malware has a component designed to clean up and remove all evidence of its passing. It isn’t particularly difficult to do that 100% perfectly. In fact I’d even go so far as to say that if evidence was found, the simplest explanation would be that the someone was trying to leave fake evidence to implicate someone else. This is squarely within the expected behavior of LARPing intelligence officers, and I believe there are public reports of intelligence agencies localizing their malware’s string tables into each others’ languages for just this reason.
It’s often said that in languages, the syllable-count of words eventually converges to something based on the frequency with which words are used, so that more-commonly-used concepts get words with fewer syllables.
There’s an important caveat to this, which I have never seen stated anywhere: the effect is strongly weighted towards vocabulary used by children, especially small children. Hence why “ma”, the lowest-entropy word, means mother in so many different languages, and why toddler-concepts are all monosyllables or twice-repeated monosyllables. So, for example. “probability” being 5 syllables isn’t just a reflection of it being used, it’s also a reflection of an (imo bad) societal judgement that people won’t be ready for that word until they’re adults.
I think we should be putting pretty substantial probability mass on the possibility that Omicron was the result of a successful, secret project to create a less-severe but more-contagious strain of COVID-19 in a lab, release it, and have it crowd out the original strain.
The cruxes of this belief are:
The genome of Omicron is not consistent with natural evolution, in any environment
Omicron produces substantially less severe disease than any earlier strains of COVID-19
Producing substantially less severe disease isn’t something that happens by default, if you’re manipulating a virus in a lab
If you’re already manipulating COVID-19 in a lab with all the difficulties that entails, making a less-severe variant does not add significant difficulty on top of that
If you do have a less-severe lab-grown variant of COVID-19, and you think it probably confers cross-immunity to older variants, you will probably do a moral calculation that finds it’s good to release it on purpose.
This could be done unilaterally by a small group or even a single individual, in any of a very large number of biology labs all over the world
I’m not fully confident in any of these cruxes, but consider each of them highly probable.
If this were publicly confirmed to be true, I would expect the public reaction to be strongly negative, even if the project was good on net. So I would expect anyone trying to do this would choose secrecy, would support their decision to remain secret, and would tentatively support their decision to undertake the project in the first place, depending on details related to how much less severe the variant is, how strong the cross-immunity is, and how these were verified.
(I have not seen anyone else suggest this possibility, and I think this possibility falls inside an almost-universal blind spot people have; people incorrectly assume that nothing done in secret and with a conspiratorial flavor can ever be well-intentioned and good.)
This feels like a nice crisp example of how Twitter is broken in ways that generate disinformation. A user with 531 followers, not someone who can reasonably be expected to treat their tweets as a journalistic product or employ a factchecker, made an understandable mistake. This produced a politically-potent but inaccurate soundbite.
A substantial fraction of the quote-tweets are refutations. These are only visible if you check for them explicitly. This creates a confirmation bias trap: if you think the economic left is pushing a false narrative, you’ll be more likely to check, see that the tweet is wrong, and reinforce that belief. And if you think that inequality is a huge problem and the economic situation of the poor is dire, then you’ll be less likely to check, again reinforcing that belief.
This is specifically a property of systems that don’t have downvotes and of systems that don’t have a good way of sorting replies, which is why this sort of thing is especially common and especially bad on Twitter..
I definitely see major falsehoods in this post, particularly with regards to cancellations (which happened online in places I read). Other parts of the post describe things I didn’t witness, but which seem dubious when placed in the context of a hit piece with known falsehoods in it.
I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints.
There’s a huge gulf between “AGI ideal that sounds like it will work to the researcher who came up with it” and “AGI idea that actually works when tried.” Like, to the point where AGI researchers having ideas that they’re insanely overconfident in, that don’t work when tried, is almost being a trope. How much of an epistemic problem this is depends on the response to outside view, I think. If you remove the bluster, “working AGI design” turns into “research direction that sounds promising”. I do think that, if it’s going to be judged by the “promising research direction” standard rather than the “will actually work when tried” standard, then coming up with an AGI design is a pretty reasonable standard?
There’s a model-fragment that I think is pretty important to understanding what’s happened around Michael Vassar, and Scott Alexander’s criticism.
Helping someone who is having a mental break is hard. It’s difficult for someone to do for a friend. It’s difficult for professionals to do in an institutional setting, and I have tons of anecdotes from friends and acquaintances, both inside and outside the rationality community, of professionals in institutions fucking up in ways that were traumatizing or even abusive. Friends have some natural advantages over institutions: they can provide support in a familiar environment instead of a prison-like environment, and make use of context they have with the person.
When you encounter someone who’s having a mental break or is giving off signs that they’re highly stressed and at risk of a mental break, the incentivized action is to get out of the radius of blame (see Copenhagen Interpretation of Ethics). I think most people do this instinctively. Attempting to help someone through a break is a risky and thankless job; many more people will hear about it if it goes badly than if it goes well. Anyone who does it repeatedly will probably find name attached to a disaster and a mistake they made that sounds easier to avoid than it really was. Nevertheless, I think people should try to help their friends (and sometimes their acquaintances) in those circumstances, and that when we hear how it went, we should adjust our interpretation accordingly.
I’ve seen Michael get involved in a fair number of analogous situations that didn’t become disasters and that no one heard about, and that significantly affects my interpretation, when I hear that he’s been in the blast-radius of situations that did.
I think Scott Alexander looked at some stories (possibly with some rumor-mill distortions added on), and took a “this should be left to professionals” stance. And I think the “this should be left to professionals” stance looks better to him, as a professional who’s worked only in above-average institutions and who can fix problems when he sees them, than it does to people collecting anecdotes from others who’ve been involuntarily committed.
I think this is based on an incorrect model of what the Pomodoro technique is for, and where its benefits come from. I think most people don’t actually have a model of that, they just empirically observe that things go better when they use it than when they don’t. So here’s my attempt to model what the Pomodoro technique does that’s good.
Pomodoro breaks act as a stop-loss function on getting stuck or sidetracked, on things that are going to take much longer than expected, etc. The “break” is an opportunity to do metacognition.
Pomodoros create a steady supply of minor deadlines to work towards.
“Work for one pomodoro on X” tends to be less intimidating than “do X”, which makes some tasks easier to get started on.
When working in a group, synchronized timed breaks allow for a social aspect with less risk of allowing the socializing to overrun the actual work.
I don’t think third-time really substitutes for any of this. None of these fit into a depletion-and-recovery model; for me, that’s not what the breaks are actually for.