The other day, during an after-symposium discussion on detecting BS AI/ML papers, one of my colleagues suggested doing a text search for “random split” as a good test.
Ponder Stibbons
Karma: 153
A lot of what you write is to the point and very valid. However, I think you are missing part of the story. Let’s start with
“Unlike drug development, where you’re trying to precisely hit some key molecular mechanism, assessing toxicity almost feels…brutish in nature”
I assume you don’t really believe this. Toxicity is often exactly about precisely hitting some key molecular mechanism. A mechanism that you may have no idea your chemistry is going to hit before hand. A mechanism moreover that you cannot use a straight forward ML to find because your chemistry is not in any training set that an ML model could access. It is very easy to underestimate the vastness of drug-like chemical space, and it is generally the case any given biological target molecule (desired or undesired) can be inhibited or otherwise interfered with a wide range of different chemical moieties (thus keeping medicinal chemists very well employed, and patent lawyers busy). There is unlikely to be toxicological data on any of them unless the target is quite old and there is publically available data on some clinical candidates.
We look to AlphaFold as the great success for ML in the biological chemistry field, and so we should, but we need to remember that AlphaFold is working on an extremely small portion of chemical space, not much more than that covered by the 20 natural amino acids. So AlphaFold’s predictions can be comfortably within distribution of what is already established by structural biology. ML models for toxicology, on the other hand, are very frequently predicting out of distribution.
In point of fact the most promising routes to avoiding toxicity reside in models that are wholly or partially physics-based. If we are targeting a particular kinase (say) we can create models (using AlphaFold if necessary) of all the most important kinases we don’t want to hit and, using physics-based modelling, predict whether we could get unwanted activity against any of these targets. We still have the problem of hitting unrelated protein targets but even here we could, in principle, screen for similarities in binding cavities over a wide range of off-targets and use physics-based modelling to assess cases where there is a close enough match.
Needless to say that requires an awful lot of compute and no-one is really doing this to scale yet. It is a very difficult problem.
Yes, I agree, I think it is pretty unlikely. But not completely impossible. As I said it should be pretty easy to find them if they are in the lysate via, HP liquid chromatography. Brain penetrant cyclic peptides should on the whole be significantly less polar than acyclic polypeptides of similar mass.
An excellent analysis and I’m almost sure your mistrust in the pharmaceutical efficacy of Cerebrolysin is well founded. However, having some experience working in the field of brain-penetrant drugs, I can comment that your restrictions on molecular weight and properties are too conservative. Small molecules of >550 dalton are capable of crossing the blood brain barrier if very well tailored. Also small cyclic peptides can hide their polar backbones within buried intramolecular hydrogen bond networks and become membrane permeable. The bicyclic peptide SFTI-1, a 14mer peptide, has been shown brain penetrant in rat in what looks to me a reasonable study. So, playing devil’s advocate, there is a hypothesis that the lysis procedure generates certain cyclic peptides of 500-1000 Dalton that could penetrate the BBB and have a biological effect.
I don’t believe this hypothesis but it does need to be discounted. Such cyclic peptides should be straight-forward to detect by HPLC/MS, I’d have thought, through their significantly less polar nature. Has anyone published work looking for these in Cerebrolysin?
There is an additional important point that needs to be made. Alphafold3 is using predominantly “positive” data. By this I mean the training data encapsulates considerable knowledge of favourable atom-atom or group-group interactions and relative propensities can be deduced. But “negative” data, in other words repulsive electrostatic or Van der Waals interactions, are only encoded by absence because these are naturally not often found in stable biochemical systems. There are no relative propensities available for these interactions. So AF3 can be expected to not perform as well when applied to real-world drug design problems where such interactions have to be taken into account and balanced against each other and against favourable interactions. Again, this issue can be mitigated by creating hybrid physics compliant models.
It is worth also noting that ligand docking is not generally considered a high accuracy technique and, these days is often used to 1st pass screen large molecular databases. The hits from docking are then further assessed using an accurate physics-based method such as Free Energy Perturbation.
I have similar concerns regarding the ligand sets used to test Alphafold3. I’ve had a cursory look at them and it seemed to me there were a lot phosphate containing molecules, a fair few sugars, and also some biochemical co-factors. I haven’t done a detailed analysis, so some caveats. But if true, there are two points here. Firstly there will be a lot of excellent crystallographic training material available on these essentially biochemical entities, so AlphaFold3 is more likely to get these particular ones right. Secondly, these are not drug-like molecules and docking programs are generally parameterized to dock drug-like molecules correctly, so are likely to have a lower success rate on these structures than on drug-like molecules.
I think a more in-depth analysis of performance of AF3 on the validation data is required, as the OP suggests. The problem here is that biochemical chemical space, which is very well represented by experimental 3D structure, is much smaller than potential drug-like chemical space, which is poorly represented by experimental 3D structure comparatively speaking. So inevitably AF3 will often be operating beyond the zone of applicability, for any new drug series. There are ways of getting round this data restriction, including creating physics compliant hybrid models (and thereby avoiding clashing atoms). I’d be very surprised if such approaches are not currently being pursued.
So after tearing my hair out trying to generate increasingly complex statistical analyses of scientific data in Excel, my world changed completely when I started using KNIME to process and transform data tables. It is perfect for a non-programmer such as myself, allowing the creation of complex yet easily broken-down workflows, that use spreadsheet input and output. Specialist domain tools are easily accessible (e.g chemical structure handling and access to the RDKit toolkit for my own speciality) and there is a thriving community generating free-to-use functionality. Best of all it is free to the single desk-top user.
Useful post. I can expand on one point and make a minor correction. Single Particle Cryo-EM is indeed a new(ish) powerful method of protein structure elucidation starting to make an impact in drug design. It is especially useful when a protein cannot easily be crystallised to allow more straightforward X-Ray structure determination. This is usually the case with transmembrane proteins for example. However it is actually best if the protein molecules are completely unaligned in any preferred direction as the simplest application of the refinement software assumes a perfectly random 3D orientation of the many thousands of protein copies imaged on the grid. In practice this is not so easy to achieve and corrections for unwanted preferred orientation need to be made.
I’m not going to say I don’t share deep disquiet about where AI is taking us, setting aside existential risk. One thing that gives me hope, however, is seeing what has happened in chess. The doom mongers might have predicted, with the advent of StockFish and AlphaZero, that human interest in chess would greatly diminish, because, after all, what is the point when the machines are so much better than the world champion ( world champion ELO ~2800, StockFish ELO ~4000) . But this hasn’t happened, chess is thriving and the games of the best human players are widely streamed and analysed and their brilliancies greatly admired. The machines have actually been of great benefit. They have, for instance demonstrated that the classical swashbuckling 19th century style of chess, replete with strategic sacrifices that lead to beautiful attacks, is a valid way to play, because they often play that way themselves. This style of play was for a long period overshadowed by a preference for the more stifling positional chess, the gaining of small advantages. The machines also provide instant feedback in analysis on what is ground truth, whether a particular move is good, bad or neutral. This too has augmented the chess players enjoyment of the game rather than reduced It.
Maybe we can hope that the same situation will apply in other fields of human endeavour.
I think this is an interesting point of view. The OP is interested in how this concept of checked democracy might work within a corporation. From a position of ignorance can I ask whether anyone familiar with German corporate governance recognises this mode of democracy within German organisations? I choose Germany because large German companies historically incorporate significant worker representation within their governance structures, and, historically, tend to perform well.
My understanding is that off-label often means that the potential patient is not within the bounds of the clique of patients included in the approved clinical trials. We don’t usually perform clinical trials on children or pregnant women, for instance. Alternatively, strong scientific evidence is found that a drug works on a related disease to the actual target. It may well make sense to use drugs off label where the clinician can be comfortable that the benefits out-way the possible harms. In other cases, of course, it would be extremely poor medicine. In any case, having statistically significant and validated evidence that a drug actual does something useful, is non-negotiable IMO.
It is true that most pharma companies concentrate on indications that supply returns to offset the cost of development. The FDA does have a mechanism for Orphan Drug approval, for rare diseases, where the registration requirements are significantly lowered. According to this site 41 orphan drug approvals were made in 2023. Whether this mechanism is good enough allow the promototion of rare disease in the larger pharmaceutical industry is a good question. I wonder how many of these drugs, or their precursors, originated in academic labs,, and were then spun out to a start-up or sold on?
Two things that happen in the pharmaceutical industry today despite the FDA.
Many drug candidates (compounds with IND status sanctioned by the FDA ) are pushed into clinical investigation prematurely by venture capital funded biotech, that more established and careful pharma companies would stay away from. These have a high rate of failure in the clinic. This is not fraud, by the way, it is usually a combination of hubris, inexperience, and a response to the necessity of rapid returns.
Marketing wins over clinical efficacy, unless the difference is large. Tagamet was the first drug for stomach ulcers released in the late ’70s.It was rapidly overtaken by Zantac, in the ’80s, through aggressive marketing, despite minimal clinical benefit. Today there is a large industry of medical writers sponsored by the pharmaceutical industry, whose job it is to present and summarise the clinical findings on a new drug in the most favourable way possible without straying into actual falsehood.
The scientists working at the sharp end of drug discovery, who fervently believe that what they do benefits mankind (this is, I believe, a gratifyingly large proportion of them) generally respect the job the FDA do. This is despite the hoops they force us to go through. Without the FDA keeping us honest, the medicines market would be swimming with highly marketed but inadequately tested products with dubious medicinal value. Investors would be less choosy about following respected well thought-out science, when placing their money. True innovation would actually be stifled because true innovation in drug discovery only shows its value once you’ve done the hard (and expensive) yards to prove medical benefit over existing treatments. Honest and well enforced regulation forces us to do the hard yards and take no short cuts.
In 2023 55 new drugs were approved by the FDA, hardly a sign that innovation is slacking. Without regulation the figure might be ten times higher but clinicians would be left swimming in a morass of claims and counter claims without good guidance (currently generally provided by the FDA) of what treatments should be applied in which situation.
Poorly regulated health orientated companies selling products that have little or no value? Seems unlikely.. Oh wait, what about Theranos?
A thought provoking post. Regarding peer reviewed science, I can offer the perspective that anonymous peer review is quite often not nice at all. But, having said that, unless a paper is extremely poor, adversarial reviews are rarely needed. A good critical constructive review can point out severe problems without raising the hackles of the author(s) unnecessarily and is more likely to get them dealt with properly than an overly adversarial review. This works so long as the process is private, the reviewer is truly anonymous, and the reviewer has the power to prevent bad work being published, even if from a respected figure in the field. Of these three criteria it is the last that I’d have most doubts about, even In well edited journals.
I’m not claiming this view to be particularly well informed, but it seems a reasonable hypothesis that the industrial revolution required the development, dispersement and application of new methods of applied mathematics. For this to happen there needed to be an easy-to-use number system with a zero and a decimal point. Use of calculus would seem to be an almost essential mathematical aid as well. Last but not least there needed to be a sizeable collaborative, communicative and practically minded scientific community who could discuss, criticise and disseminate applied mathematical ideas and apply them in physical experiments. All these three items were extant in Britain in the late 17th century, the latter being exemplified by the Royal Society. These, combined with the geologically bestowed gifts of coal and iron ore, would set Britain up to be in the best position to initiate the Industrial Revolution.
Now, can a proper historian of science critique this and show how this view is incorrect?
Anecdotal, but in the UK, in 1986, as a just graduated PhD I bought a 3 bedroom house for less than 4 times my salary. At present a similar house in a similar location, will cost roughly 10 times a starting PhD salary. House ownership for most young people in the UK is becoming a distant and ever delayed dream.
“Design is much more powerful than evolution since individually useless parts can be developed to create a much more effective whole. Evolution can’t flip the retina or reroute the recurrent laryngeal nerve even though those would be easy changes a human engineer could make.”
But directed evolution of a polymeric macromolecule (E.g. repurposing an existing enzyme to process a new substrate) is so much easier practically speaking than designing and making a bespoke macromolecule to do the same job. Synthesis and testing of many evolutionary candidates is quick and easy, so many design/make/test cycles can be run quickly. This is what is happening at the forefront of the artificial enzyme field.
So my personal viewpoint (and I could be proved wrong) is that Bing hasn’t the capability to suffer in any meaningful way, but is capable (though not necessarily sentiently capable) of manipulating us into thinking it is suffering.
I agree with the OP that the search for a broad spectrum anti-cancer drug is still a worthwhile endeavour. But I think it would be wrong to hold back on research into the more specific cancer remedies because the current most effective therapies are very often combination therapies of a type-specific anti-cancer drug co-administered or post-administered alongside a classical broad-spectrum anticancer agent such as cis-platin, or an anti-hormone agent for hormone dependant cancers. It is unlikely that new broad-spectrum treatments be so effective that this situation will change.
Having said that, there is huge research into targets ubiquitous for many cancers but formerly considered undruggable, such as P53 and Myc. We are getting better at finding ways to tackle these problem proteins. One approach,illustrated for example, which has great general promise is proximity induced degradation. A binder for the target is found, which doesn’t need to be at the main active site, if this is unattractive (for example if it is highly polar). This binder is then attached by chemical linker to a molecule that strongly binds to an E3 Ligase. This enzyme then recruits an E2 ubiquitin-conjugating enzyme which then ubiquitinylates the target protein preferentially on account of their proximity. The ubiquitinylated target protein is then recognised by the proteasome for degradation.
AOH1996 has, according to some accounts, been oversold as a cancer cure-all by the media. However, even if that is true, it could still have value as part of a combination therapy.