Your scaffold split is not one thing.

We argue about splits by name. Someone says “we used a scaffold split” and the room relaxes, because a scaffold split is the hard one. Someone says “random split” and the room gets suspicious.

But a split is not its name. A split is a distance: the chemical gap it puts between training and test. And that distance gets decided as much by undocumented implementation details as by the rule you cite in your methods section.

I measured this on four MoleculeNet datasets with five seeds, and trained a model on every split so I could say what the choices actually cost. Three results:

1. Two free choices inside “scaffold split” swing a reported score by up to 0.185 AUC. Neither one appears in anybody’s methods section.
2. Which choice matters is dataset-dependent. It tracks a statistic you can compute in ten seconds. On the dataset where 22.7% of molecules have no scaffold, the acyclic policy is everything and group ordering does nothing. On another dataset it’s the other way round.
3. Split distance predicts the reported score (pooled Spearman +0.67, p ≈ 9×10⁻⁶). This is not a geometric curiosity. It’s your leaderboard.

There’s a fourth result that came out of auditing my own first draft, which had a confounded comparison sitting in it. That story is at the end, because it’s the most useful part.

This alarm is not new. The quantification is what’s missing.

Worth placing this honestly before going further:

• Wallach & Heifets (2018) introduced AVE bias and showed that redundancy between training and validation sets explains much of the reported performance of ligand-based methods. (JCIM, arXiv:1706.06619)
• Steshin’s Lo-Hi benchmark (NeurIPS 2023) makes essentially the measurement below, and reports that under a recommended scaffold split, 78% of test molecules still have a training neighbour above 0.4 Tanimoto. (arXiv:2310.06399)
• Yang et al. (2019), the chemprop paper, already randomizes scaffold-set assignment in its scaffold_balanced splitter. So the fact that assignment order is a free choice is documented, not discovered. (JCIM, arXiv:1904.01561)

So “scaffold splits leak” is old news. What I couldn’t find anywhere was a number for what the undocumented choices cost you, held constant one at a time, over multiple seeds, with a model attached. That’s what follows.

What a scaffold actually is

Bemis and Murcko (1996) split a molecule into four disjoint parts: ring systems, linkers (the paths joining rings), side chains, and the framework, which is rings plus linkers with the side chains deleted. That framework is what everyone now means by “the scaffold.” A scaffold split groups molecules by it, so no framework shows up on both sides of the split.

Strip the side chains, then optionally strip the atom types as well:

Two consequences fall straight out of that definition, and between them they explain most of what follows.

Stripping side chains merges molecules that are not alike

And a molecule with no ring has no scaffold at all

The measurement

For each test molecule, find its nearest neighbour in the training set by Morgan fingerprint Tanimoto.

from rdkit import DataStructs
from rdkit.Chem import rdFingerprintGenerator

gen = rdFingerprintGenerator.GetMorganGenerator(radius=2, fpSize=2048)
fps = [gen.GetFingerprint(m) for m in mols]

train_fps = [fps[i] for i in train_idx]
nn = [max(DataStructs.BulkTanimotoSimilarity(fps[i], train_fps)) for i in test_idx]

print(f"mean NN Tanimoto: {sum(nn) / len(nn):.3f}")

The mean is a summary; the distribution is the story. Three splits of the same 4,200 ChEMBL compounds (MoleculeNet Lipophilicity):

What the splits deliver

Four datasets, 70/15/15, deduplicated to unique canonical SMILES, Morgan r=2/2048. Shuffled arms are mean ± sd over 5 seeds; deterministic arms have no seed. Lower = harder test set.

Look at the scaffold rows. They span 0.398 to 0.505 on Tox21 alone, and every one of them would go into a paper as “a scaffold split.” One arm also carries a standard deviation of 0.077, which is bigger than most of the differences people publish between methods.

Choice #1: what you do with molecules that have no scaffold

MurckoScaffoldSmiles returns "" for any molecule with no ring. There’s no principled scaffold for ethanol, and implementations disagree about what to do:

• DeepChem’s ScaffoldSplitter keys on the returned string, so every acyclic molecule lands in one shared group that moves as a unit.
• The common alternative falls back to the molecule’s own SMILES, giving each acyclic its own group, which is a random split for that slice.
• Or you cluster them by fingerprint, which is what I’d argue for.

Holding assignment order fixed (shuffled, 5 seeds) and changing only this:

BBBP, Tox21 and HIV are AUC. Lipophilicity is Spearman ρ.

On Tox21 this one undocumented choice is worth 0.185 AUC, larger than the gap between most published methods on that benchmark. On Lipophilicity it’s worth nothing at all, because Lipophilicity is 0.1% acyclic.

Choice #2: how you break ties between equal-sized scaffold groups

DeepChem sorts scaffold groups largest-first. But roughly 75% of scaffold groups contain exactly one molecule, and that holds on every dataset here, from 2k to 41k compounds. So “sort by size” leaves most of the ordering undetermined. Something has to break the ties, and that something is an implementation detail.

DeepChem breaks them by first-index descending:

scaffold_sets = [s for (scaffold, s) in sorted(
    scaffolds.items(), key=lambda x: (len(x[1]), x[1][0]), reverse=True)]

Change only the tie-break, holding the grouping rule and the size ordering fixed:

On BBBP one tie-break gives you a working benchmark and the other gives you a test set with no negatives in it, so AUC can’t be computed. Same grouping rule, same sort key, same fractions.

Does any of this change the number you’d report?

This is the question my first draft never answered. A RandomForest (200 trees, Morgan counts) on every split:

Across all 36 (dataset, split) combinations, z-scored within dataset, mean NN Tanimoto correlates with the reported score at Spearman +0.67, p ≈ 9×10⁻⁶. Per dataset it runs from +0.88 on HIV and +0.85 on Lipophilicity down to +0.39 on Tox21, which isn’t significant on its own. Call it a strong pooled relationship rather than a law.

One anomaly worth flagging rather than burying. On Tox21 the scaffold split scores higher than the random split (0.826 vs 0.810), which is the opposite of the standard story. It sits inside the error bars, but it’s there, and anyone who says “scaffold splits always lower your score” should go and look at it.

The trap I fell into

My first draft recommended Butina clustering on the strength of one observation: it drove near-duplicates down to 0.2% to 1.8%, where scaffold splits left 2% to 9%.

That recommendation was circular. I’d clustered molecules by Morgan/Tanimoto and then scored the resulting split by Morgan/Tanimoto nearest-neighbour distance. Butina wins that comparison because it directly optimises the thing being measured.

The fix is to score with a fingerprint that had no part in building the split. Re-measuring with MACCS keys, 166 substructure keys on a completely different basis:

Butina’s advantage largely evaporates. On BBBP a plain DeepChem-ordered scaffold split produces a harder test set (0.718) than Butina does (0.755). On Tox21 they tie. Only on Lipophilicity does Butina still clearly win.

I published the confound first

The first version of this post claimed that group ordering was the big undocumented lever, and quoted a 0.078 swing on Tox21 as proof.

That comparison was confounded. My “random order” arm gave every acyclic molecule its own group, while my “DeepChem order” arm pooled them. The two arms differed in grouping rule as well as ordering. And I’d headlined the effect on Tox21, the one dataset out of four where 22.7% acyclics made that confound as large as it could possibly get. On the other three the ordering effect came out between 0.015 and 0.029, nowhere near 0.078.

I’d also labelled an arm “DeepChem order” without ever running DeepChem. When I finally transcribed ScaffoldSplitter out of the installed source and ran it properly, the real algorithm behaved differently from my reimplementation. That’s how I found the tie-break result, which is now the more interesting half of this post.

So: a post telling you to measure your split instead of trusting its label, with a comparison in it whose label didn’t match what it measured. I’d rather say that out loud than quietly fix it and move on.

Cheaper to learn from my draft than from your paper: the mislabelled arm looked completely fine until someone ran the code.

Reproducing this

MoleculeNet CSVs from DeepChem’s S3 bucket:

https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/BBBP.csv
https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/Lipophilicity.csv
https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/tox21.csv.gz
https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/HIV.csv

• split_audit.py is the full audit: grouping and ordering as orthogonal axes, 5 seeds, MACCS cross-check, RF training.
• verify_deepchem.py transcribes DeepChem’s ScaffoldSplitter verbatim and run under its own defaults, to check my reimplementation against the real thing.
• make_histogram.py, make_spread.py and make_scaffold_figs.py build the figures. The molecule drawings are RDKit depictions of real dataset entries, not illustrations.
• audit_results.json holds every number in this post.

Needs rdkit, pandas, numpy, scikit-learn, scipy, and RDKit 2022.09+ for rdFingerprintGenerator. Butina is skipped above 15,000 molecules, since the distance matrix is O(n²), so the Butina and hybrid rows are missing for HIV.

References

• Bemis & Murcko, The Properties of Known Drugs. 1. Molecular Frameworks, J. Med. Chem. 39(15), 1996.
• Butina, Unsupervised Data Base Clustering Based on Daylight’s Fingerprint and Tanimoto Similarity, J. Chem. Inf. Comput. Sci. 39(4), 1999.
• Wu et al., MoleculeNet: A Benchmark for Molecular Machine Learning, Chem. Sci. 9, 2018.
• Wallach & Heifets, Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization, JCIM 58(5), 2018.
• Yang et al., Analyzing Learned Molecular Representations for Property Prediction, JCIM 59(8), 2019.
• Steshin, Lo-Hi: Practical ML Drug Discovery Benchmark, NeurIPS 2023 Datasets & Benchmarks.