Writing
Notes on machine learning, cheminformatics and how benchmarks quietly mislead us.
The Technique Behind Jev TypeSafe shipped a model that answers questions without generating text, and no paper with it. I rebuilt a 206K-parameter version and made four guesses about how it works. Two were wrong. The clever attention mask changed nothing, and reordering the answer options flipped 36% of answers. Comparing turn-taking in GPT-Live-1 and gpt-realtime-2.1 The same turn-taking tests on OpenAI’s full-duplex model: it kept talking through 19 of 20 backchannels, spoke during 54 of 117 mid-sentence pauses, and took about 1.4 s to stop when interrupted. Measuring turn detection in OpenAI’s Realtime API Hundreds of mid-sentence pauses, recorded where a listener hears them: server VAD ends your turn at any pause over 0.6 s, usually without a sound; semantic VAD rarely interrupts but can wait eight seconds; and a listener’s “mm-hm” stops the model every time. Your scaffold split is not one thing. Measured across four MoleculeNet datasets and five seeds: two implementation choices that no paper reports swing a reported AUC by 0.185, and which one bites you depends on a property of your data you probably haven’t checked.