Small, Private, and First on All Fifteen: The New Medical LLM Benchmark Results

John Snow Labs’ latest Medical LLM results are in. Across fifteen clinical and biomedical benchmarks, the Medical LLM – Medium ranks first on every one, against the newest frontier releases from OpenAI, Anthropic, and Google. It averages 80.9 to their 76.5, 75.0, and 74.7 respectively. The model that produces those scores runs on a single GPU, entirely inside your own environment, with no external API call. It is licensed per server per year, not per token.

A leaderboard sweep is not worth much on its own, but this combination of accuracy, cost, and compliance is what deserves attention. It contradicts an assumption that governs most healthcare AI budgets right now: that the accurate option and the affordable, deployable option are different options, and you have to pick one.

The suite comes from MedHELM, not licensing exams

A benchmark suite assembled only from USMLE-style questions and PubMed recall measures one capability and calls it medicine. Most of these 15 tasks therefore come from or overlap with MedHELM, the Stanford-led open benchmark published in Nature Medicine, which organizes clinical AI into a clinician-validated taxonomy of 121 distinct tasks. Each selected task asks a model to do something a health system needs done.

Medec gives the model a clinical note and asks whether a sentence contains an error, such as a wrong diagnosis, a wrong drug, or a wrong causal organism, then asks it to point to the sentence and rewrite it. This is the inverse of a quiz: the note is written to look correct, and the model has to disagree with it. The best models scored around 70% on the original benchmark, against roughly 80% for physicians.

MedCalc-Bench asks for a clinical calculation from a patient vignette. Computing a Cockcroft-Gault creatinine clearance means pulling the right values from the right encounter and applying the right formula. It is the hardest task in the set, and the absolute scores of 48.0 against 34.0 for Claude Opus 4.8 show unsolved work rather than a lead.

EHRSQL turns a clinician’s plain-English question into SQL over a hospital database, along the lines of “how many patients were prescribed warfarin in the last month.” Built from questions posed by 200+ real hospital staff, it includes unanswerable questions on purpose, to test whether a model abstains instead of inventing a query. Gemini 3.5 Flash scores 14.0 against 34.0, the widest proportional gap in the table.

ACI-Bench takes the raw transcript of a doctor-patient conversation and asks for a structured visit note. The input has interruptions, repetition, and facts stated out of order. The output is documentation a clinician signs.

MTSamples Procedures asks the model to find and extract the surgical and medical procedures buried in a transcribed report. Procedures are rarely announced in a labeled field, and one report will mention procedures performed, considered, declined, or done elsewhere years earlier. Scoring well means separating what was done from what was discussed.

MedDialog scores comprehension of patient-provider conversation, measuring whether a model follows what a patient said rather than a nearby topic. At 76.3 against 76.2, this row is a four-way tie in everything but the decimal.

MedicationQA uses real consumer medication questions submitted to a health authority, rather than questions written to be answerable. The 9.5-point margin over the best frontier model is the second-widest in the suite.

The remaining tasks cover knowledge and reasoning (HeadQA, MedBullets, PubMedQA, MEDIQA, MedQA, MMLU Clinical Knowledge), hallucination control (Med-Hallu), and fairness across demographics (RaceBias).

Gaps are near zero on saturated exams and widest on operational tasks

Two patterns emerge from this table, the first coming from the medical exam rows. On MMLU Clinical Knowledge the spread from top to bottom is 2 points. On MedQA the margin is 1.2 points, and Gemini ties GPT outright. These benchmarks are saturated: everyone scores in the 90s, the differences approach noise, and a strong score is partly a reading comprehension test of the model’s own training data, as our previous discussion on data contamination detailed. No healthcare-specific model opens a real lead on a saturated public benchmark, and none should claim to.

The second pattern comes from the operational tasks, where the margins are an order of magnitude larger: 15 points on Medec, 9.5 on MedicationQA, 6 on PubMedQA, 5 on EHRSQL and RaceBias, 4 on MedCalc and Med-Hallu. The closer a task gets to clinical operations, meaning read a messy source document, extract the right facts, do the arithmetic, catch the error, and know when the answer isn’t supported, the wider the margin gets.

The saturated exam benchmarks stay in the suite so both patterns remain visible. Publishing only the wide margins would tell you less.

Private deployment costs an order of magnitude less at production volume

Accuracy decides whether a pipeline is worth running. Cost decides whether it survives a budget cycle. A node that is paid for processes its millionth token at the same marginal cost as its first: zero.

I modeled this last month on a real workload: a de-identified oncology real-world evidence dataset at 1 million tokens per patient, four passes over every document. The model uses list API prices, grants every volume discount, and skips the document filtering optimization that would favor local deployment. What frontier APIs cost, relative to a right-sized model running locally:

• 10,000 patients (a pilot or one service line): 1.35x to 3.17x. At this scale the API route is competitive, and for a short proof of concept the premium can be rational.

• 100,000 patients (a small health system or a focused research cohort): 3.3x to 7.7x.

• 1,000,000 patients (a midsize health system, a payer, or a multi-site research net