MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine

System One decision models such as Jev answer structured questions in a single forward pass without generating text, offering fast and computationally efficient clinical decision support.

像 Jev 这样的“系统一”（System One）决策模型可以在单次前向传递中回答结构化问题，而无需生成文本，从而提供快速且计算高效的临床决策支持。

We evaluate MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks spanning medical knowledge, diagnosis, and patient level prediction using MIMIC IV and MIMIC IV ED.

我们评估了 MedJev，这是一个可在笔记本电脑上本地运行的开源“系统一”模型。评估涵盖了十项医学基准测试，涉及医学知识、诊断以及使用 MIMIC IV 和 MIMIC IV ED 进行的患者层面预测。

We compare MedJev against zero shot LLM baselines, published GPT results, and supervised models. MedJev achieves accuracies of 72.5 percent on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM, approaching published GPT 4 performance while maintaining well calibrated confidence estimates (ECE 0.028 to 0.063).

我们将 MedJev 与零样本大语言模型（LLM）基线、已发表的 GPT 结果以及监督学习模型进行了比较。MedJev 在 MedQA 上的准确率为 72.5%，在 USMLE 上为 75.4%，在 NEJM 上为 80.1%，其表现接近已发表的 GPT-4 水平，同时保持了校准良好的置信度估计（ECE 为 0.028 至 0.063）。

On patient level tasks, MedJev achieves an AUROC of 0.926 for In Hospital Mortality, exceeding XGBoost (0.902), but underperforms supervised models on operational outcomes such as ED Triage Admission (0.745 vs. 0.809). Patient level predictions also exhibit substantial miscalibration (In Hospital Mortality ECE 0.286).

在患者层面任务中，MedJev 在院内死亡率预测上的 AUROC 达到 0.926，超过了 XGBoost（0.902），但在急诊分诊入院等运营结果方面表现不及监督学习模型（0.745 对比 0.809）。患者层面的预测也表现出明显的校准偏差（院内死亡率 ECE 为 0.286）。

We introduce a lightweight, two parameter recalibration method that adapts predictions to local patient populations using 50 to 200 labeled examples, reducing ECE to 0.007. Our findings establish open System One models as a promising zero shot baseline for medical AI, combining competitive predictive performance with low computational cost and efficient local calibration.

我们引入了一种轻量级的双参数重校准方法，利用 50 到 200 个标记样本使预测适应当地患者群体，从而将 ECE 降低至 0.007。我们的研究结果表明，开源“系统一”模型是医学人工智能领域一种有前景的零样本基线，它将具有竞争力的预测性能与低计算成本及高效的本地校准结合在了一起。