Critical assessment of missense variant effect predictors on disease-relevant variant data - Ruchir Rastogi; Ryan Chung; Sindy Li; Chang Li; Kyoungyeul Lee; Junwoo Woo; Dong-Wook Kim; Changwon Keum; Giulia Babbi; Pier Luigi Martelli; Castrense Savojardo; Rita Casadio; et al. (Human Genetics, 2025)
Regular, systematic, and independent assessments of computational tools that are used to predict the pathogenicity of missense variants are necessary to evaluate their clinical and research utility and guide future improvements. The Critical Assessment of Genome Interpretation (CAGI) conducts the ongoing Annotate-All-Missense (Missense Marathon) challenge, in which missense variant effect predictors (also called variant impact predictors) are evaluated on missense variants added to disease-relevant databases following the prediction submission deadline. Here we assess predictors submitted to the CAGI 6 Annotate-All-Missense challenge, predictors commonly used in clinical genetics, and recently developed deep learning methods. We examine performance across a range of settings relevant for clinical and research applications, focusing on different subsets of the evaluation data as well as high-specificity and high-sensitivity regimes. Our evaluations reveal notable advances in current methods relative to older, well-cited tools in the field. While meta-predictors tend to outperform their constituent individual predictors, several newer individual predictors perform comparably to commonly used meta-predictors. Predictor performance varies between high-specificity and high-sensitivity regimes, highlighting that different methods may be optimal for different use cases. We also characterize two potential sources of bias. Predictors that incorporate allele frequency as a predictive feature tend to have reduced performance when distinguishing pathogenic variants from very rare benign variants, and predictors trained on pathogenicity labels from curated variant databases often inherit gene-level label imbalances. Our findings help illuminate the clinical and research utility of modern missense variant effect predictors and identify potential areas for future development.

Evaluation of enzyme activity predictions for variants of unknown significance in Arylsulfatase A - Shantanu Jain; Marena Trinidad; Thanh Binh Nguyen; Kaiya Jones; Santiago Diaz Neto; Fang Ge; Ailin Glagovsky; Cameron Jones; Giankaleb Moran; Boqi Wang; Kobra Rahimi; Sümeyra Zeynep Çalıcı; et al. (Human Genetics, 2025)

Assessing the predicted impact of single amino acid substitutions in MAPK proteins for CAGI6 challenges - Paola Turina; Maria Petrosino; Carlos A. Enriquez Sandoval; Leonore Novak; Alessandra Pasquo; Emil Alexov; Muttaqi Ahmad Alladin; David B. Ascher; Giulia Babbi; Constantina Bakolitsa; Rita Casadio; Jianlin Cheng; et al. (Human Genetics, 2025)

Evaluating predictors of kinase activity of STK11 variants identified in primary human non-small cell lung cancers - Yile Chen; Kyoungyeul Lee; Junwoo Woo; Dong-wook Kim; Changwon Keum; Giulia Babbi; Rita Casadio; Pier Luigi Martelli; Castrense Savojardo; Matteo Manfredi; Yang Shen; Yuanfei Sun; et al. (Human Genetics, 2025)
Critical evaluation of computational tools for predicting variant effects is important considering their increased use in disease diagnosis and driving molecular discoveries. In the sixth edition of the Critical Assessment of Genome Interpretation (CAGI) challenge, a dataset of 28 STK11 rare variants (27 missense, 1 single amino acid deletion), identified in primary non-small cell lung cancer biopsies, was experimentally assayed to characterize computational methods from four participating teams and five publicly available tools. Predictors demonstrated a high level of performance on key evaluation metrics, measuring correlation with the assay outputs and separating loss-of-function (LoF) variants from wildtype-like (WT-like) variants. The best participant model, 3Cnet, performed competitively with well-known tools. Unique to this challenge was that the functional data was generated with both biological and technical replicates, thus allowing the assessors to realistically establish maximum predictive performance based on experimental variability. Three out of the five publicly available tools and 3Cnet approached the performance of the assay replicates in separating LoF variants from WT-like variants. Surprisingly, REVEL, an often-used model, achieved a comparable correlation with the real-valued assay output as that seen for the experimental replicates. Performing variant interpretation by combining the new functional evidence with computational and population data evidence led to 16 new variants receiving a clinically actionable classification of likely pathogenic (LP) or likely benign (LB). Overall, the STK11 challenge highlights the utility of variant effect predictors in biomedical sciences and provides encouraging results for driving research in the field of computational genome interpretation.

Assessing the predicted impact of single amino acid substitutions in calmodulin for CAGI6 challenges - Paola Turina; Giuditta Dal Cortivo; Carlos A. Enriquez Sandoval; Emil Alexov; David B. Ascher; Giulia Babbi; Constantina Bakolitsa; Rita Casadio; Piero Fariselli; Lukas Folkman; Akash Kamandula; Panagiotis Katsonis; et al. (Human Genetics, 2025)

Assessing predictions on fitness effects of missense variants in HMBS in CAGI6 - Jing Zhang; Lisa Kinch; Panagiotis Katsonis; Olivier Lichtarge; Milind Jagota; Yun S. Song; Yuanfei Sun; Yang Shen; Nurdan Kuru; Onur Dereli; Ogun Adebali; Muttaqi Ahmad Alladin; et al. (Human Genetics, 2025)

Predicting the impact of rare variants on RNA splicing in CAGI6 - Jenny Lord; Carolina Jaramillo Oquendo; Htoo A. Wai; Andrew G. L. Douglas; David J. Bunyan; Yaqiong Wang; Zhiqiang Hu; Zishuo Zeng; Daniel Danis; Panagiotis Katsonis; Amanda Williams; Olivier Lichtarge; et al. (Human Genetics, 2025)
Variants which disrupt splicing are a frequent cause of rare disease that have been under-ascertained clinically. Accurate and efficient methods to predict a variant’s impact on splicing are needed to interpret the growing number of variants of unknown significance (VUS) identified by exome and genome sequencing. Here, we present the results of the CAGI6 Splicing VUS challenge, which invited predictions of the splicing impact of 56 variants ascertained clinically and functionally validated to determine splicing impact. The performance of 12 prediction methods, along with SpliceAI and CADD, was compared on the 56 functionally validated variants. The maximum accuracy achieved was 82% from two different approaches, one weighting SpliceAI scores by minor allele frequency, and one applying the recently published Splicing Prediction Pipeline (SPiP). SPiP performed optimally in terms of sensitivity, while an ensemble method combining multiple prediction tools and information from databases exceeded all others for specificity. Several challenge methods equalled or exceeded the performance of SpliceAI, with ultimate choice of prediction method likely to depend on experimental or clinical aims. One quarter of the variants were incorrectly predicted by at least 50% of the methods, highlighting the need for further improvements to splicing prediction methods for successful clinical application.

Computational interpretation of human genetic variation - Yana Bromberg; Predrag Radivojac (Human Genetics, 2022)

Prioritizing de novo autism risk variants with calibrated gene- and variant-scoring models - Yuxiang Jiang; Jorge Urresti; Kymberleigh A. Pagel; Akula Bala Pramod; Lilia M. Iakoucheva; Predrag Radivojac (Human Genetics, 2022)