ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Modeling Studies: Development and Validation

Klement, William; El Emam, Khaled*
JOURNAL OF MEDICAL INTERNET RESEARCH2023-8-31DOI: 10.2196/48763
17被引4阅读
引用收藏
分享

摘要

Background: The reporting of machine learning (ML) prognostic and diagnostic modeling studies is often inadequate, making it difficult to understand and replicate such studies. To address this issue, multiple consensus and expert reporting guidelines for ML studies have been published. However, these guidelines cover different parts of the analytics lifecycle, and individually, none of them provide a complete set of reporting requirements. Objective: We aimed to consolidate the ML reporting guidelines and checklists in the literature to provide reporting items for prognostic and diagnostic ML in in-silico and shadow mode studies. Methods: We conducted a literature search that identified 192 unique peer-reviewed English articles that provide guidance and checklists for reporting ML studies. The articles were screened by their title and abstract against a set of 9 inclusion and exclusion criteria. Articles that were filtered through had their quality evaluated by 2 raters using a 9-point checklist constructed from guideline development good practices. The average kappa was 0.71 across all quality criteria. The resulting 17 high-quality source papers were defined as having a quality score equal to or higher than the median. The reporting items in these 17 articles were consolidated and screened against a set of 6 inclusion and exclusion criteria. The resulting reporting items were sent to an external group of 11 ML experts for review and updated accordingly. The updated checklist was used to assess the reporting in 6 recent modeling papers in JMIR AI. Feedback from the external review and initial validation efforts was used to improve the reporting items. Results: In total, 37 reporting items were identified and grouped into 5 categories based on the stage of the ML project: defining the study details, defining and collecting the data, modeling methodology, model evaluation, and explainability. None of the 17 source articles covered all the reporting items. The study details and data description reporting items were the most common in the source literature, with explainability and methodology guidance (ie, data preparation and model training) having the least coverage. For instance, a median of 75% of the data description reporting items appeared in each of the 17 high-quality source guidelines, but only a median of 33% of the data explainability reporting items appeared. The highest-quality source articles tended to have more items on reporting study details. Other categories of reporting items were not related to the source article quality. We converted the reporting items into a checklist to support more complete reporting. Conclusions: Our findings supported the need for a set of consolidated reporting items, given that existing high-quality guidelines and checklists do not individually provide complete coverage. The consolidated set of reporting items is expected to improve the quality and reproducibility of ML modeling studies.

学科与关键词

关键词
machine learningprognostic modelsprediction modelsreporting guidelinesreproducibility guidelinesdiagnosticprognosticmodel evaluationmodel training
相关
加载中…
  • Assessment of Adherence to Reporting Guidelines by Commonly Used Clinical Prediction Models From a Single Vendor A Systematic Review
    Lu, Jonathan H.*; Callahan, Alison; Patel, Birju S.; Morse, Keith E.; Dash, Dev; Pfeffer, Michael A.; Shah, Nigam H.JAMA Network Open, 2022, 5(8): e2227779.被引 23
  • Recommendations for Reporting Machine Learning Analyses in Clinical Research
    Stevens, Laura M.; Mortazavi, Bobak J.; Deo, Rahul C.; Curtis, Lesley; Kao, David P.*Circulation. Cardiovascular quality and outcomes, 2020, 13(10): e006556.被引 146
  • Proposed Requirements for Cardiovascular Imaging-Related Machine Learning Evaluation (PRIME): A Checklist Reviewed by the American College of Cardiology Healthcare Innovation Council
    Sengupta, Partho P.*; Shrestha, Sirish; Berthon, Beatrice; Messas, Emmanuel; Donal, Erwan; Tison, Geoffrey H.; Min, James K.; D'hooge, Jan; Voigt, Jens-Uwe; Dudley, Joel; Verjans, Johan W.; Shameer, Khader; Johnson, Kipp; Lovstakken, Lasse; Tabassian, Mahdi; Piccirilli, Marco; Pernot, Mathieu; Yanamala, Naveena; Duchateau, Nicolas; Kagiyama, Nobuyuki; Bernard, Olivier; Slomka, Piotr; Deo, Rahul; Arnaout, RimaJACC-Cardiovascular Imaging, 2020, 13(9): 2017-2035.被引 144
查看全部 →

发表信息

论文状态
公开发表
期刊
JOURNAL OF MEDICAL INTERNET RESEARCH
发表日期
2023-08-31
卷期
第 25 卷
页码
e48763
DOI
10.2196/48763
语言
ENGLISH
成果收录
SCIE
资助标注
Precision Child and Youth Mental Health funding initiative at the Children's Hospital of Eastern Ontario Research Institute; Canadian Institutes of Health Research; Natural Sciences and Engineering Research Council of Canada [RGPIN-2022-04811]

作者

  • K

    Klement, William (第一作者)

  • E

    El Emam, Khaled (通讯)

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417