ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Assessment of Adherence to Reporting Guidelines by Commonly Used Clinical Prediction Models From a Single Vendor A Systematic Review

Lu, Jonathan H.*; Callahan, Alison; Patel, Birju S.; Morse, Keith E.; Dash, Dev; Pfeffer, Michael A.; Shah, Nigam H.
JAMA Network Open2022-8-19DOI: 10.1001/jamanetworkopen.2022.27779
23被引3阅读
引用收藏
分享

摘要

IMPORTANCE Various model reporting guidelines have been proposed to ensure clinical prediction models are reliable and fair. However, no consensus exists about which model details are essential to report, and commonalities and differences among reporting guidelines have not been characterized. Furthermore, how well documentation of deployed models adheres to these guidelines has not been studied. OBJECTIVES To assess information requested by model reporting guidelines and whether the documentation for commonly used machine learning models developed by a single vendor provides the information requested. EVIDENCE REVIEW MEDLINE was queried using machine learning model card and reporting machine learning from November 4 to December 6, 2020. References were reviewed to find additional publications, and publications without specific reporting recommendations were excluded. Similar elements requested for reporting were merged into representative items. Four independent reviewers and 1 adjudicator assessed how often documentation for the most commonly used models developed by a single vendor reported the items. FINDINGS From 15 model reporting guidelines, 220 unique items were identified that represented the collective reporting requirements. Although 12 items were commonly requested (requested by 10 or more guidelines), 77 items were requested by just 1 guideline. Documentation for 12 commonly used models from a single vendor reported a median of 39% (IQR, 37%-43%; range. 31%-47%) of items from the collective reporting requirements. Many of the commonly requested items had 100% reporting rates, including items concerning outcome definition, area under the receiver operating characteristics curve, internal validation, and intended clinical use. Several items reported half the time or less related to reliability, such as external validation, uncertainty measures, and strategy for handling missing data. Other frequently unreported items related to fairness (summary statistics and subgroup analyses, including for race and ethnicity or sex). CONCLUSIONS AND RELEVANCE These findings suggest that consistent reporting recommendations for clinical predictive models are needed for model developers to share necessary information for model deployment. The many published guidelines would, collectively, require reporting more than 200 items. Model documentation from 1 vendor reported the most commonly requested items from model reporting guidelines. However, areas for improvement were identified in reporting items related to model reliability and fairness. This analysis led to feedback to the vendor, which motivated updates to the documentation for future users.

学科与关键词

关键词
ARTIFICIAL-INTELLIGENCEINDIVIDUAL PROGNOSISDIAGNOSIS TRIPODSPIRIT 2013RISKBIASSTATEMENTHEALTHAPPLICABILITYEXPLANATION
相关
加载中…
  • Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Modeling Studies: Development and Validation
    Klement, William; El Emam, Khaled*JOURNAL OF MEDICAL INTERNET RESEARCH, 2023, 25: e48763.被引 17
  • Recommendations for Reporting Machine Learning Analyses in Clinical Research
    Stevens, Laura M.; Mortazavi, Bobak J.; Deo, Rahul C.; Curtis, Lesley; Kao, David P.*Circulation. Cardiovascular quality and outcomes, 2020, 13(10): e006556.被引 146
  • Proposed Requirements for Cardiovascular Imaging-Related Machine Learning Evaluation (PRIME): A Checklist Reviewed by the American College of Cardiology Healthcare Innovation Council
    Sengupta, Partho P.*; Shrestha, Sirish; Berthon, Beatrice; Messas, Emmanuel; Donal, Erwan; Tison, Geoffrey H.; Min, James K.; D'hooge, Jan; Voigt, Jens-Uwe; Dudley, Joel; Verjans, Johan W.; Shameer, Khader; Johnson, Kipp; Lovstakken, Lasse; Tabassian, Mahdi; Piccirilli, Marco; Pernot, Mathieu; Yanamala, Naveena; Duchateau, Nicolas; Kagiyama, Nobuyuki; Bernard, Olivier; Slomka, Piotr; Deo, Rahul; Arnaout, RimaJACC-Cardiovascular Imaging, 2020, 13(9): 2017-2035.被引 144
查看全部 →

发表信息

论文状态
公开发表
期刊
JAMA Network Open
发表日期
2022-08-19
卷期
第 5 卷, 第 8 期
页码
e2227779
DOI
10.1001/jamanetworkopen.2022.27779
期刊等级
1区
成果收录
SCIE
资助标注
Stanford University School of Medicine MedScholars grant; Stanford Medicine Program for AI in Healthcare - Debra; Department of Medicine and Stanford Healthcare

作者

  • L

    Lu, Jonathan H. (通讯) (第一作者)

  • C

    Callahan, Alison

查看全部 7 位作者收起作者
  • P

    Patel, Birju S.

  • M

    Morse, Keith E.

  • D

    Dash, Dev

  • P

    Pfeffer, Michael A.

  • S

    Shah, Nigam H.

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417