ScholarMate
English简体中文繁體中文
客服热线:400-1616-289
登录注册
期刊论文

Maximal relevance and optimal learning machines

Duranthon, O.; Marsili, M.*; Xie, R.i
JOURNAL OF STATISTICAL MECHANICS-THEORY AND EXPERIMENT2021-3DOI: 10.1088/1742-5468/abe6ff
6被引12阅读
引用收藏
分享

摘要

We explore the hypothesis that learning machines extract representations of maximal relevance, where the relevance is defined as the entropy of the energy distribution of the internal representation. We show that the mutual information between the internal representation of a learning machine and the features that it extracts from the data is bounded from below by the relevance. This motivates our study of models with maximal relevance-that we call optimal learning machines-as candidates of maximally informative representations. We analyse how the maximisation of the relevance is constrained both by the architecture of the model used and by the available data, in practical cases. We find that sub-extensive features that do not affect the thermodynamics of the model, may affect significantly learning performance, and that criticality enhances learning performance, but the existence of a critical point is not a necessary condition. On specific learning tasks, we find that (i) the maximal values of the likelihood are achieved by models with maximal relevance, (ii) internal representations approach the maximal relevance that can be achieved in a finite dataset and (iii) learning is associated with a broadening of the spectrum of energy levels of the internal representation, in agreement with the maximum relevance hypothesis.

学科与关键词

关键词
learning theorymachine learningstatistical inference
相关
加载中…
  • A simple probabilistic neural network for machine understanding
    Xie, Rongrong; Marsili, Matteo*JOURNAL OF STATISTICAL MECHANICS-THEORY AND EXPERIMENT, 2024, 2024(2): 023403.
  • A random energy approach to deep learning
    Xie, Rongrong; Marsili, Matteo*JOURNAL OF STATISTICAL MECHANICS-THEORY AND EXPERIMENT, 2022, 2022(7): 073404.被引 1
  • LearningWithout Neurons in Physical Systems
    Stern, Menachem*; Murugan, ArvindAnnual Review of Condensed Matter Physics, 2023, 14: 417-441.被引 26
查看全部 →

发表信息

论文状态
公开发表
期刊
JOURNAL OF STATISTICAL MECHANICS-THEORY AND EXPERIMENT
发表日期
2021-03
卷期
第 2021 卷, 第 3 期
页码
033409
DOI
10.1088/1742-5468/abe6ff
语言
ENGLISH
期刊收录
NSL
期刊等级
3区
成果收录
SCIE
资助标注
Chinal Scholarship Council (CSC) [202006770018]

作者

  • D

    Duranthon, O. (第一作者)

  • M

    Marsili, M. (通讯)

    i

查看全部 3 位作者收起作者
  • X

    Xie, R.

    cent china normal univ

登录解锁更多功能

  • 收藏这篇,随时在“我的 · 收藏”找回
  • 认领你的成果,生成推广素材
  • 关注作者获取最新成果
  • 加入项目组、协同写作
注册 / 登录

产品服务

  • 科研之友
  • 创新城
  • 科创云

服务支持

  • 帮助中心
  • 隐私政策
  • 服务条款

联系方式

在线客服:【】
客服热线:400-1616-289
电子邮箱:support@scholarmate.com

关注或下载科研之友

微信二维码
微信公众号
客户端下载二维码
下载客户端
科研成果科研人员科研机构科研动态爱瑞思软件

©2026 深圳市科研之友网络服务有限公司

公安备案图标粤公网安备 44030502000213
粤ICP备 16046710 号粤B2-20110417