Distributed Machine Learning

Goal of the course

The goal of the mini-course is to provide a comprehensive overview of state-of-the-art of distributed machine learning. All the team members come together to learn the fundamental technical issues in depth, and discuss the possible generalizations of this field. The entire course is divided into ten sections, mainly covering the basic knowledge of machine learning, the framework of distributed machine learning, the distributed machine learning algorithms and the mainstream distributed machine learning systems.

Course Schedule

Section 1:Machine learning foundation and distributed machine learning framework——2020.07.03 Slides

Presenter:Qingyang Duan
· Basic concepts and processes of machine learning
· Common machine learning models and optimization methods
· The framework of distributed machine learning

Section 2:The deterministic algorithms of single machine optimization——2020.07.06 Slides

Presenter:Yicheng Wang
· First order deterministic algorithm
· Second order deterministic algorithm
· The dual method

Section 3:Single machine stochastic optimization algorithms——2020.07.08 Slides

Presenter:Simiao Jiao
· Basic stochastic optimization algorithm
· Improvement of stochastic optimization algorithm
· Non-convex random optimization algorithm

Section 4:Data parallelism and model parallelism——2020.07.10 Slides

Presenter:Haoyu Chen
· Computational parallel mode
· Data parallel mode
· Model parallel pattern

Section 5:Communication mechanism——2020.07.13 Slides

Presenter:Xinyu You
· The content of communication
· The topology of communication
· The pace of communication
· The frequency of communication

Section 6:Data and model aggregation——2020.07.15 Slides

Presenter:Qingyang Duan
· The aggregation method based on model sum
· Aggregation approach based on model integration

Section 7:Distributed machine learning algorithms——2020.07.17 Slides

Presenter:Simiao Jiao
· Synchronization algorithm
· Asynchronous algorithm
· Comparison and fusion of synchronous and asynchronous
· Model parallel algorithm

Section 8:Distributed machine learning theory——2020.07.20 Slides

Presenter:Ying Zheng
· Convergence analysis
· Acceleration ratio analysis
· Generalized analysis

Section 9:Distributed machine learning systems——2020.07.22 Slides

Presenter:Bowen Ding / Haoran Chen
· Spark
· Multiverso
· TensorFlow