Training deep neural networks with less energy - 万博官网APP,狗万官方客户端下载,万博客户端登录

代表性图像显示了深度学习中的神经网络。

早鸟是一种节能方法，用于训练深层神经网络（DNN），支持自动驾驶车辆的人工智能，面部识别和类似的应用类型。它是developed by researchersfrom Texas A&M University and Rice University.

首席作者Haoran You和Rice的Chaojian Li对赖斯有效且智能计算（EIC）实验室的一项研究表明，早鸟可能使用的能量减少10.7倍，将DNN训练至相同的准确性或比典型训练更好。EIC实验室主任Yingyan Lin与Rice的Richard Baraniuk和Texas A＆M的Zhangyang Wang一起领导了这项研究。

林说：“最近AI突破的主要推动力是引入更大，更昂贵的DNN。”“但是培训这些DNN需要大量能量。为了揭幕更多的创新，必须找到“更绿”的培训方法，这些培训方法既解决环境问题，又减少了人工智能研究的财务障碍。”

The reason for Early Bird
Training cutting-edge DNNs is costly. A 2019 study by the Allen Institute for AI in Seattle found the number of computations needed to train a top-flight deep neural network increased 300,000 times between 2012-2018. Another 2019 study by researchers at the University of Massachusetts Amherst found the carbon footprint for training a single, elite DNN was roughly equivalent to the lifetime carbon dioxide emissions of five U.S. automobiles.

DNNs are constituted by billions of artificial neurons that learn to perform specific tasks. They are capable of taking decisions very much like human beings. Without any explicit programming, deep networks of artificial neurons can learn to make human-like decisions — and even outperform human experts — by “studying” a large number of previous examples. One example isAlphago，在研究了成千上万先前玩过的游戏后，一个深厚的网络训练了玩棋盘游戏，在2015年击败了一名专业人类球员。

林说：“进行DNN培训的最先进方式称为渐进rune and Train。”“首先，您训练一个密集的巨型网络，然后删除看起来不重要的零件，例如修剪树。然后，您重新训练修剪的网络以恢复性能，因为修剪后的性能会降低。实际上，您需要修剪和再培训多次以获得良好的表现。”

Rice University’s Early Bird method for training deep neural networks finds key connectivity patterns early in training, reducing the computations and carbon footprint for the increasingly popular form of artificial intelligence known as deep learning. (Graphic courtesy of Y. Lin/Rice University)

How it works
训练加强了最有用的神经元之间的键合，并过滤了那些可以修剪的神经元之间的键合。因此，可以将网络“修剪”为专业任务，仅由一小部分人工神经元执行。修剪在降低模型规模和计算成本方面起着关键作用，从而增加了DNN培训的负担能力。

“Our idea in this work is to identify the final, fully functional pruned network, which we call the ‘early-bird ticket’ in the beginning stage of this costly first step.” Lin and her team searched for early-bird tickets by studying key network connectivity patterns in the initial stage of training. The team found that Early Bird could emerge as one-tenth or less of the way through the initial phase of training.

该项目是使AI技术更环保和包容性的关键步骤。如果一切顺利，它将用单个笔记本电脑或有限的计算资源为AI创新打开大门。

“我们的方法可以在密集，巨型网络的培训的前10％或更少的时间内自动识别早期鸟票。这意味着您可以训练DNN，以在传统培训所需的10％或更少的时间内实现相同甚至更高的准确性，这可以导致计算和能源的一种以上的订单节省。”林。