Deep Learning Weekly: Issue #216
AI Hardware Summit, Maze for Applied Reinforcement Learning, Neurips Data-Centric AI Workshop, PECOS (Predictions for Enormous and Correlated Output Spaces), OpenAI Codex, and more
Hey folks,
This week in deep learning, we bring you AI to accelerate the energy transition, an AI-based sensing carpet, a tutorial about on-device ML for audio, and a paper about text data augmentation.
You may also enjoy a tutorial about ML compilers, Google’s latest image upscaling tech, a new framework for applied reinforcement learning, and more!
As always, happy reading and hacking. If you have something you think should be in next week's issue, find us on Twitter: @dl_weekly.
Until next week!
Industry
This is how AI will accelerate the energy transition
A new report by the World Economic Forum, “Harnessing AI to Accelerate the Energy Transition”, defines the actions needed to unlock AI’s potential to enable the energy transition away from fossil fuels and towards renewable alternatives.
Google’s New AI Photo Upscaling Tech is Jaw-Dropping
Google AI recently shared new breakthroughs they’ve made to turn low-res photos into detailed high-res photos using ML. These breakthroughs are based on the use of a new concept called “diffusion models”.
China Is Still the World's Factory—And It's Designing the Future With AI
In this post, Kai-Fu Lee describes China’s vision regarding the use of AI in manufacturing, and states that China is quickly becoming a leader in how to do it intelligently.
Intelligent carpet gives insight into human poses
A tactile sensing carpet from MIT’s CSAIL lab can estimate human poses without using cameras. The group envisions this as a step towards improving self-powered personalized healthcare, smart homes, and gaming.
Now That Machines Can Learn, Can They Unlearn?
Researchers are testing whether they can remove sensitive data from AI systems without retraining them. This would be very helpful to comply with the best privacy standards.
An AI for Fine Art: Attorney Trains NVIDIA RTX 2070 to Authenticate Masterpieces
This post details how an intellectual property lawyer and his spouse have used deep learning to authenticate that Salvator Mundi, the most expensive painting ever, was indeed likely made by Leonardo Da Vinci.
Mobile & Edge
A friendly introduction to machine learning compilers and optimizers
A very clear introduction to how ML compilers work. Having a deep understanding of this topic is very helpful for deploying ML models on different hardware without compromising performance.
The AI Hardware Summit took place last week in Mountain View, and a number of prestigious speakers explained how to accelerate AI workloads in the cloud and at the edge.
Easy Machine Learning for On-Device Audio
This blog post gives a nice introduction to ML for on-device audio. It shows how to develop and customize an audio classification model for a mobile app, including a couple of real world examples for inspiration.
Learning
Toward Fast and Accurate Neural Networks for Image Recognition
Google Research introduces two new families of neural networks for state-of-the-art image recognition—one for fast training on small-scale datasets, and the other for high accuracy on large-scale datasets.
Maze: Applied Reinforcement Learning for Real-World Problems
Maze is a framework for applied reinforcement learning that focuses on solutions for practical problems arising when dealing with use cases outside of academic and toy settings.
Neurips Data-Centric AI Workshop
The biggest workshop on data-centric AI, which represents the recent transition from focusing on modeling to focusing on the underlying data used to train and evaluate models.
A Dataset Exploration Case Study with Know Your Data
Google AI introduces Know Your Data, a dataset exploration tool which helps better understanding datasets, thus helping to mitigate fairness and bias issues.
WarpDrive: Extremely Fast Reinforcement Learning on an NVIDIA GPU
WarpDrive is an open-source library for end-to-end multi-agent reinforcement learning on a GPU, achieving orders of magnitude faster training than existing frameworks.
Libraries & Code
A list of various products, demos, and articles based on OpenAI’s Codex, an AI system that translates natural language to code.
PECOS - Predictions for Enormous and Correlated Output Spaces
PECOS is a ML framework for fast learning and inference on problems with large output spaces, such as extreme multi-label ranking and large-scale retrieval.
Papers & Publications
Datasets: A Community Library for Natural Language Processing
Abstract:
The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks.
Text Data Augmentation for Deep Learning
Abstract:
Natural Language Processing (NLP) is one of the most captivating applications of Deep Learning. In this survey, we consider how the Data Augmentation training strategy can aid in its development. We begin with the major motifs of Data Augmentation summarized into strengthening local decision boundaries, brute force training, causality and counterfactual examples, and the distinction between meaning and form. We follow these motifs with a concrete list of augmentation frameworks that have been developed for text data. Deep Learning generally struggles with the measurement of generalization and characterization of overfitting. We highlight studies that cover how augmentations can construct test sets for generalization. NLP is at an early stage in applying Data Augmentation compared to Computer Vision. We highlight the key differences and promising ideas that have yet to be tested in NLP. For the sake of practical implementation, we describe tools that facilitate Data Augmentation such as the use of consistency regularization, controllers, and offline and online augmentation pipelines, to preview a few. Finally, we discuss interesting topics around Data Augmentation in NLP such as task-specific augmentations, the use of prior knowledge in self-supervised learning versus Data Augmentation, intersections with transfer and multi-task learning, and ideas for AI-GAs (AI-Generating Algorithms). We hope this paper inspires further research interest in Text Data Augmentation.
The Impact of Reinitialization on Generalization in Convolutional Neural Networks
Abstract:
Recent results suggest that reinitializing a subset of the parameters of a neural network during training can improve generalization, particularly for small training sets. We study the impact of different reinitialization methods in several convolutional architectures across 12 benchmark image classification datasets, analyzing their potential gains and highlighting limitations. We also introduce a new layerwise reinitialization algorithm that outperforms previous methods and suggest explanations of the observed improved generalization. First, we show that layerwise reinitialization increases the margin on the training examples without increasing the norm of the weights, hence leading to an improvement in margin-based generalization bounds for neural networks. Second, we demonstrate that it settles in flatter local minima of the loss surface. Third, it encourages learning general rules and discourages memorization by placing emphasis on the lower layers of the neural network. Our takeaway message is that the accuracy of convolutional neural networks can be improved for small datasets using bottom-up layerwise reinitialization, where the number of reinitialized layers may vary depending on the available compute budget.
 |
|