Deep Learning Weekly Issue #139

OpenAI adopts PyTorch, Colab goes Pro, TFJS comes to React Native, PyTorch3D, and more...

Hey folks,

This week in deep learning we bring you Google Colab Pro, AMD GPU acceleration for ONNX, OpenAI switching to PyTorch, autonomous drones using sonar to map underground lakes, and new research from Apple’s self-driving team.

You may also enjoy loads of new data augmentors in the latest release of imgaug, a deep dive into upsampling in Core ML, PyTorch3D, an opinionated guide to ML research, quantifying factors important for reproducibility, machine UN-learning, TensorFlow.js for React Native, and more

As always, happy reading and hacking. If you have something you think should be in next week's issue, find us on Twitter: @dl_weekly.

Until next week!


Google announces Colab Pro

For $9.99 per month, users get access to faster GPUs, a longer 24 hours runtime limit, and more RAM. Free service is still offered.

This AI-Guided Drone Has Mapped One of Earth's Deepest Subterranean Lakes

Researchers use an autonomous robot to explore and map the Dragon’s Breath cave in Namibia.

AMD adds ONNX runtime support.

AMD has contributed a new backend to ONNX bringing GPU acceleration to those using AMD graphics cards for deep learning.

OpenAI → PyTorch

OpenAI announces they will be transitioning to PyTorch.

Apple researchers train AI drivers to merge lanes in a simulated environment

Apple’s self-driving car team has a new paper out on lane merging in simulated environments.

Mobile + Edge

Upsampling in Core ML

An incredibly thorough look at how different implementations of a simple resize operator can make porting models to mobile formats a frustrating process.

TensorFlow.js adds React Native support.

TFJS for React Native has officially been released. GPU acceleration provided via a WebGL backend.

Arm unveils 2 new AI edge computing chips

New chips for IoT devices provide a large boost in model performance while being more energy efficient.


Introducing PyTorch3D: An open-source library for 3D deep learning

Facebook open-sources PyTorch tools for working with 3D data (meshes, vertices, etc.).

ImageNet Validation results.

Results for 164 PyTorch trained ImageNet models. Interesting for comparing various architectures.

Headliner — Easy training and deployment of seq2seq models

A neat writeup of using BERT to generate candidate headlines for a newspaper.

An Opinionated Guide to ML Research

A worthwhile read on how ML researchers should choose topics to work on.

Quantifying Independently Reproducible Machine Learning

A discussion of factors important for reproducibility based on replicating 255 ML papers.

Libraries & Code

[Github] aleju/imgaug

Version 0.4.0 of imgaug is out with a TON of new augmentors.

[Github] google-research/fixmatch

Google open sources code for FixMatch: A simple method to perform semi-supervised learning with limited data.

[Github] facebookresearch/meshrcnn

The implementation of Mesh R-CNN is based on Detectron2 and PyTorch3D. Builds 3D meshes from 2D images.

HiPlot: High-dimensional interactive plots made easy

A new plotting library from Facebook makes it easier to create interactive plots of things like the impact of hyperparameters on model performance.


Virtual KITTI 2 Dataset

Virtual KITTI 2 is a more photo-realistic and better-featured version of the original virtual KITTI dataset.

Papers & Publications

Machine Unlearning

Abstract: ….We introduce SISA training, a framework that decreases the number of model parameters affected by an unlearning request and caches intermediate outputs of the training algorithm to limit the number of model updates that need to be computed to have these parameters unlearn. This framework reduces the computational overhead associated with unlearning, even in the worst-case setting where unlearning requests are made uniformly across the training set. In some cases, we may have a prior on the distribution of unlearning requests that will be issued by users....We also validate how knowledge of the unlearning distribution provides further improvements in retraining time by simulating a scenario where we model unlearning requests that come from users of a commercial product that is available in countries with varying sensitivity to privacy....

Capsule Routing via Variational Bayes

Abstract: …. In this paper, we propose a new capsule routing algorithm derived from Variational Bayes for fitting a mixture of transforming gaussians, and show it is possible transform our capsule network into a Capsule-VAE. Our Bayesian approach addresses some of the inherent weaknesses of MLE based models such as the variance-collapse by modelling uncertainty over capsule pose parameters. We outperform the state-of-the-art on smallNORB using 50% fewer capsules than previously reported, achieve competitive performances on CIFAR-10, Fashion-MNIST, SVHN, and demonstrate significant improvement in MNIST to affNIST generalisation over previous works.