Deep Learning Weekly: Issue 261

AlphaFold reveals the structure of the protein universe, faster text generation with TensorFlow and XLA, approximating self-attention in linear time and memory, a layerwise loss construction framework

Hello folks: This week in deep learning, we bring you these updates: AlphaFold reveals the structure of the protein universe, Faster Text Generation with TensorFlow and XLA, approximating self-attention in linear time and memory via the Nyström method, and a paper introduces a general framework of layerwise loss construction for multilayer neural networks.

You may also enjoy improving journey time predictions with GNNs on the Graphcore IPU, Parallel Programming for Training and Productionization of ML/AI Systems, a collection of lectures from industry leaders, a paper on neural density-distance fields, and more.

As always, happy reading and hacking. If you have something you think should be in next week's issue, find us on Twitter: @dl_weekly.

Until next week!

---

Industry

AlphaFold reveals the structure of the protein universe

DeepMind releases predicted structures for nearly all cataloged proteins known to science, which will expand the AlphaFold DB by over 200x, with the potential to dramatically increase our understanding of biology.

Efficient, accurate object detection for hundreds of uncommon object classes

To enable the research community to reproduce and build upon these advancements, Meta AI releases the ViTDet code and training recipes as new baselines in the open source Detectron2 object detection library.

Using artificial intelligence to control digital manufacturing

MIT researchers have now developed a machine-learning system that uses computer vision to watch the manufacturing process and then correct errors in how it handles the material in real-time.

Improving journey time predictions with GNNs on the Graphcore IPU

Researchers at the National University of Singapore (NUS) are using spatio-temporal graph convolutional networks with a Mixture of Experts (MoE) approach to deliver fast, accurate, large scale journey time predictions.

Can artificial intelligence really help us talk to the animals?

A California-based organization wants to harness the power of machine learning to decode communication across the entire animal kingdom.

MLOps

Setting up MLOps at a Reasonable Scale with Jacopo Tagliabue - neptune.ai

An in-depth article on how to set up MLOps at a reasonable scale, what tools to use, and whether to buy or build them.

Faster Text Generation with TensorFlow and XLA

A blog on the lessons and findings learned from conducting load tests for an image classification model across numerous deployment configurations. These configurations involve REST-based deployments with TensorFlow Serving.

Parallel Programming for Training and Productionization of ML/AI Systems

This article will go in-depth on the fundamentals of parallel processing, how it may be used in ML/AI systems, and how it can be used for productionisation.

Simplify iterative machine learning model development by adding features to existing feature groups in Amazon SageMaker Feature Store

A post demonstrating how to add features to a feature group using the newly released UpdateFeatureGroup API on Amazon SageMaker Feature Store.

Learning

Summer school on Statistical Physics & Machine learning

A collection of lectures, from industry leaders, covering the frontiers of high-dimensional statistics, machine learning, theory of computing, and the related mathematics and probability theory.

Improve Your Data Science Workflow with Rolling Functions in Pandas

A comprehensive guide on rolling features in Pandas and how you can use them in your next tabular data analysis for time-series analysis.

Building a Speech-Enabled AI Virtual Assistant with NVIDIA Riva on Amazon EC2

In this post, we walk through how you can simplify the speech AI development process by using NVIDIA Riva to run GPU-optimized applications.

Major Problems of Machine Learning Datasets

A practical blog showcasing some major problems that occur with many supervised machine learning datasets, as well as how to deal with them.

Nyströmformer: Approximating self-attention in linear time and memory via the Nyström method

The aim of this blog post is to give readers an overview of the Nyström method and how it can be adapted to approximate self-attention.

Libraries & Code

basetenlabs/truss: Serve any model without boilerplate code

An open-source standard for packaging models built in any framework for sharing and deployment in any environment, local or production

Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.

Detectron2 is Facebook AI Research's next generation library that provides state-of-the-art detection and segmentation algorithms.

Papers & Publications

LocoProp: Enhancing BackProp via Local Loss Optimization

Abstract:

Second-order methods have shown state-of-the-art performance for optimizing deep neural networks. Nonetheless, their large memory requirement and high computational complexity, compared to first-order methods, hinder their versatility in a typical low-budget setup. This paper introduces a general framework of layerwise loss construction for multilayer neural networks that achieves a performance closer to second-order methods while utilizing first-order optimizers only. Our methodology lies upon a three-component loss, target, and regularizer combination, for which altering each component results in a new update rule. We provide examples using squared loss and layerwise Bregman divergences induced by the convex integral functions of various transfer functions. Our experiments on benchmark models and datasets validate the efficacy of our new approach, reducing the gap between first-order and second-order optimizers.

Rewriting Geometric Rules of a GAN

Abstract:

Deep generative models make visual content creation more accessible to novice users by automating the synthesis of diverse, realistic content based on a collected dataset. However, the current machine learning approaches miss a key element of the creative process – the ability to synthesize things that go far beyond the data distribution and everyday experience. To begin to address this issue, we enable a user to "warp" a given model by editing just a handful of original model outputs with desired geometric changes. Our method applies a low-rank update to a single model layer to reconstruct edited examples. Furthermore, to combat overfitting, we propose a latent space augmentation method based on style-mixing. Our method allows a user to create a model that synthesizes endless objects with defined geometric changes, enabling the creation of a new generative model without the burden of curating a large-scale dataset. We also demonstrate that edited models can be composed to achieve aggregated effects, and we present an interactive interface to enable users to create new models through composition. Empirical measurements on multiple test cases suggest the advantage of our method against recent GAN fine-tuning methods. Finally, we showcase several applications using the edited models, including latent space interpolation and image editing.

Neural Density-Distance Fields

Abstract:

The success of neural fields for 3D vision tasks is now indisputable. Following this trend, several methods aiming for visual localization (e.g., SLAM) have been proposed to estimate distance or density fields using neural fields. However, it is difficult to achieve high localization performance by only density fields-based methods such as Neural Radiance Field (NeRF) since they do not provide density gradient in most empty regions. On the other hand, distance field-based methods such as Neural Implicit Surface (NeuS) have limitations in objects' surface shapes. This paper proposes Neural Density-Distance Field (NeDDF), a novel 3D representation that reciprocally constrains the distance and density fields. We extend distance field formulation to shapes with no explicit boundary surface, such as fur or smoke, which enable explicit conversion from distance field to density field. Consistent distance and density fields realized by explicit conversion enable both robustness to initial values and high-quality registration. Furthermore, the consistency between fields allows fast convergence from sparse point clouds. Experiments show that NeDDF can achieve high localization performance while providing comparable results to NeRF on novel view synthesis.