Deep Learning Weekly: Issue #264

AI for detecting Parkinson's from breathing patterns, system design for recommendations and search, a static analysis library for computing graph representations of Python programs, and more

---

Hey Folks,

This week in deep learning, we bring you AI for detecting Parkinson's from breathing patterns, system design for recommendations and search, a static analysis library for computing graph representations of Python programs, and a paper on fast infinite waveform music generation.

You may also enjoy practical federated learning for pneumonia detection with Azure, the future of automatic speech recognition, a tool for building feature stores, a paper on cold diffusion, and more!

As always, happy reading and hacking. If you have something you think should be in next week's issue, find us on Twitter: @dl_weekly.

Until next week!

---

Industry

Easily list and initialize models with new APIs in TorchVision

TorchVision now supports listing and initializing all available built-in models and weights by name. This new API builds upon the recently introduced Multi-weight support API, is currently in Beta, and it addresses a long-standing request from the community.

Predict, Detect, Mitigate: AI for Climate Science Takes the Stage at NVIDIA GTC

A dozen sessions at the global AI conference will highlight how sustainability can be bolstered by accelerated computing, industrial digital twins, and more.

AI Could Make Air Conditioners 10x Better

Hyperganic is using AI to design new heat exchangers that can be 3D-printed in metal.

Artificial intelligence model can detect Parkinson’s from breathing patterns

An MIT-developed device with the appearance of a Wi-Fi router uses a neural network to discern the presence and severity of one of the fastest-growing neurological diseases in the world.

MLOps

Automated Testing of MLOps Pipelines

Automated testing is a crucial component of the machine learning process that can have a long-term impact on the success of your project. This article discusses types of auto testing and how it saves time and money.

How to Solve the Model Serving Component of the MLOps Stack

An article that dives into and talks about what a basic, intermediate, and advanced setup for model serving looks like.

Deploying Hugging Face ViT on Vertex AI

A post that shows you how to deploy Hugging Face Vision Transformers on the Vertex AI platform with the same scalability level as Kubernetes-based deployment but with significantly less code.

System Design for Recommendations and Search

A technical blog on the system designs for industrial recommendations and search. 

Learning

Respond to Incoming Twilio Phone Calls Using NestJS and TensorFlow

A full-code tutorial that utilizes NLP, BERT, and QA (Question-Answer) Annotators to automatically assist in phone call responses.

An organized collection of 94 articles with NLP tips

A collection of articles on topics such as embeddings, language models, knowledge graphs, etc.

Practical Federated Learning with Azure Machine Learning

An article that describes the conceptual basis of Federated Learning, and walks through the key elements of a demo on a global federated learning scenario for pneumonia detection.

The Future of Speech Recognition: Where Will We Be in 2030?

A comprehensive article that analyzes the possible future states of automatic speech recognition.

4 Techniques To Tackle Overfitting In Deep Neural Networks

Overfitting is a condition that occurs when a model performs significantly better for training data than it does for new data. This blog looks at some of the techniques that are helpful for tackling overfitting in neural networks.

Applying Language Model Techniques to Compose AI Music

This post provides an account of a series of experiments performed in the field of AI music using the NVIDIA DGX-2 platform.

Libraries & Code

quintoandar/butterfree

A tool for building feature stores. Transform your raw data into beautiful features.

Recipes for H2O Driverless AI

A repository containing recipes for H20 Driverless AI which  automates feature engineering, model building, visualization, and interpretability.

google-research/python-graphs

A static analysis library for computing graph representations of Python programs suitable for use with graph neural networks.

Papers & Publications

Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise

Abstract:

Standard diffusion models involve an image transform -- adding Gaussian noise -- and an image restoration operator that inverts this degradation. We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact an entire family of generative models can be constructed by varying this choice. Even when using completely deterministic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models can be easily generalized to create generative models. The success of these fully deterministic models calls into question the community's understanding of diffusion models, which relies on noise in either gradient Langevin dynamics or variational inference, and paves the way for generalized diffusion models that invert arbitrary processes.

Towards Learning Universal Hyperparameter Optimizers with Transformers

Abstract:

Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efficiency over objective functions from a similar distribution. However, existing methods are restricted to learning from experiments sharing the same set of hyperparameters. In this paper, we introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction when trained on vast tuning data from the wild. Our extensive experiments demonstrate that the OptFormer can imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates. Compared to a Gaussian Process, the OptFormer also learns a robust prior distribution for hyperparameter response functions, and can thereby provide more accurate and better calibrated predictions. This work paves the path to future extensions for training a Transformer-based model as a general HPO optimizer.

Musika! Fast Infinite Waveform Music Generation

Abstract:

Fast and user-controllable music generation could enable novel ways of composing or performing music. However, state-of-the-art music generation systems require large amounts of data and computational resources for training, and are slow at inference. This makes them impractical for real-time interactive use. In this work, we introduce Musika, a music generation system that can be trained on hundreds of hours of music using a single consumer GPU, and that allows for much faster than real-time generation of music of arbitrary length on a consumer CPU. We achieve this by first learning a compact invertible representation of spectrogram magnitudes and phases with adversarial autoencoders, then training a Generative Adversarial Network (GAN) on this representation for a particular music domain. A latent coordinate system enables generating arbitrarily long sequences of excerpts in parallel, while a global context vector allows the music to remain stylistically coherent through time. We perform quantitative evaluations to assess the quality of the generated samples and showcase options for user control in piano and techno music generation.