Deep Learning Weekly: Issue #226
TorchGeo, OpenFold, Earth-2, DCGANs, PyTorch Lightning v1.5, and more
As always, happy reading and hacking. If you have something you think should be in next week's issue, find us on Twitter: @dl_weekly.
Until next week!
NVIDIA announced Modulus, an AI framework that provides a customizable and easy-to-adopt physics-based toolkit to build neural network models. It has applications in various fields such as climate science or protein engineering.
Berlin-based Jina.ai, an open source startup that uses neural search to help its users find information in their unstructured data, announced that it has raised a $30 million Series A funding round.
Foundation models like GPT-3 and BERT have acquired outsized influence. This emphasizes the need for stronger regulations before the harms of individual models are perpetuated across application ecosystems.
Latest version of PyTorch Lightning, a framework to simplify coding complex networks, includes many exciting features including Fault-Tolerant Training, LightningLite, Loops Customization, Lightning Tutorials, and many more.
University College London has published its insights about future uses of AI, encapsulating the belief that the purpose of research and innovation in AI is ultimately to benefit people and societies around the world, and to make a positive impact on the planet.
Mobile & Edge
Nvidia revealed plans to build the world’s most powerful AI supercomputer dedicated to predicting climate change. Named Earth-2, the system would create a digital twin of Earth in Omniverse, Nvidia’s platform for 3D-simulation.
This discussion highlights the benefits, but also the limits, of running today’s AI systems on smaller devices like microprocessors. This will enable running inferences on the edge, where the data is generated.
There is now a superabundance of start-ups taking on the challenge of moving neural network-based machine learning from the cloud data center to embedded systems in the field. For the first time in a couple of decades, hardware starts to be at the center of attention.
A beginner-level tutorial for generating images of handwritten digits using a Deep Convolutional Generative Adversarial Network (DCGAN), the first GAN introduced to generate images.
This blog post outlines theoretical and practical insights about how neural networks are able to generalize to unseen data. It can be summarized in one sentence: “Memorization is the first step towards generalization!”
This TensorFlow-sponsored Kaggle challenge aims at locating and identifying harmful crown-of-thorns starfish (COTS). This is an amazing opportunity to have a real impact protecting coral reefs everywhere!
This post builds, step-by-step, a from-scratch implementation of a Transformer, the most performant architecture for many NLP applications. It is long but worth it if you want to understand in-depth how transformers work.
This guide walks through the Keras Model class, one of the centerpieces of the framework. Understanding this call is necessary if one wants to build and train their own custom models.
Libraries & Code
OpenFold carefully reproduces all of the features of DeepMind’s AlphaFold2. Unlike DeepMind's public code, OpenFold is also trainable and can be trained with DeepSpeed and with mixed precision.
TorchGeo is a PyTorch domain library, similar to torchvision, that provides datasets, transforms, samplers, and pre-trained models specific to geospatial data.
Salina is a library extending PyTorch modules for developing sequential decision models. It can be used both for Reinforcement Learning and in supervised / unsupervised learning settings (for instance for NLP, Computer Vision, etc...).
Papers & Publications
We introduce PyTorchVideo, an open-source deep-learning library that provides a rich set of modular, efficient, and reproducible components for a variety of video understanding tasks, including classification, detection, self-supervised learning, and low-level processing. The library covers a full stack of video understanding tools including multimodal data loading, transformations, and models that reproduce state-of-the-art performance. PyTorchVideo further supports hardware acceleration that enables real-time inference on mobile devices. The library is based on PyTorch and can be used by any training framework; for example, PyTorchLightning, PySlowFast, or Classy Vision.
Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.
What is being learned by superhuman neural network agents such as AlphaZero? This question is of both scientific and practical interest. If the representations of strong neural networks bear no resemblance to human concepts, our ability to understand faithful explanations of their decisions will be restricted, ultimately limiting what we can achieve with neural network interpretability. In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network. We also provide a behavioural analysis focusing on opening play, including qualitative analysis from chess Grandmaster Vladimir Kramnik. Finally, we carry out a preliminary investigation looking at the low-level details of AlphaZero's representations, and make the resulting behavioural and representational analyses available online.