Hey Folks,

This week in deep learning, we bring you Microsoft extends partnership with OpenAI, building a GPT with Andrej Karpathy, Google's playbook for systematically tuning models, and a paper on Multiview Compressive Coding for 3D Reconstruction.

You may also enjoy Sweden's National Library Turns Page to AI, mechanisms for effective machine learning projects, do large language models learn world models or just surface statistics?, a paper on Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling, and more.

As always, happy reading and hacking. If you have something you think should be in next week's issue, find us on Twitter: @dl_weekly.

Until next week!

Industry

Microsoft announces new multibillion-dollar investment in ChatGPT-maker OpenAI

Microsoft just announced a new multiyear, multibillion-dollar investment with ChatGPT-maker OpenAI.

Optical AI Could Feed Voracious Data Needs

A neural network that employs photons instead of electrons could rapidly analyze vast amounts of data by running many computations simultaneously using thousands of wavelengths of light, a new study finds.

Booked for Brilliance: Sweden’s National Library Turns Page to AI to Parse Centuries of Data

The library is training state-of-the-art AI models on a half-millennium of Swedish text to support humanities research in history, linguistics, media studies and more.

Doordash Introduces ML to Understand the Marketplaces Status

DoorDash introduces an ML model to predict the operational status of a store in order to increase the user experience and save thousands of orders cancellation.

AI21 Labs has created a co-writing bot that can suggest quotes, statistics, provide citations and more

AI21 Labs, an AI research lab focused on NLP and generative AI, is hoping to spice up the lives of writers with the launch of Wordtune Spices, a new feature within its popular Wordtune editing platform.

New York City embarks on creating an AI strategy

The Office of Technology and Innovation is hiring a director of artificial intelligence and machine learning.

MLOps

Scalable Serving with Kubernetes and Seldon Core: A tutorial

A technical tutorial on how to deploy ML models in Kubernetes clusters with Seldon Core, and implement auto-scaling for the deployment with HPA and KEDA.

Introduction to Artifacts In Comet

An article that discusses how to keep track of one of the precious assets of an ML experiment: a dataset (which Comet classifies as Artifacts).

Mechanisms for Effective Machine Learning Projects

Eugene Yan shares his mechanisms for running effective machine learning projects.

How to Version Control Data in ML for Various Data Sources

An article that discusses the relevance of data version control in machine learning, and explores various methods and tools for implementing it with different types of data sources.

Learning

Building a GPT with Andrej Karpathy

The companion notebook for Andrej Karpathy’s GPT-from-scratch tutorial.

A Better Way to Train Your Neural Networks

Five tips from experts in deep learning and computer vision.

Point Net for Semantic Segmentation

A technical tutorial on how to train Point Net for semantic segmentation on the Stanford 3D Indoor Scene Dataset (S3DIS).

Temporal Graph Learning in 2023

A blog post that presents major progress in Temporal Graph Learning until 2022, and discusses promising future directions.

Do Large Language Models learn world models or just surface statistics?

A blog post that discusses the underlying mechanisms of large language models when performing more complex tasks (ex. writing basic code and solving puzzles).

Libraries & Code

google-research/tuning_playbook

A playbook for systematically maximizing the performance of deep learning models.

happy-machine/FastQL

Spin up a super fast Rust powered GraphQL API to prototype your ML model in one line of Python code.

gdikov/hypertunity

Hypertunity is a lightweight, high-level library for hyperparameter optimization.

Papers & Publications

Do Users Write More Insecure Code with AI Assistants?

Abstract:

We conduct the first large-scale user study examining how users interact with an AI Code assistant to solve a variety of security related tasks across different programming languages. Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access. Additionally, participants with access to an AI assistant were more likely to believe they wrote secure code than those without access to the AI assistant. Furthermore, we find that participants who trusted the AI less and engaged more with the language and format of their prompts (e.g. re-phrasing, adjusting temperature) provided code with fewer security vulnerabilities. Finally, in order to better inform the design of future AI-based Code assistants, we provide an in-depth analysis of participants' language and interaction behavior, as well as release our user interface as an instrument to conduct similar studies in the future.

Multiview Compressive Coding for 3D Reconstruction

Abstract:

A central goal of visual recognition is to understand objects and scenes from a single image. 2D recognition has witnessed tremendous progress thanks to large-scale learning and general-purpose representations. Comparatively, 3D poses new challenges stemming from occlusions not depicted in the image. Prior works try to overcome these by inferring from multiple views or rely on scarce CAD models and category-specific priors which hinder scaling to novel settings. In this work, we explore single-view 3D reconstruction by learning generalizable representations inspired by advances in self-supervised learning. We introduce a simple framework that operates on 3D points of single objects or whole scenes coupled with category-agnostic large-scale training from diverse RGB-D videos. Our model, Multiview Compressive Coding (MCC), learns to compress the input appearance and geometry to predict the 3D structure by querying a 3D-aware decoder. MCC's generality and efficiency allow it to learn from large-scale and diverse data sources with strong generalization to novel objects imagined by DALL⋅E 2 or captured in-the-wild with an iPhone.

Ankh: Optimized Protein Language Model Unlocks General-Purpose Modeling

Abstract:

As opposed to scaling-up protein language models (PLMs), we seek improving performance via protein-specific optimization. Although the proportionality between the language model size and the richness of its learned representations is validated, we prioritize accessibility and pursue a path of data-efficient, cost-reduced, and knowledge-guided optimization. Through over twenty experiments ranging from masking, architecture, and pre-training data, we derive insights from protein-specific experimentation into building a model that interprets the language of life, optimally. We present Ankh, the first general-purpose PLM trained on Google's TPU-v4 surpassing the state-of-the-art performance with fewer parameters (<10% for pre-training, <7% for inference, and <30% for the embedding dimension). We provide a representative range of structure and function benchmarks where Ankh excels. We further provide a protein variant generation analysis on High-N and One-N input data scales where Ankh succeeds in learning protein evolutionary conservation-mutation trends and introducing functional diversity while retaining key structural-functional characteristics. We dedicate our work to promoting accessibility to research innovation via attainable resources.

Deep Learning Weekly

Deep Learning Weekly: Issue #285