Ravid Shwartz-Ziv

Faculty Fellow

New York University

Biography

I’m a CDS Faculty Fellow at the NYU Center for Data Science, where I mostly work with Andrew Gordon Wilson and Yann Lecun on Bayesian deep networks, information theory, and self-supervised learning.

I completed my Ph.D. under the supervision of Prof Naftali Tishby and Prof Haim Sompolinsky at the Hebrew University of Jerusalem. In my Ph.D., I focused on the connection between deep neural networks (DNNs) and information theory. I tried to develop a deeper understating of DNNs based on information theory and to implement it over large scale problems. I recived the Google PhD Fellowship.

In parallel, I work as a researcher at the A.I. & data science research team of Intel’s Advanced Analytics group. There, I am involved in several projects. Mainly, development of deep learning, computer vision, and sensory data solutions for healthcare, manufacturing, and marketing, for both internal and external uses.

In 2019-2020 I had the opportunity to work as a research student at Google Brain, CA, USA. In this position, I explored the generalization ability of DNNs using information theory tools.

In the past, I was also involved in developing some projects for Wikipedia.

In my free, I volunteer as a developer at The Public Knowledge Workshop.

And I love basketball :)

Interests

Artificial Intelligence
Computational Neuroscience
Information Theory
Bayesian deep networks

Education

PhD in Computer Science and Neuroscience, 2021
The Hebrew University of Jerusalem
MSc in Computer Science and Neuroscience, 2016
The Hebrew University of Jerusalem
BSc in Computer Science and Bioinformatics, 2014
The Hebrew University of Jerusalem

Experience

Research Fellow

NYU, Center for Data Science

Sep 2021 – Present New York, NY, USA

Studying information theory, Bayesian deep networks, and representations of self-supervised learning.

Research Student

Google AI, Host: Dr. Alex Alemi

Jun 2019 – May 2020 Mountain View, CA, USA

Exploration of generalization by information quantities for infinitely-wide neural networks.

Graduate Researcher

Advisor: Professor Naftali Tishby - The Hebrew University of Jerusalem

Jan 2016 – Present Israel

Empirical and theoretical study of DNNs based on information-theoretical principles.

Development of a deeper understating of DNNs based on information theory.
Devise large scale implementation algorithms for the information bottleneck theory.

Graduate Researcher

Advisor: Professor Haim Sompolinsky - The Hebrew University of Jerusalem

Jan 2016 – Present Israel

Development of models for perceptual and transfer learning in DNNs, which are biologically plausible.

Senior AI and Data Science Researcher

Intel

Feb 2013 – Present Israel

Developing novel deep learning, computer vision and sensory data solutions for healthcare, manufacturing, sales, and marketing for both internal and external usage. Selected Projects

Automated testing of graphics units with video anomaly detection.
Optimization of the validation process using images transfer learning.
Healthcare solutions using physical and virtual sensors.
Gait recognition for smartphones using sensors fusion.

Research Assistant

Advisor: Professor Leo Joskowicz - The Hebrew University of Jerusalem

Feb 2012 – Feb 2014 Israel

Development of image segmentation algorithms using DNNs used for extracting medical indicators to detect abnormality in embryos.

Featured Publications

Ravid Shwartz-Ziv, Armon Amitai

January 2022 In TInformation Fusion

Tabular Data: Deep Learning is Not All You Need

We explored whether deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.

PDF

Zoe Piran(?), Ravid Shwartz-Ziv(?), Naftali Tishby

May 2020

The Dual Information Bottleneck

A new framework, which resolves some of the known drawbacks of the Information Bottleneck. We provide a theoretical analysis of the framework, finding the structure of its solutions and present a novel variational formulation for DNNs.

PDF Code

Ravid Shwartz-Ziv, Alexander A Alemi

January 2020 In The Symposium on Advances in Approximate Bayesian Inference

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Study the generalization properties of infinite ensembles of infinitely-wide neural networks. We report analytical and empirical investigations in the search for signals that correlate with generalization.

PDF Code

Ben-Ari Itamar(?), Ravid Shwartz-Ziv(?)

December 2018 NIPS 2018 Workshop on Systems for ML

Attentioned Convolutional LSTM Inpaintingv Network for Anomaly Detection in Videos

A semi supervised model for detecting anomalies in videos inspiredby the Video Pixel Network. We extend the Convolutional LSTM video encoder of the VPN with a novel convolutional based attention. This approach could be a component in applications requiring visual common sense.

PDF

Ravid Shwartz-Ziv(?), Ben-Ari Itamar(?)

December 2017 NIPS 2017 Time Series Workshop

Sequence Modeling Using a Memory Controller Extension for LSTM

We extend the standard LSTM architecture by augmenting it with an additional gate which produces a memory control vector signal. This vector is fed back to the LSTM instead of the original output prediction. By decoupling the LSTM prediction from its role as a memory controller we allow each output to specialize in its own task.

PDF

Ravid Shwartz-Ziv, Tishby, Naftali

July 2017

Opening the Black Box of Deep Neural Networks via Information

We demonstrate the effectiveness of the Information-Plane visualization of DNNs. (i) Most of the training epochs are spent on compression of the input to efficient representation. (ii) The representation compression phase begins when the SGD steps change from a fast drift into a stochastic relaxation (iii) The converged layers lie very close to the information bottleneck theoretical bound, and the maps to the hidden layers satisfy the IB self-consistent equations (iv) The training time is dramatically reduced when adding more hidden layers.

PDF Code Article

Ravid Shwartz-Ziv, Armon Amitai

January 2017 In TICML 2021 Workshop AutoML

Tabular Data: Deep Learning is Not All You Need

We explored whether deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.

PDF

Recent Publications

Quickly discover relevant content by filtering publications.

Tabular Data: Deep Learning is Not All You Need

We explored whether deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.

Ravid Shwartz-Ziv, Armon Amitai

PDF

The Dual Information Bottleneck

A new framework, which resolves some of the known drawbacks of the Information Bottleneck. We provide a theoretical analysis of the framework, finding the structure of its solutions and present a novel variational formulation for DNNs.

Zoe Piran(?), Ravid Shwartz-Ziv(?), Naftali Tishby

PDF Code

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Study the generalization properties of infinite ensembles of infinitely-wide neural networks. We report analytical and empirical investigations in the search for signals that correlate with generalization.

Ravid Shwartz-Ziv, Alexander A Alemi

PDF Code

Neural Correlates of Learning Pure Tones or Natural Sounds in the Auditory Cortex

Analysing perceptual learning of pure tones in the auditory cortex. Using a novel computational model, we show that overrepresentation of the learned tones does not improve along the training.

Ido Maor, Ravid Shwartz-Ziv, Libi Feigin, Yishai Elyada, Haim Sompolinsky, Adi Mizrahi

PDF

Neural Correlates of Learning Pure Tones or Natural Sounds in the Auditory Cortex

Attentioned Convolutional LSTM Inpaintingv Network for Anomaly Detection in Videos

A semi supervised model for detecting anomalies in videos inspiredby the Video Pixel Network. We extend the Convolutional LSTM video encoder of the VPN with a novel convolutional based attention. This approach could be a component in applications requiring visual common sense.

Ben-Ari Itamar(?), Ravid Shwartz-Ziv(?)

PDF

See all publications

Talks

Compression in deep learning - an information theory perspective

While DNNs have achieved many breakthroughs, our understanding of their internal structure, optimization process, and generalization is poor, and we often treat them as black boxes. We attempt to resolve these issues by suggesting that DNNs learn to optimize the Information Bottleneck (IB) principle - the tradeoff between information compression and prediction quality. In the first part of the talk, I presented this approach, showing an analytical and numerical study of DNNs in the information plane. This analysis reveals how the training process compresses the input to an optimal, efficient representation. I discussed recent works inspired by this analysis and show how we can apply them to real-world problems. In the second part of the talk, I will discuss information in infinitely-wide neural networks using recent results in Neural Tangent Kernels (NTK) networks. The NTK allows us to derive many tractable information-theoretic quantities. By utilizing these derivations, we can do an empirical search to find the important information-theoretic quantities that affect generalization in DNNs. I aslo presented the Dual Information Bottleneck (dualIB) framework, to find an optimal representation that resolves some of the drawbacks of the original IB. A theoretical analysis of the dualIB shows the structure of its solution and its ability to preserve the original distribution’s statistics. Within this, we focused on the variational form of the dualIB, allowing its application to DNNs.

Sep 22, 2021 12:00 AM NYU Center for Data Science

Video

Compression in deep learning - an information theory perspective

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Finding generalization signals using information for infinitely-wide neural networks.

Dec 8, 2019 12:00 AM 2nd Symposium on Advances in Approximate Bayesian Inference, Vancouver, Canada

PDF

Representation Compression in Deep Neural Network

An information theoretic viewpoint on the behavior of deep networks optimization processes and their generalization abilities by the information plane and how compression can help.

Aug 29, 2018 12:00 AM Google PhD Fellowship Summit, Mountain View, CA, USA