Ravid Shwartz-Ziv

Ravid Shwartz-Ziv

Research scientist & PhD Student

New York University

Biography

I’m a CDS Faculty Fellow at the NYU Center for Data Science, where I mostly work with Andrew Gordon Wilson and Yann Lecun on Bayesian deep networks, information theory, and self-supervised learning.

I completed my Ph.D. under the supervision of Prof Naftali Tishby and Prof Haim Sompolinsky at the Hebrew University of Jerusalem. In my Ph.D., I focused on the connection between deep neural networks (DNNs) and information theory. I tried to develop a deeper understating of DNNs based on information theory and to implement it over large scale problems. I recived the Google PhD Fellowship.

In parallel, I work as a researcher at the A.I. & data science research team of Intel’s Advanced Analytics group. There, I am involved in several projects. Mainly, development of deep learning, computer vision, and sensory data solutions for healthcare, manufacturing, and marketing, for both internal and external uses.

In 2019-2020 I had the opportunity to work as a research student at Google Brain, CA, USA. In this position, I explored the generalization ability of DNNs using information theory tools.

In the past, I was also involved in developing some projects for Wikipedia.

In my free, I volunteer as a developer at The Public Knowledge Workshop.

And I love basketball :)

Interests

  • Artificial Intelligence
  • Computational Neuroscience
  • Information Theory
  • Bayesian deep networks

Education

  • PhD in Computer Science and Neuroscience, 2021

    The Hebrew University of Jerusalem

  • MSc in Computer Science and Neuroscience, 2016

    The Hebrew University of Jerusalem

  • BSc in Computer Science and Bioinformatics, 2014

    The Hebrew University of Jerusalem

Experience

 
 
 
 
 

Reserach Fellow

NYU, Center for Data Science

Sep 2021 – Present New York, NY, USA
Stuying information thoery, batsian deep networks and self supervised represenastion.
 
 
 
 
 

Research Student

Google AI, Host: Dr. Alex Alemi

Jun 2019 – May 2020 Mountain View, CA, USA
Exploration of generalization by information quantities for infinitely-wide neural networks.
 
 
 
 
 

Graduate Researcher

Advisor: Professor Naftali Tishby - The Hebrew University of Jerusalem

Jan 2016 – Present Israel

Empirical and theoretical study of DNNs based on information-theoretical principles.

  • Development of a deeper understating of DNNs based on information theory.
  • Devise large scale implementation algorithms for the information bottleneck theory.
 
 
 
 
 

Graduate Researcher

Advisor: Professor Haim Sompolinsky - The Hebrew University of Jerusalem

Jan 2016 – Present Israel
Development of models for perceptual and transfer learning in DNNs, which are biologically plausible.
 
 
 
 
 

Senior AI and Data Science Researcher

Intel

Feb 2013 – Present Israel

Developing novel deep learning, computer vision and sensory data solutions for healthcare, manufacturing, sales, and marketing for both internal and external usage. Selected Projects

  • Automated testing of graphics units with video anomaly detection.
  • Optimization of the validation process using images transfer learning.
  • Healthcare solutions using physical and virtual sensors.
  • Gait recognition for smartphones using sensors fusion.
 
 
 
 
 

Research Assistant

Advisor: Professor Leo Joskowicz - The Hebrew University of Jerusalem

Feb 2012 – Feb 2014 Israel
Development of image segmentation algorithms using DNNs used for extracting medical indicators to detect abnormality in embryos.

Recent Publications

Quickly discover relevant content by filtering publications.

Tabular Data: Deep Learning is Not All You Need

We explored whether deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.
Tabular Data: Deep Learning is Not All You Need

The Dual Information Bottleneck

A new framework, which resolves some of the known drawbacks of the Information Bottleneck. We provide a theoretical analysis of the framework, finding the structure of its solutions and present a novel variational formulation for DNNs.
The Dual Information Bottleneck

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Study the generalization properties of infinite ensembles of infinitely-wide neural networks. We report analytical and empirical investigations in the search for signals that correlate with generalization.
Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Neural Correlates of Learning Pure Tones or Natural Sounds in the Auditory Cortex

Analysing perceptual learning of pure tones in the auditory cortex. Using a novel computational model, we show that overrepresentation of the learned tones does not improve along the training.
Neural Correlates of Learning Pure Tones or Natural Sounds in the Auditory Cortex

Attentioned Convolutional LSTM Inpaintingv Network for Anomaly Detection in Videos

A semi supervised model for detecting anomalies in videos inspiredby the Video Pixel Network. We extend the Convolutional LSTM video encoder of the VPN with a novel convolutional based attention. This approach could be a component in applications requiring visual common sense.
Attentioned Convolutional LSTM Inpaintingv Network for Anomaly Detection in Videos

Talks

Compression in deep learning - an information theory perspective

While DNNs have achieved many breakthroughs, our understanding of their internal structure, optimization process, and generalization is poor, and we often treat them as black boxes. We attempt to resolve these issues by suggesting that DNNs learn to optimize the Information Bottleneck (IB) principle - the tradeoff between information compression and prediction quality. In the first part of the talk, I presented this approach, showing an analytical and numerical study of DNNs in the information plane. This analysis reveals how the training process compresses the input to an optimal, efficient representation. I discussed recent works inspired by this analysis and show how we can apply them to real-world problems. In the second part of the talk, I will discuss information in infinitely-wide neural networks using recent results in Neural Tangent Kernels (NTK) networks. The NTK allows us to derive many tractable information-theoretic quantities. By utilizing these derivations, we can do an empirical search to find the important information-theoretic quantities that affect generalization in DNNs. I aslo presented the Dual Information Bottleneck (dualIB) framework, to find an optimal representation that resolves some of the drawbacks of the original IB. A theoretical analysis of the dualIB shows the structure of its solution and its ability to preserve the original distribution’s statistics. Within this, we focused on the variational form of the dualIB, allowing its application to DNNs.
Compression in deep learning -  an information theory perspective

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Finding generalization signals using information for infinitely-wide neural networks.
Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Representation Compression in Deep Neural Network

An information theoretic viewpoint on the behavior of deep networks optimization processes and their generalization abilities by the information plane and how compression can help.
Representation Compression in Deep Neural Network

On the Information Theory of Deep Neural Networks

Understanding Deep Neural Networks with the information bottleneck principle.
On the Information Theory of Deep Neural Networks

Open the Black Box of Deep Neural Networks

Where is the information in deep neural networks? trying to find it by looking on the information plane.
Open the Black Box of Deep Neural Networks

Contact