Clik here to view.
How to Escape Saddle Points Efficiently
A core, emerging problem in nonconvex optimization involves the escape of saddle points. While recent research has shown that gradient descent (GD) generically escapes saddle points asymptotically (see...
View ArticleUnsupervised learning, one notion or many?
Unsupervised learning, as the name suggests, is the science of learning from unlabeled data. A look at the wikipedia page shows that this term has many interpretations:(Task A)Learning a distribution...
View ArticleClik here to view.
Do GANs actually do distribution learning?
This post is about our new paper, which presents empirical evidence that current GANs (Generative Adversarial Nets) are quite far from learning the target distribution. Previous posts had introduced...
View ArticleClik here to view.
Generalization Theory and Deep Nets, An introduction
Deep learning holds many mysteries for theory, as we have discussed on this blog. Lately many ML theorists have become interested in the generalization mystery: why do trained deep nets perform well on...
View ArticleClik here to view.
Proving generalization of deep nets via compression
This post is about my new paper with Rong Ge, Behnam Neyshabur, and Yi Zhang which offers some new perspective into the generalization mystery for deep nets discussed in my earlier post. The new paper...
View ArticleClik here to view.
Can increasing depth serve to accelerate optimization?
“How does depth help?” is a fundamental question in the theory of deep learning. Conventional wisdom, backed by theoretical studies (e.g. Eldan & Shamir 2016; Raghu et al. 2017; Lee et al. 2017;...
View ArticleClik here to view.
Limitations of Encoder-Decoder GAN architectures
This is yet another post about Generative Adversarial Nets (GANs), and based upon our new ICLR’18 paper with Yi Zhang. A quick recap of the story so far. GANs are an unsupervised method in deep...
View ArticleClik here to view.
Deep-learning-free Text and Sentence Embedding, Part 1
Word embeddings (see my old post1 and post2) capture the idea that one can express “meaning” of words using a vector, so that the cosine of the angle between the vectors captures semantic similarity....
View ArticleClik here to view.
Deep-learning-free Text and Sentence Embedding, Part 2
This post continues Sanjeev’s post and describes further attempts to construct elementary and interpretable text embeddings. The previous post described the the SIF embedding, which uses a simple...
View ArticleClik here to view.
When Recurrent Models Don't Need to be Recurrent
In the last few years, deep learning practitioners have proposed a litany of different sequence models. Although recurrent neural networks were once the tool of choice, now models like the...
View ArticleClik here to view.
Simple and efficient semantic embeddings for rare words, n-grams, and...
Distributional methods for capturing meaning, such as word embeddings, often require observing many examples of words in context. But most humans can infer a reasonable meaning from very few or even a...
View ArticleClik here to view.
Understanding optimization in deep learning by analyzing trajectories of...
Neural network optimization is fundamentally non-convex, and yet simple gradient-based algorithms seem to consistently solve such problems. This phenomenon is one of the central pillars of deep...
View ArticleClik here to view.
The search for biologically plausible neural computation: A...
This is the second post in a series reviewing recent progress in designing artificial neural networks (NNs) that resemble natural NNs not just superficially, but on a deeper, algorithmic level. In...
View ArticleClik here to view.
Contrastive Unsupervised Learning of Semantic Representations: A...
Semantic representations (aka semantic embeddings) of complicated data types (e.g. images, text, video) have become central in machine learning, and also crop up in machine translation, language...
View ArticleIs Optimization a Sufficient Language for Understanding Deep Learning?
In this Deep Learning era, machine learning usually boils down to defining a suitable objective/cost function for the learning task at hand, and then optimizing this function using some variant of...
View ArticleClik here to view.
Landscape Connectivity of Low Cost Solutions for Multilayer Nets
A big mystery about deep learning is how, in a highly nonconvex loss landscape, gradient descent often finds near-optimal solutions —those with training cost almost zero— even starting from a random...
View ArticleClik here to view.
Understanding implicit regularization in deep learning by analyzing...
Sanjeev’s recent blog post suggested that the conventional view of optimization is insufficient for understanding deep learning, as the value of the training objective does not reliably capture...
View ArticleClik here to view.
Ultra-Wide Deep Nets and Neural Tangent Kernel (NTK)
(Crossposted at CMU ML.)Traditional wisdom in machine learning holds that there is a careful trade-off between training error and generalization gap. There is a “sweet spot” for the model complexity...
View ArticleClik here to view.
Exponential Learning Rate Schedules for Deep Learning (Part 1)
This blog post concerns our ICLR20 paper on a surprising discovery about learning rate (LR), the most basic hyperparameter in deep learning.As illustrated in many online blogs, setting LR too small...
View ArticleClik here to view.
An equilibrium in nonconvex-nonconcave min-max optimization
While there has been incredible progress in convex and nonconvex minimization, a multitude of problems in ML today are in need of efficient algorithms to solve min-max optimization problems. Unlike...
View Article