Clik here to view.
Implicit Regularization in Hierarchical Tensor Factorization and Deep...
The ability of large neural networks to generalize is commonly believed to stem from an implicit regularization — a tendency of gradient-based optimization towards predictors of low complexity. A lot...
View ArticleClik here to view.
Predicting Generalization using GANs
A central problem of generalization theory is the following: Given a training dataset and a deep net trained with that dataset, give a mathematical estimate of the test error.While this may seem...
View ArticleClik here to view.
Does Gradient Flow Over Neural Networks Really Represent Gradient Descent?
TL;DRA lot was said in this blog (cf. post by Sanjeev) about the importance of studying trajectories of gradient descent (GD) for understanding deep learning. Researchers often conduct such studies by...
View ArticleClik here to view.
Implicit Regularization in Tensor Factorization: Can Tensor Rank Shed...
In effort to understand implicit regularization in deep learning, a lot of theoretical focus is being directed at matrix factorization, which can be seen as linear neural networks. This post is based...
View ArticleClik here to view.
Rip van Winkle's Razor, a Simple New Estimate for Adaptive Data Analysis
Can you trust a model whose designer had access to the test/holdout set? This implicit question in Dwork et al 2015 launched a new field, adaptive data analysis. The question referred to the fact that...
View ArticleClik here to view.
When are Neural Networks more powerful than Neural Tangent Kernels?
The empirical success of deep learning has posed significant challenges to machine learning theory: Why can we efficiently train neural networks with gradient descent despite its highly non-convex...
View ArticleClik here to view.
Beyond log-concave sampling (Part 3)
In the first post of this series, we introduced the challenges of sampling distributions beyond log-concavity. In Part 2 we tackled sampling from multimodal distributions: a typical obstacle occuring...
View ArticleClik here to view.
Beyond log-concave sampling (Part 2)
In our previous blog post, we introduced the challenges of sampling distributions beyond log-concavity. We first introduced the problem of sampling from a distibution $p(x) \propto e^{-f(x)}$ given...
View ArticleClik here to view.
Can implicit regularization in deep learning be explained by norms?
This post is based on my recent paper with Noam Razin (to appear at NeurIPS 2020), studying the question of whether norms can explain implicit regularization in deep learning. TL;DR: we argue they...
View ArticleClik here to view.
How to allow deep learning on your data without revealing the data
Today’s online world and the emerging internet of things is built around a Faustian bargain: consumers (and their internet of things) hand over their data, and in return get customization of the world...
View ArticleClik here to view.
Mismatches between Traditional Optimization Analyses and Modern Deep Learning
You may remember our previous blog post showing that it is possible to do state-of-the-art deep learning with learning rate that increases exponentially during training. It was meant to be a dramatic...
View ArticleClik here to view.
Beyond log-concave sampling
As the growing number of posts on this blog would suggest, recent years have seen a lot of progress in understanding optimization beyond convexity. However, optimization is only one of the basic...
View ArticleClik here to view.
Training GANs - From Theory to Practice
GANs, originally discovered in the context of unsupervised learning, have had far reaching implications to science, engineering, and society. However, training GANs remains challenging (in part) due to...
View ArticleClik here to view.
An equilibrium in nonconvex-nonconcave min-max optimization
While there has been incredible progress in convex and nonconvex minimization, a multitude of problems in ML today are in need of efficient algorithms to solve min-max optimization problems. Unlike...
View ArticleClik here to view.
Exponential Learning Rate Schedules for Deep Learning (Part 1)
This blog post concerns our ICLR20 paper on a surprising discovery about learning rate (LR), the most basic hyperparameter in deep learning.As illustrated in many online blogs, setting LR too small...
View ArticleClik here to view.
Ultra-Wide Deep Nets and Neural Tangent Kernel (NTK)
(Crossposted at CMU ML.)Traditional wisdom in machine learning holds that there is a careful trade-off between training error and generalization gap. There is a “sweet spot” for the model complexity...
View ArticleClik here to view.
Understanding implicit regularization in deep learning by analyzing...
Sanjeev’s recent blog post suggested that the conventional view of optimization is insufficient for understanding deep learning, as the value of the training objective does not reliably capture...
View ArticleClik here to view.
Landscape Connectivity of Low Cost Solutions for Multilayer Nets
A big mystery about deep learning is how, in a highly nonconvex loss landscape, gradient descent often finds near-optimal solutions —those with training cost almost zero— even starting from a random...
View ArticleIs Optimization a Sufficient Language for Understanding Deep Learning?
In this Deep Learning era, machine learning usually boils down to defining a suitable objective/cost function for the learning task at hand, and then optimizing this function using some variant of...
View ArticleClik here to view.
Contrastive Unsupervised Learning of Semantic Representations: A...
Semantic representations (aka semantic embeddings) of complicated data types (e.g. images, text, video) have become central in machine learning, and also crop up in machine translation, language...
View Article![ONEW – KAKERA -Unmei no Piece- – EP [iTunes Plus M4A]](http://is1-ssl.mzstatic.com/image/thumb/Music221/v4/d0/fb/86/d0fb86a5-ba60-4bea-41fb-7f54c125f518/VEATP-45908.jpg/296x296bb.webp)
/pic9814888.png)
