Why go off the convex path?
The notion of convexity underlies a lot of beautiful mathematics. When combined with computation, it gives rise to the area of convex optimization that has had a huge impact on understanding and...
View ArticleClik here to view.
Semantic Word Embeddings
This post can be seen as an introduction to how nonconvex problems arise naturally in practice, and also the relative ease with which they are often solved.I will talk about word embeddings, a...
View ArticleClik here to view.
Tensor Methods in Machine Learning
Tensors are high dimensional generalizations of matrices. In recent years tensor decompositions were used to design learning algorithms for estimating parameters of latent variable models like Hidden...
View ArticleClik here to view.
Nature, Dynamical Systems and Optimization
The language of dynamical systems is the preferred choice of scientists to model a wide variety of phenomena in nature. The reason is that, often, it is easy to locally observe or understand what...
View ArticleNIPS 2015 workshop on non-convex optimization
While convex analysis has received much attention by the machine learning community, theoretical analysis of non-convex optimization is still nascent. This blog as well as the recent NIPS 2015 workshop...
View ArticleClik here to view.
Word Embeddings: Explaining their properties
This is a followup to an earlier post about word embeddings, which capture the meaning of a word using a low-dimensional vector, and are ubiquitous in natural language processing. I will talk about my...
View ArticleClik here to view.
Evolution, Dynamical Systems and Markov Chains
In this post we present a high level introduction to evolution and to how we can use mathematical tools such as dynamical systems and Markov chains to model it. Questions about evolution then translate...
View ArticleClik here to view.
Stability as a foundation of machine learning
Central to machine learning is our ability to relate how a learning algorithm fares on a sample to its performance on unseen instances. This is called generalization.In this post, I will describe a...
View ArticleClik here to view.
Escaping from Saddle Points
Convex functions are simple — they usually have only one local minimum. Non-convex functions can be much more complicated. In this post we will discuss various types of critical points that you might...
View ArticleSaddles Again
Thanks to Rong for the very nice blog post describing critical points of nonconvex functions and how to avoid them. I’d like to follow up on his post to highlight a fact that is not widely appreciated...
View ArticleClik here to view.
Markov Chains Through the Lens of Dynamical Systems: The Case of Evolution
In this post, we will see the main technical ideas in the analysis of the mixing time of evolutionary Markov chains introduced in a previous post. We start by introducing the notion of the expected...
View ArticleClik here to view.
A Framework for analysing Non-Convex Optimization
Previously Rong’s post and Ben’s post show that (noisy) gradient descent can converge to local minimum of a non-convex function, and in (large) polynomial time (Ge et al.’15). This post describes a...
View ArticleClik here to view.
Linear algebraic structure of word meanings
Word embeddings capture the meaning of a word using a low-dimensional vector and are ubiquitous in natural language processing (NLP). (See my earlier post 1 and post2.) It has always been unclear how...
View ArticleClik here to view.
Gradient Descent Learns Linear Dynamical Systems
From text translation to video captioning, learning to map one sequence to another is an increasingly active research area in machine learning. Fueled by the success of recurrent neural networks in its...
View ArticleClik here to view.
The search for biologically plausible neural computation: The...
Inventors of the original artificial neural networks (NNs) derived their inspiration from biology. However, as artificial NNs progressed, their design was less guided by neuroscience facts. Meanwhile,...
View ArticleClik here to view.
Back-propagation, an introduction
Given the sheer number of backpropagation tutorials on the internet, is there really need for another? One of us (Sanjeev) recently taught backpropagation in undergrad AI and couldn’t find any account...
View ArticleClik here to view.
Generative Adversarial Networks (GANs), Some Open Questions
Since ability to generate “realistic-looking” data may be a step towards understanding its structure and exploiting it, generative models are an important component of unsupervised learning, which has...
View ArticleClik here to view.
Generalization and Equilibrium in Generative Adversarial Networks (GANs)
The previous post described Generative Adversarial Networks (GANs), a technique for training generative models for image distributions (and other complicated distributions) via a 2-party game between a...
View ArticleUnsupervised learning, one notion or many?
Unsupervised learning, as the name suggests, is the science of learning from unlabeled data. A look at the wikipedia page shows that this term has many interpretations:(Task A)Learning a distribution...
View ArticleClik here to view.
Do GANs actually do distribution learning?
This post is about our new paper, which presents empirical evidence that current GANs (Generative Adversarial Nets) are quite far from learning the target distribution. Previous posts had introduced...
View Article