Deep Learning for Beginners

I recently found some good resources to learn deep learning. I would like to share my findings here:

- Representation Learning: A Review and New Perspectives: This is a good paper to start learning the history of deep learning. It includes references of important papers that make deep learning popular. One thing I really appreciate about the paper is that it uses PCA, one of the well-known algorithms, to learn deep learning in depth. It also provides references to important results of deep learning in many real-world problems. It is a long paper to read but it is definitely worth the time.

- Deep Learning Tutorial: If you want some background trainings in deep learning, this is a good place to start. It goes through the math behind it and have some exercises. Although it is academic-driven, it is necessary to understand the theory behind applying it in some real world problems.

Books on Deep Learning

Microsoft has published a book on Deep Learning and it is called "DEEP LEARNING: METHODS AND APPLICATIONS". This is the reference book to go for if you want to learn about Deep Learning from theory to practice.

Paper Reviewed: DeCAF - A Deep Convolutional Activation Feature for Generic Visual Recognition

I just read a very interesting and exciting paper on Computer Vision based on Deep Learning. The title of the paper is "DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition". It demonstrates methods to visualize features learned inside a Deep Neural Network more specifically the neuron activations of a specific layer inside the network. It also uses the parameters learned in the convolutional layers to demonstrate that the features learned are generic enough to be used for other classification tasks. The author holds the parameters learned in the convolutional layers and only trained the network in the final layer for classification. This demonstrates that the parameters are be "reused" across different classification problems based on images and achieve good generalization.

The big deal here is both theoretical and practical. It is theoretical because the learning is "transferrable" meaning once the machine has done learning a visual task (e.g. differentiate a face or non-face images), it is relatively easy to train it to learn another visual task. It is practical because you can now train your machine on a big dataset that is similar to your task and then use the learned parameters to fine-tune your machine for the task of interest. Note that for deep learning, the most difficult layers to train is the first few layers of the network. Using human vision as an analogy, the first few layers are modelling the eye (i.e. iris, cornea, retina) the last few layers are the visual cortex of the brain. It is very important to get the right information to the brain before making any classification! However, once you have a developed eye (i.e. a set of trained parameters), the brain just needs to discriminate the inputs to make a decision. It is common that the training dataset for the task of interest is small which is easy to be overfitted but now we can try to train the machine with a similar and big dataset and then fine-tune with the actual dataset of the task, this will make it very useful!

Practical Applications of Deep Learning in Facebook

In the master thesis I wrote which is based on this paper, it emphasizes the practical aspects of convolutional neural networks for image/video processing. The convolutional neural network which is developed by Yann LeCun demonstrates in the early days before deep/representation learning that Neural Network can be trained to extract features by learning application specific feature detectors directly from images. Note that this algorithm (or variant of it) has been used in banks to read cheques for very long time. So I would expect that there will be a lot of practical applications of it in Facebook as well after Yann LeCun starts working with Facebook.

A Great Tutorial for NLP using Deep Learning

There is a tutorial given at Stanford University on NLP (Natural Language Processing) and how to make use of deep learning for NLP.

You can find the video tutorial here: Part 1 and Part 2.

From this tutorial, I learned that the difficulty of NLP is to transform the word (in text) into a vector (a tuple of length N with numerical values) which preserve the property of the word (e.g. its synonyms) in high dimensional space. For instance, the distance between the vector of the word "Happy" and the vector of the word "Joyful" should be very closed in the subspace that is defined by the representation. Learning a good representation of the words or sequence of words (e.g. sentences) is very important because it has a direct effect to the performance of the NLP task you have in hand. This is where deep learning might fill in the gap. Instead of handcrafting the representation of the words, it goes a step further to learn the representation of the words using deep learning techniques presented in the tutorial. You can see the results of many published papers about the performance improvements of using deep learning for NLP.