Curated list of awesome lists
Awesome Deep Learning Resources
This is a rough list of my favorite deep learning resources. It has been useful to me for learning how to do deep learning, I use it for revisiting topics or for reference.
I (Guillaume Chevalier) have built this list and got through all of the content listed here, carefully.
Here are the all-time Google Trends, from 2004 up to now, September 2017:
You might also want to look at Andrej Karpathy's new post about trends in Machine Learning research.
I believe that Deep learning is the key to make computers think more like humans, and has a lot of potential. Some hard automation tasks can be solved easily with that while this was impossible to achieve earlier with classical algorithms.
Moore's Law about exponential progress rates in computer science hardware is now more affecting GPUs than CPUs because of physical limits on how tiny an atomic transistor can be. We are shifting toward parallel architectures
[read more]. Deep learning exploits parallel architectures as such under the hood by using GPUs. On top of that, deep learning algorithms may use Quantum Computing and apply to machine-brain interfaces in the future.
I find that the key of intelligence and cognition is a very interesting subject to explore and is not yet well understood. Those technologies are promising.
Machine Learning by Andrew Ng on Coursera - Renown entry-level online class with certificate. Taught by: Andrew Ng, Associate Professor, Stanford University; Chief Scientist, Baidu; Chairman and Co-founder, Coursera.
Deep Learning Specialization by Andrew Ng on Coursera - New Deep Learning series of courses (1, 2, 3, 4, 5) by Andrew Ng, now with Python rather than Matlab/Octave.
Deep Learning by Google - Good intermediate to advanced-level course covering high-level deep learning concepts, I found it helps to get creative once the basics are acquired.
Machine Learning for Trading by Georgia Tech - Interesting class for acquiring basic knowledge of machine learning applied to trading and some AI and finance concepts. I especially liked the section on Q-Learning.
Neural networks class by Hugo Larochelle, Université de Sherbrooke - Interesting class about neural networks available online for free by Hugo Larochelle, yet I have watched a few of those videos.
Posts and Articles
Librairies and Implementations
Those are resources I have found that seems interesting to develop models onto.
Other Math Theory
Gradient Descent Algorithms & Optimization Theory
Complex Numbers & Digital Signal Processing
Okay, signal processing might not be directly related to deep learning, but studying it is interesting to have more intuition in developing neural architectures based on signal.
Recurrent Neural Networks
Convolutional Neural Networks
What is the Best Multi-Stage Architecture for Object Recognition? - Awesome for the use of "local contrast normalization".
ImageNet Classification with Deep Convolutional Neural Networks - AlexNet, 2012 ILSVRC, breakthrough of the ReLU activation function.
Visualizing and Understanding Convolutional Networks - For the "deconvnet layer".
Fast and Accurate Deep Network Learning by Exponential Linear Units - ELU activation function for CIFAR vision tasks.
Very Deep Convolutional Networks for Large-Scale Image Recognition - Interesting idea of stacking multiple 3x3 conv+ReLU before pooling for a bigger filter size with just a few parameters. There is also a nice table for "ConvNet Configuration".
Going Deeper with Convolutions - GoogLeNet: Appearance of "Inception" layers/modules, the idea is of parallelizing conv layers into many mini-conv of different size with "same" padding, concatenated on depth.
Highway Networks - Highway networks: residual connections.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift - Batch normalization (BN): to normalize a layer's output by also summing over the entire batch, and then performing a linear rescaling and shifting of a certain trainable amount.
U-Net: Convolutional Networks for Biomedical Image Segmentation - The U-Net is an encoder-decoder CNN that also has skip-connections, good for image segmentation at a per-pixel level.
Deep Residual Learning for Image Recognition - Very deep residual layers with batch normalization layers - a.k.a. "how to overfit any vision dataset with too many layers and make any vision model work properly at recognition given enough data".
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning - For improving GoogLeNet with residual connections.
WaveNet: a Generative Model for Raw Audio - Epic raw voice/music generation with new architectures based on dilated causal convolutions to capture more audio length.
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling - 3D-GANs for 3D model generation and fun 3D furniture arithmetics from embeddings (think like word2vec word arithmetics with 3D furniture representations).
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour - Incredibly fast distributed training of a CNN.
Densely Connected Convolutional Networks - Best Paper Award at CVPR 2017, yielding improvements on state-of-the-art performances on CIFAR-10, CIFAR-100 and SVHN datasets, this new neural network architecture is named DenseNet.
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation - Merges the ideas of the U-Net and the DenseNet, this new neural network is especially good for huge datasets in image segmentation.
YouTube and Videos
Misc. Hubs & Links
Hacker News - Maybe how I discovered ML - Interesting trends appear on that site way before they get to be a big deal.
DataTau - This is a hub similar to Hacker News, but specific to data science.
Naver - This is a Korean search engine - best used with Google Translate, ironically. Surprisingly, sometimes deep learning search results and comprehensible advanced math content shows up more easily there than on Google search.
Arxiv Sanity Preserver - arXiv browser with TF/IDF features.
To the extent possible under law, Guillaume Chevalier has waived all copyright and related or neighboring rights to this work.