The Unreasonable Effectiveness of Recurrent Neural Networks There’s something magical about Recurrent Neural Networks (RNNs). I still remember when I trained my first recurrent network for Image Captioning. Within a few dozen minutes of training my first baby model (with rather arbitrarily-chosen hyperparameters) started to generate very nice looking descriptions of images that were on the edge of making sense. Sometimes the ratio of how simple your model is to the quality of the results you get out of it blows past your expectations, and this was one of those times. What made this result so shocking at the time was that the common wisdom was that RNNs were supposed to be difficult to train (with more experience I’ve in fact reached the opposite conclusion).
Open Learning As a person who does a lot of autonomous learning, the Internet in these days offer a huge amount of possibilities to read/learn about any topic you might think of. There might be more the problem of filtering out useful/good content from the nearly infinite amount of sources. Inspired by a colleague I will try to give a record of whatever I read/saw and can recommend on specific topics.
Recurrent Neural Networks Tutorial, Part 1 – Introduction to RNNs – WildML Recurrent Neural Networks (RNNs) are popular models that have shown great promise in many NLP tasks. But despite their recent popularity I’ve only found a limited number of resources that throughly explain how RNNs work, and how to implement them. That’s what this tutorial is about. Yes you should understand backprop – Andrej Karpathy – Medium When we offered CS231n (Deep Learning class) at Stanford, we intentionally designed the programming assignments to include explicit calculations involved in backpropagation on the lowest level. The students had to implement the forward and the backward pass of each layer in raw numpy. Inevitably, some students complained on the class message boards: “Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?”
Understanding LSTM Networks Posted on August 27, 2015 Recurrent Neural Networks Humans don’t start their thinking from scratch every second. As you read this essay, you understand each word based on your understanding of previous words. You don’t throw everything away and start thinking from scratch again. An overview of gradient descent optimization algorithms Note: If you are looking for a review paper, this blog post is also available as an article on arXiv. Table of contents: Gradient descent is one of the most popular algorithms to perform optimization and by far the most common way to optimize neural networks. At the same time, every state-of-the-art Deep Learning library contains implementations of various algorithms to optimize gradient descent (e.g. lasagne's, caffe's, and keras' documentation). These algorithms, however, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by.
Understanding Machine Learning Infographic Other Infographics Understanding Machine Learning Infographic Understanding Machine Learning Infographic We now live in an age where machines can teach themselves without human intervention. Gradient Descent Derivation · Chris McCormick Andrew Ng’s course on Machine Learning at Coursera provides an excellent explanation of gradient descent for linear regression. To really get a strong grasp on it, I decided to work through some of the derivations and some simple examples here. This material assumes some familiarity with linear regression, and is primarily intended to provide additional insight into the gradient descent technique, not linear regression in general. I am making use of the same notation as the Coursera course, so it will be most helpful for students of that course. For linear regression, we have a linear hypothesis function, h(x) = Ѳ0 + Ѳ1*x.
Google researchers teach AIs to see the important parts of images — and tell you about them This week is the Computer Vision and Pattern Recognition conference in Las Vegas, and Google researchers have several accomplishments to present. They’ve taught computer vision systems to detect the most important person in a scene, pick out and track individual body parts and describe what they see in language that leaves nothing to the imagination. First, let’s consider the ability to find “events and key actors” in video — a collaboration between Google and Stanford. Footage of scenes like basketball games contain dozens or even hundreds of people, but only a few are worth paying attention to. The CV system described in this paper uses a recurrent neural network to create an “attention mask” for every frame, then track relevance of each object as time proceeds.
The Neural Network Zoo - The Asimov Institute With new neural network architectures popping up every now and then, it’s hard to keep track of them all. Knowing all the abbreviations being thrown around (DCIGN, BiLSTM, DCGAN, anyone?) can be a bit overwhelming at first. So I decided to compose a cheat sheet containing many of those architectures. Most of these are neural networks, some are completely different beasts. Though all of these architectures are presented as novel and unique, when I drew the node structures… their underlying relations started to make more sense.
AI just defeated a human fighter pilot in an air combat simulator Retired United States Air Force Colonel Gene Lee recently went up against ALPHA, an artificial intelligence developed by a University of Cincinnati doctoral graduate. The contest? A high-fidelity air combat simulator. And the Colonel lost. In fact, all the other AI’s that the Air Force Research Lab had in their possession also lost to ALPHA…and so did all of the other human experts who tried their skills against ALPHA’s superior algorithms. Understanding Convolutional Neural Networks for NLP – WildML When we hear about Convolutional Neural Network (CNNs), we typically think of Computer Vision. CNNs were responsible for major breakthroughs in Image Classification and are the core of most Computer Vision systems today, from Facebook’s automated photo tagging to self-driving cars. More recently we’ve also started to apply CNNs to problems in Natural Language Processing and gotten some interesting results. In this post I’ll try to summarize what CNNs are, and how they’re used in NLP. The intuitions behind CNNs are somewhat easier to understand for the Computer Vision use case, so I’ll start there, and then slowly move towards NLP. What is Convolution?
Five surprising ways AI could be a part of our lives by 2030 Artificial intelligence (AI) has gradually become an integral part of modern life, from Siri and Spotify’s personalized features on our phones to automatic fraud alerts from our banks whenever a transaction appears suspicious. Defined simply, a computer with AI is able to respond to its environment by learning on its own—without humans providing specific instructions. A new report from Stanford University in Palo Alto, California, outlines how AI could become more integrated into people’s lives by 2030, and recommends how best to regulate it and make sure its benefits are shared equally. Here are five examples—some from this report—of AI technology that could become a part of our lives by 2030. Smart traffic lights