# Residual neural networks are an exciting area of deep learning research

The identify function is simply id(x) = x; given an input x it returns the same value x as output.

I am highlighting several recent papers that show the potential of residual neural networks. Residual neural networks, or ResNets ( Deep Residual Learning for Image Recognition ), are a technique Microsoft introduced in 2015. The ResNet technique allows deeper neural networks to be effectively trained. ResNets won the ImageNet competition in December with a 3.57% error score. Recently, researchers have published several papers augmenting the ResNet model with some interesting improvements.

A residual neural network layer approximately calculates y = f(x) + id(x) = f(x) + x.

ResNets tweak the mathematical formula for a deep neural network. They tweak the layers’ equations to introduce identity connections between them. The identify function is simply id(x) = x ; given an input x it returns the same value x as output. A layer in a traditional neural network learns to calculate a function y = f(x) . A residual neural network layer approximately calculates y = f(x) + id(x) = f(x) + x . Identity connections enable the layers to learn incremental, or residual, representations. The layers can start as the identity function and gradually transform to be more complex. Such evolution occurs if the parameters for the f(x) part begin at or near zero .

The ResNet technique has shown deeper neural network models can train successfully. The model Microsoft used in ImageNet 2015 has 152 layers. That’s significantly deeper than previous competition winners. Deeper models tend to hit obstacles during the training process. The gradient signal vanishes with increasing network depth. But the identity connections in ResNets propagate the gradient throughout the model.

Researchers have hypothesized that deeper neural networks have more representational power. Deeper nets gain this power from hierarchically composing shallower feature representations into deeper representations. For instance, in face recognition, pixels make edges and edges make corners. Corners define facial features such as eyes, noses, mouths and chins. Facial features compose to define faces.