Introduction to Recurrent Neural Networks
- quentinaudy
- 21 oct. 2022
- 6 min de lecture
Dernière mise à jour : 23 oct. 2022
Now that we studied classic models of convolutional neural networks, we faced some problems. How can we apply these structures to inputs that don't have a fixed length ? How can we, for example, translate a sentence in real time by using AI ? This problem can be solved by introducting RNN.
In this article, I will try to give you my understanding of this course, and to talk about the laboratory we did. I also made some researches on my own to have a deeper vision of this field.
Summary
General principle (vanilla)
To begin, we can say that Recurrent Neural Networks are models that, as the other networks, takes inputs and are able by training to give an output. However, classic neural networks are not able to handle variable size data. Here, RNN can be put sequential data as input. Here, informations can also propagate in both directions, and neurons take into consideration events from the past. For that reason, RNN are really efficient for applications involving context, and can preserve sequential informations. Also, they can process sequences in real time.
First, we studied the basic model. As we saw, this model is really flexible, and can adapt to the length of the input sequence. Here is an example of a general model:
x are inputs and y are outputs associated and h are internal states. We can add the number of blocks that we want, depending on the length of our sequence. So which parameters can we train in our case ? The answer is simple because we already did that before with classic neural networks: weights. However, these latter are the same for each neuron, which is really different from classic models.
Despite all this, the training process will remain the same. We compute a loss for each output, we calculate a global loss that we want to minimize by using a gradient descent. Then, we can conduct a Back Propagation Through Time (BPTT). As the name suggests, this involves going back in time to adapt the weights of each neuron, thus improving accuracy. This backpropagation is a little different in its implementation, the gradient calculation is a little more difficult, because of the dependencies between the blocks. The calculation of the gradient loss for a state depends on the previous state, etc. It is called chain rule.
This model therefore has a major problem, which makes it hardly used in practice. Indeed, the gradient loss depends on the gradient cell state that comes from the multiplication of the same factor over and over. But if this factor is greater than 1, we can have an exploding gradient: it becomes very large, and the calculation of the loss minimisation is much more difficult. In the other case, if this factor is smaller than 1, this gradient loss will vanish. This is why other models have been created to avoid these problems.
Long short term memory
LSTM is a kind of RNN, so they have the same chain-like structure than the previous one. Only cell architecture changes, in order to avoid the vanishing gradient that we saw on the last part.
These models are defined by two internal states:
- the hidden states (same that we used in the vanilla model, used to make decisions over shorts periods of time)
- the cell states ("memory of the cell", used to avoid vanishing gradient)
Here is the structure of one cell:
One cell is also composed of 4 gates, used to filter informations:
- the forget gate (ft): decides which informations should be kept
- the input gate (it): decides the current cell's value
- the input modulation gate (gt): decides how much should be written into the cell
- the output gate (ot) decides which value will be the output of the cell
In this model, the backpropagation through time is easier. It uses similar processes as the vanilla structure, but here the loss is propagated by ct. It resolves the vanishing gradient problem.
This structure is widely used in practice, and deep RNN architectures are created in order to improve accuracies, because they extract different features that are really useful.
RNN Applications
Now that we have seen the first models used and how they work, we can talk about the possible and widespread applications that use RNN.
The first important application is to predict the next element using the previous ones. It can be used to generate parts of images that miss (we can complete areas in Photoshop with this method), but also can be used to predict the next word of a sentence given the first words. This latter is really efficient for a task that we will study during the laboratory: image captioning. Indeed, this model can encode an image by passing it through a classic CNN, which will be linked to our RNN that will generate the words. Here is a schematic version:
This is an example of what we call Seq2Seq (Sequence to Sequence), it is also used for real time translation for example.
Also, if we couple different neural networks, we can obtain more diverse models with larger tasks. We can recognize gestures captured by cameras if we enter the video into CNNs, and if we use an RNN to generate the name of the gesture.
This model is really efficient, but it has some limitations. In fact, the first problem is that the encoding part can be really long, and act like a bottle neck before decoding with the RNN. The second problem is that if the sequence is too long, the real time action won't be efficient.
The solution to this issue is to add another notion to our RNN: attention.
Attention
Attention is the ability to decide what to focus on, and to be selective about what you are looking at. It is particullary useful when learning. We can find two types of attention.
The first one is implicit attention. While learning, our RNN imitates naturally a human comportment by focusing on the most important parts. Everything is done while trying to minimize the loss. This implicit attention is computed thanks to Jacobian, which will be able to identify the important areas of the input, that will take an essential role in the prediction. However, this attention is not enough.
For that reason, we introduced explicit attention. This one is more close-to-human, and enables to reduce computations, better understand what the neural network does because it acts like us, and simplify sequential processing.
This attention is an additionnal set of outputs, that is implemented in parallel from the classical model.
These models of LSTM with attention can handle well long-range dependencies between elements of a sequence. These kinds of networks can capture strict temporal context in real-time. It exists different types of attention models.
For example, during the image captioning process, we can more easily associate words with parts of the image by focusing only on areas. It is more visible with an image version, where the white part is the part were the attention is maximum:
The applications are multiple (sentiment analysis, image captioning or image generation for example). However, some limitations remains, because of the RNN structure. In fact, transfer learning does not work very well, and the parallelization is not possible. We can resolve a part of these problems by using another structure: transformers.
Transformers
Transformers are different kinds of structures that focused on one idea: get rid of the reccurent aspects, in order to only keep attention on attention mechanisms. They are formed of an encoder and a decoder architecture, and used what we call a self-attention mechanism. The Transformers are multi-headed, they can track different types of attention, and therefore pay attention to different properties of a sequence at the same time. This model is for example used for the Alpha Star system that beats players in the strategy game Starcraft.
Now that we studied the theorical aspect, let's speak a little bit about what we did during the laboratory.
Laboratory
The code was really difficult and long in this laboratory, so I will not detail what we did. But to summazie in a few sentences, the goal was to have a RNN capable of captioning images.
The first part was to implement the dataset, with 5 differents captions for each image (it will correspond to our labels). Now, we will pass through the pre-processing part. The pre-processing is here to make some operations on the images, in order to directly put them as input. This part creates a tokenizer that will convert words into tokens, it creates a vocabulary that will store all the words that will be read, and the token associated. Unknown words will have a default token. On the other hand, we take the images of the dataset and do the opposite: we convert and reshape the images, and convert their labels into tokens. Then, we padd the captions, and create batches with collate. We have a function that display the batches.
In this example, we have an image that is pre-processed, and his ground truth that was tokenized to be the input of the neural network.
Then, we take all this, and put the data into the three folders: train, validation and test. We can now build the model (what we did), and train our batches on it. I managed to create the model and started performing some training, but I didn't reach the goal, that was to obtain an accuracy.
Commentaires