Final Project: Robot responds to human emotions

In this article, we will merge the two last robotics homeworks to have a solid final project in this module. In the first homework, we learned how to make our robot move depending on situations, thanks to the Choregraphe software. Here, it will be really useful because we want the robot to react according to what we are doing. In the second homework, we have trained a KNN model that is able to recognize the facial emotions. We will use it to make predictions, that will be sent to the robot.

If you want more details about these two parts, you can read the two articles.

---

1) Not working Project

At the beginning, we wanted to create a real face detection and use our own model. So we saved this one, and try to test it in real-time. However, it didn't work very well, so we decided to change the KNN, and use a CNN (that we have seen in the laboratories in previous articles). So we created this code, to be able to create a box around the face, with the emotion prediction as an output.

At the end, we can obtain these kinds of results:

The first half was ready. However, we were not able to put this code directly on Choregraphe, to adapt the actions of the robot to the emotion. In fact, the OpenCV library that we use a lot was disable on the software.

So tried to link the robot to our Python IDE by using the NAOSdk. We computed this code in order to make this link and send the emotion outputs:

In this part we had several problems. The main one was a conflict of version between Tensorflow, and the Python version used for Choregraphe. We first tried to create a text file, to have an intermediate (the Tensorflow version send the emotion on this file, and the Python other version can read this and send it to the robot). However, it took us a lot of time, and we had to present a final project so we didn't try more.

---

2) Working Choregraphe Project

In order to have a project that we can present, we gave up this part. We decided to use only the Choregraphe software, that worked properly. I will not enter into detail because I already wrote an article about that, but we made a small conversation between the robot and the human. We used the NAO robot here (and not Pepper like the other time).

To resume, the robot asks us if we want to play. If no, he asks us again, and if we don't want to, he becomes angry and leaves. If we want to play, he asks us to show an emotion. He has custom reactions with funny sentences. At the end, the robot asks us if we want to try again. If we want, we can do it several times. If we want to quit, the robot leaves us with a disco dance.

Here is a screenshot of this working project:

---

Conclusion

Unfortunately, we were not able to have a full final project as we wanted. However, we tried with our own model, and if we had more time, we would be able to finish it properly. Also, we did a really great job with Choregraphe, with a small amount of time. We are proud of that, even if it would have been better with our own emotion recognition model.