Emotional Speech and Chat

This project aims to implement and train all of the component models necessary for a chatbot which focuses on emotional relevance in chat.

The dialogue generation is completed using a reinforcement learning model.

The emotion recognition model classifies an input audio signal into one of six emotions:

• happy
• sad
• frustrated
• neutral
• angry
• excited

The speech recognition model is a reimplementation of Listen, Attend, and Spell by William Chen, et. al

Training functions are located in Jupyter notebooks for ease of running.

Datasets used for training

• IEMOCAP emotional dialogue: emotional speech recognition
• Mozilla Common Voice: speech recognition
• Cornell Movie Dialog Corpus: conversation generation

Directories

• rl_model: reinforcement learning models and training
• speech_models: emotion and speech recognition models
• utils: feature extraction and data loading helpers

Key References

[1] Chan, William, et al. "Listen, attend and spell." (2015).

[2] Chernykh, Vladimir, Grigoriy Sterling, and Pavel Prihodko. "Emotion recognition from speech with recurrent neural networks." arXiv preprint arXiv:1701.08071 (2017).

[3] Li, Jiwei, et al. "Deep reinforcement learning for dialogue generation." rXiv preprint arXiv:1606.01541 (2016).