It is important to comprehend students’ academic emotions in interactive teaching environments. Academic emotions refer to facial expressions that students display along with their academic performance in a learning process. By noting students’ academic emotions, teachers can provide the most suitable teaching material according to the emotions to improve their academic performance and motivation. The results can also be subsequently applied to adaptive learning. Recently, some researchers have attempted to study academic emotions with the aid of facial and emotion recognition technologies. However, most studies focused on the analysis and recognition of a single image. It was not considered that academic emotions are a continuous expression in response to the learning situation over a period of time. To address this problem, a continuous facial emotional pattern recognition method based on deep learning is proposed in this study to analyze academic emotions. This method combines the convolutional neural network (CNN) and the long short-term memory (LSTM) network for deep learning to recognize and analyze the continuous facial academic emotional pattern of students and thus recognize academic emotions. Through this method, the e-learning system can understand the learning progress of students quickly and accurately, and offer the students appropriate teaching materials to enhance their academic performance and motivation. The experimental results showed that the recognition accuracies of the CNN model and CNN plus LSTM were 72.47 and 84.33%, respectively. The combination of two neural networks improved the accuracy by approximately 12% compared with that for the CNN alone.