Recently, acoustic speech recognition and visual face identification have become mature techniques widely used in real-life applications. However, human cognitive recognition issues such as human emotion classification are still a major challenge. In this study, restlessness level recognition using a deep learning scheme of the Visual Geometry Group (VGG) convolution neural network (CNN) with input acoustic speech and visual face sensor data is presented for home care applications. The well-known Microsoft Kinect device is employed with a red–green–blue sensor and an array of microphones to acquire facial expression and vocal variation data, respectively. Both VGG-16 and VGG-19 CNN deep learning models are used to evaluate the effectiveness of restlessness level classification in three different data modality inputs: acoustic speech observations alone, visual face observations alone, and combined speech and face observations. Experimental results on categorizing nine defined restlessness levels demonstrate the effectiveness of the presented approach. A specific group with problems of restlessness can benefit from the immediate care that can be provided intelligently by using the system proposed in this study.