Nowadays, more and more people are wearing smartwatches in their daily lives. The various sensors embedded in smartwatches bring the ability to evaluate users’ status as well as the risk of privacy issues. For example, if users are typing on keyboards while wearing smartwatches, the attacker can know the typed contents from the sensor data collected by the malicious applications that are installed on the targets’ smartwatches. In this paper, we propose WatchLogger, a framework using audio and accelerometer signals to recognize the English words being typed, to demonstrate how to implement the smartwatch-based side-channel attack. In contrast with previous studies that focused on the recognition of each key or pair of keys being pressed, WatchLogger aims to perform recognition on the scale of words. To achieve this goal, WatchLogger exploits the audio signals for segmentation and the accelerometer signals for classification. In addition, we propose an ensemble classification model to deal with the problem caused by too many words. Finally, we build the WTW-100 dataset (Wearable Typed Words dataset with 100 classes of words) using data from four participants and conduct experiments on the basis of this dataset. The experimental results show accuracies of 98.31 and 99.62% and F1 scores of 0.9745 and 0.9855 for keystroke detection and classification, respectively, and an accuracy of 79.76% for word classification, indicating a considerable performance of WatchLogger.