Short stick exercises have been attracting attention from the viewpoint of preventing falls and improving the health of elderly people and are generally performed under the guidance of instructors and nursing staff at nursing homes. However, in situations such as the COVID-19 pandemic, where people should refrain from unnecessary outings, it is advisable that individuals perform short stick exercises at home and record their exercise implementation status. In this paper, we propose an inertial measurement unit (IMU)-sensor-based short stick exercise tracking method that can automatically record the types and amounts of exercises performed using a short stick equipped with an IMU sensor. The proposed method extracts time-domain and frequency-domain features from linear acceleration and quaternion time-series data obtained from the IMU sensor and classifies the type of exercise using an inference model based on machine learning algorithms. To evaluate the proposed method, we collected sensor data from 21 young subjects (in their 20s) and 14 elderly subjects (79–95 years old), where the participants performed three sets (10 times per set) of eight basic types of short stick exercises (five types for elderly people). As a result of evaluating the proposed method using this data set, we confirmed that when LightGBM was used as the learning algorithm, it achieved F values of 90.0% and 86.6% for recognizing the type of exercise for young and elderly people, respectively.