In this study, we developed a sport instructor robot system for rehabilitation and exercise training of senior persons. In the system, a popular Kinect sensor, viewed as the eyes of a humanoid robot, was employed to capture and recognize the gestures made by a person. For the design of the Kinect-sensor-based sport instructor robot system, there are primarily two investigating phases: the establishment of an expert system for the sport instructor robot, and the development of Kinect-sensor-based gesture recognition. The humanoid robot system of the sport instructor and the Kinect-sensor-based gesture recognition system were successfully bundled together using a state machine scheme for effectively performing exercise training of the elderly. In this study, three different types of state machine dominating three different exercise training strategies were developed. In Kinect-sensor-based gesture recognition to check the correctness of a person's active gesture, to further increase the recognition accuracy, gesture activity detection (GAD) was investigated, and some GAD methods were proposed. The efficiency and effectiveness of the system were evaluated using a gesture activity database composed of seven different elderly rehabilitation actions. Experimental results demonstrated the feasibility and the superiority of the Kinect-sensor-based sport instructor robot system.