In this paper, the design of a quadrotor vehicle having a person-tracking and observation system, which uses human gesture recognition, is described. The system has three operating functions, namely, object tracking, human gesture recognition, and fixed-point cruising. The tracking–learning–detection (TLD) algorithm was used to enable the autonomous tracking of the object from images. An extended Kalman filter (EKF) provides an estimate of the current position of the quadrotor vehicle, and a fuzzy-proportional integral derivative (PID) controller provides position error compensation. The principle of the human gesture recognition system is as follows. A background model is first built from images using a Gaussian mixture model (GMM)to detect the foreground image. A nonlinear support vector machine (SVM) is then employed to recognize changes of gesture and establish interactivity between the vehicle and the user. The coordinates of the vehicle are marked using a GPS for fixed-point cruising. The coordinates and parameters of the points are set so that the quadrotor vehicle can follow them during cruising. Lastly, all of the functions are incorporated into the person-tracking and gesture-recognition system in the quadrotor. The experimental results show the feasibility of the above-mentioned methods, which can help us easily recognize the various gestures in this study.