Recently, compound image sensor devices have been widely used to construct many next-generation human–machine interaction applications, including hand gesture action recognition. Such devices, generally known as RGB-D devices, contain an RGB color camera and a depth sensor set. The depth sensor in RGB-D devices is essentially a set of a specific type of sensor and comprises one IR projector and one IR camera. This structure of the depth sensor inevitably generates an undesired shadow effect, adversely affecting hand gesture recognition. To tackle this issue and alleviate the shadow effect on hand gesture recognition, we have developed a serial binary image extraction approach. The proposed approach is essentially composed of two consecutive computation phases, phase-1 and phase-2 binary image extraction. In this work, the Kinect compound sensor device is employed to capture hand gesture depth images. The deep learning model, a visual geometry group (VGG)-type convolutional neural network (CNN), i.e., the well-known VGG-CNN, is utilized to evaluate the recognition effectiveness of improved hand gesture depth images derived from serial binary image extraction. Ten hand gestures that are common in daily life are chosen to evaluate depth-sensor-based interactive action recognition. Experimental results show that the proposed serial binary image extraction can effectively eliminate the undesired shadow region in hand gesture depth images and significantly improve the recognition accuracy of VGG-type CNN hand gesture recognition. The proposed depth-sensor-based hand gesture recognition approach can benefit people requiring interaction action recognition and further promote smart gesture communication.