Human–computer interaction (HCI) is a trend of technology evolution toward making our life more convenient and easier. In some circumstances where hands cannot conveniently touch equipment, the hand gesture recognition system is a good solution for HCI. A real-time hand gesture sensing and recognition system is proposed and its application to TV channel and volume control is examined in this study. The digital signal processor (DSP) DM6437 of Texas Instruments is used in our portable hand gesture recognition system. For the real-time recognition of hand gestures, we propose a novel finger skin pixel algorithm to quickly and easily distinguish the hand in a complex image. The region of interest is used to reduce the amount of computation. Hand features are detected in two steps. First, the thumb and pinkie are recognized on the basis of significant shape features. Second, a circle is defined from the hand center for threshold decomposition to count the number of fingers. The hand gesture recognition rate is 94.3%. The hand trace direction can be found easily by using a hand gesture center point in our system. Finally, our system is applied to TV channel and volume control using hand gestures and hand tracing.