A gestural input to control artifacts and access the digital world is an essential part of highly usable systems. In this article, we propose a gesture recognition method that leverages the sound generated by the friction between a surface such as a table and a finger or pen, in which 17 different gestures are defined. The gesture recognition process is regarded as a 17-class classification problem; 89 classification features are defined to represent the envelope of each input sound, while a hierarchical classifier structure is employed to increase the accuracy of confusable classes. Offline experiments show that the highest accuracy is 0.954 under a condition where the classifiers are customized for each user, while an accuracy of 0.854 is obtained under a condition where the classifiers are trained without using the data from test users. We also confirm the effectiveness of the hierarchical classifier approach compared with a single-flat-classifier approach and that of a feature engineering approach compared with a feature learning approach. The information of individual features is also presented.