At present, the main focus in the development of convolutional neural networks (CNNs) is deepening the network model to improve accuracy. However, this may increase the numbers of parameters and calculations in the network architecture. When the network model is applied to mobile devices and embedded systems, the storage capacity, computing performance, and memory will become major limitations. A local binary convolutional neural network (LBCNN) has been proposed to reduce the numbers of parameters and calculations. In the LBCNN, the convolutional layer of the CNN is replaced by a local binary convolution (LBC) module. In the LBC module, there is a pre-initialized fixed parametric filter layer. Since the parameters of the filter are generated in a random manner, the result is different each time and therefore unstable. Therefore, to provide a stable and efficient recognition technique for image sensors, we propose a genetic-algorithm-based local binary convolutional neural network (GA-LBCNN) for gender recognition in this study. The genetic algorithm (GA) is used to search for the best filter parameters of the LBCNN. LeNet is adopted as the basic model architecture, and two datasets acquired from image sensors, the CIA and MORPH datasets, are used to perform face gender classification. According to the evaluation results, LBC successfully reduces the numbers of parameters and calculations. Experimental results show that the classification accuracy of the proposed GA-LBCNN reaches 88.8 and 98.2% for the CIA and MORPH datasets, respectively. Compared with the conventional LBCNN, the classification accuracy of the proposed GA-LBCNN is increased by 7.2 and 1.1%, respectively, for the two datasets.