With the rapid development of sensors and sensor technology, facial expression recognition (FER) systems can be developed and applied to real-world scenarios. Vision scan sensors and ambient light sensors capture clear and noise-free images of faces. However, in the real world, annotating large facial expressions is challenging owing to inconsistent labels, which are caused by the annotators’ subjectivity and the facial expressions’ ambiguity. Moreover, current studies present limitations when addressing facial expression differences due to the gender gap. We not only rely on visual sensors for FER but also utilize nonvisual sensors. Therefore, in this paper, we propose a self-cure dual-branch network (SC-DBN) for FER, which automatically prevents deep networks from overfitting ambiguous samples. First, on the basis of SC-DBN, a two-branch training method is designed, taking full advantage of the gender information. Furthermore, a self-attention mechanism highlights the essential samples and weights, each with a regular weighting. Finally, a relabeling module is used to modify the labels of these samples in inconsistent labels. Many experiments on public datasets show that SC-DBN can effectively integrate gendered information and self-cure networks to improve performance.