The collapse of buildings caused by destructive earthquakes often leads to severe casualties and economic losses. After an earthquake, an accurate assessment of building damage will be essential in making plans of emergency responses. Four-polarimetric synthetic aperture radar (PolSAR) data have advantages over synthetic aperture radar (SAR) imaging data, because they are not occluded by sunlight or clouds. They also contain the most abundant information of four polarimetric channels. Therefore, a single PolSAR image can be used to identify post-earthquake building damage. It is easy to overestimate the number of collapsed buildings and the degree of damage by earthquakes when using only a traditional polarimetric decomposition method for PolSAR data. In urban areas, buildings can stand in parallel in typical SAR imaging with strong scattering features, and there are also some oriented standing buildings with lower scattering intensity or similar scattering characteristics to collapsed buildings; thus, these oriented standing buildings are often misconstrued as collapsed buildings. In this study, we propose a new texture feature, namely, the mean standard deviation (MSD) index based on the gray-level co-occurrence matrix (GLCM), to solve the overestimation of building damage caused by earthquakes. Moreover, on the basis of the improved Yamaguchi four-component decomposition method and the MSD index, we develop a method of identifying the damage of buildings using only a single post-earthquake PolSAR image. In our study case, 75000 undamaged and damaged building samples are used in the experiment. The proposed method has greatly improved the accuracy and reliability of extracted building damage information. The experimental results show identification accuracies of 82.43 and 80.30% for damaged and undamaged buildings, respectively. Compared with the traditional polarimetric decomposition method, 66.89% standing buildings are successfully isolated from the mixture of collapsed buildings using our method.