To avoid the interference of the material’s surface factors in Brinell indentation images, which adversely affect measurement accuracy, an automatic measurement algorithm for Brinell indentations based on a convolutional neural network (CNN) is proposed. To eliminate the influence of factors such as scratches and collapses of the material surface on the measurement accuracy, the Brinell indentation image as the foreground is divided by the proposed algorithm and an indentation bounding box calculation is carried out after obtaining the binarized pixel mask of the indentation area. The measurement accuracy of the Brinell indentation image under the interference of some material background factors is thus improved. Our experimental results show that compared with the traditional automatic measurement method for Brinell indentations, Brinell indentation images with a complicated background environment can be measured more accurately by the proposed method, with the maximum relative error reduced by 20%. Moreover, the proposed method has strong applicability and high robustness for different material surfaces under different illumination conditions.