Spheroidization is a process that uses a high temperature to change the properties of metals and it is often used in physical metallurgy. Metallographic inspection is an important method of inspecting the quality of metal materials after spheroidization. In the process of metallographic inspection, a high-power optical microscope combined with a digital camera is usually used to obtain an image of the spheroidized metal. A light sensor, which is a charge-coupled device in the camera, is used to convert the image observed by the microscope into an electronic image signal. In this paper, we present an image recognition method with a deep learning neural network (NN) to inspect the metallographic grade of spheroidized metal. Three different transfer learning models are incorporated in the NN structure for feature extraction for comparison. The overall aim of our study is to reduce the shortcomings and inconvenience of traditional manual inspection and increase the judgment accuracy of metallographic analysis. In experiments, 203 metallographic images of size 1536 × 2048 were used for the learning and testing of the NN. The metallographic grade of the spheroidized metal was evaluated using the deep learning NN model.