For the intelligent manufacturing field, after finishing the cutting process, a metal surface may have various defects such as scratches, residues, and dirt. However, the conventional method of determining defects has the disadvantages of being time-consuming and expensive. In addition, it is necessary to consider the cost of collecting samples and the labor cost when practically collecting samples from industries. Therefore, in this study, we optimized the determination of the defects of the production component by a deep learning (DL) model with a few samples and used an image sensor to take pictures of the specific area of the component. Meanwhile, an entropy calculation method is proposed to determine the most suitable kernel size of a convolution layer. We analyzed and established a deep learning model to determine whether the finished products of a vision inspection machine have defects using only a few samples. We compared the pros and cons of DarkNet-53, which is a convolutional neural network (CNN) that is 53 layers deep, and AlexNet, which is a deep CNN, with the DenseNet-201 model in the experiments. The obtained experimental results indicate that the proposed method can effectively increase the rate of recognition between defective and nondefective samples and reduce the training cost. The results of this paper may contribute to the development of a novel diagnostic technique and also be helpful for the intelligent manufacturing industry.