In this study, we explored a single-shot detector (SSD) backbone and its optimized algorithm hyperparameters for object detection, and proposed a systematic method for determining appropriate algorithm hyperparameter combinations for the SSD backbone. The VGG16 backbone for SSD has been used for object detection. The Resnet backbone won first place in the 2015 ImageNet Large Scale Visual Recognition Challenge (ILSVRC), while the VGG16 backbone ranked second place in the 2014 ILSVRC. We selected the Resnet50 backbone for SSD for vehicle image detection research because the Resnet50 backbone has a high feature extraction capability. We proposed SSD with the Resnet50 backbone and its optimized algorithm hyperparameters, called the SSD-Resnet50 model, to replace SSD with the VGG16 backbone and its optimized algorithm hyperparameters, called the SSD-VGG16 model, to enhance the vehicle image detection feature extraction capability. The Taguchi method optimized the algorithm hyperparameters of the Resnet50 and VGG16 backbones, thus improving the detection accuracies of the SSD-Resnet50 and SSD-VGG16 models, respectively. Experimental results show that the SSD-Resnet50 model using 300 × 300 × 3 input images achieved a detection accuracy of an average average precision (AP) of 97.15% in three independent experiments, outperforming the SSD-VGG16 model using 300 × 300 × 3 input images with an average AP of 86.83% on the test set of vehicle images. As a result, the SSD-Resnet50 model has a higher accuracy of vehicle detection in images from the Caltech cars 1999 and 2001 datasets.