Manual railway inspection, being both time-consuming and labor-intensive, no longer meets the demands of modern railway maintenance, where efficiency and precision are essential. To address this issue, an automatic rail fastener detection system is proposed based on the lightweight You Only Look Once version 3 (YOLOv3)-tiny architecture is proposed. This approach not only leverages the speed advantages of YOLOv3-tiny but also incorporates generative adversarial networks (GANs), along with three of its variants, Wasserstein Generative Adversarial Network (WGAN), Divergence Generative Adversarial Network (WGAN-div), and Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP), to augment the dataset and alleviate the problem of limited defective fastener samples. The quality of the generated images is quantitatively evaluated using peak signal-to-noise ratio (PSNR), structural similarity (SSIM), Fréchet inception distance (FID), and inception score (IS) metrics. These synthetic samples are then integrated into the original dataset to train the YOLO models in a joint learning process. Experimental results show that, after GAN-based augmentation, YOLOv3-tiny, YOLOv3, YOLOv7-tiny, and YOLOv7 achieve mAP0.5 scores of 97.3, 98.7, 98.6, and 98.5% respectively, with particularly significant improvements observed in mAP95. These results demonstrate the effectiveness of the proposed method in addressing data imbalance and enhancing both model accuracy and generalization. In addition, analysis of computational complexity and inference speed indicates that YOLOv3-tiny, with only 14.3 Giga Floating Point Operations (GFLOPs) of computational load, achieves an inference speed of 138.9 Frames Per Second (FPS). This high level of real-time performance, combined with high accuracy, makes YOLOv3-tiny a highly suitable choice for deployment on edge devices in practical railway inspection applications.