In an elevator car, subway, bus, high-speed railway, and other public places, smoking harms the smokers themselves as well as others near them, and seriously affects the public order, and may even cause fire. Thus, smoking detection is important in some public places. The traditional target detection algorithm based on neural networks is used for detection, but it suffers the problems of low speed, high false detection rate, easy loss of the characteristics of small targets, poor generalization, and low robustness. To solve these problems, the improved You Only Look Once version 4 (YOLOv4)-based real-time smoking detection model is designed. For the common target detection algorithm for a small target, the detection effect is not ideal. Therefore, by k-means++ clustering, the a priori box is recalculated and designed to enhance the adaptability of scale and obtain a better anchor. An attention mechanism is also added to improve the accuracy of small-target (cigarette) detection. The detection performance of the model is improved by eliminating certain erroneous detection by using the intersection over union (IoU) of the cigarette and a pedestrian. Because of the lack of international public datasets, we prepare a self-made dataset about smoking by trawling the Internet using crawler technology and combining other self-made datasets. For the dataset used in this study, our improved algorithm model based on YOLOv4 performs better than other one-stage algorithms.