Toward solving the problem of the lack of useful features caused by inconspicuous vehicle features and the interference of similar features around a vehicle in the process of remote sensing image vehicle detection, we propose an algorithm based on the assistance of background features. This algorithm is based on the YOLOv4 model and includes a weight redistribution module based on the feature correlation degree. The model introduces background features around a vehicle so that the model learns to determine the environment in which the vehicle target is located during training. This can effectively reduce the occurrence of missed detection. Moreover, the algorithm increases the sensitivity of the backbone to vehicle features through the weight redistribution of the features inside the anchor box, thus making full use of the feature correlation between the vehicle target and the surrounding background. As a result, the vehicle target is effectively distinguished from the interference target. The experimental results show that the precision rate and success rate of this algorithm on the DLR-3K dataset reach 75.4 and 68.5%, respectively. The model detection rate reached 12.7 frames/s. The proposed algorithm has high performance in executing vehicle detection tasks in the presence of interference due to similar targets.