With the development of computer vision and remote sensor devices, object detection in aerial images has drawn considerable attention because of its ability to provide a wide field of view and a large amount of information. Despite this, object detection in aerial images is a challenging task owing to densely packed objects, oriented diversity, and complex background. In this study, we optimized three aspects of the YOLOv5 algorithm to detect arbitrary oriented objects in remote sensing images, including head structure, features from the backbone, and angle prediction. To improve the head structure, we decoupled it into four submodules, which are used for object localization, foreground, category, and oriented angle classification. To increase the accuracy of the features from the backbone, we designed a block dimensional attention module, which is developed by splitting the image into smaller patches based on a dimensional attention module. Compared with the original YOLOv5 algorithm, our approach has a better performance for oriented object detection—the mAP on DOTA-v1.5 is increased by 1.25%. It was tested to be effective on DOTA-v1.0, HRSC2016, and DIOR-R datasets as well.