In this study, we present a new dataset for managing asphalt pavement surfaces, especially for crack detection. To achieve this goal, we installed a multi-sensor system on the mobile mapping system (MMS) and obtained real-time RGB and IR images, and then the geometric constraint method was applied to find corresponding feature points to spatially register these images. Finally, three environmental data consisting of temperature, humidity, and wind speed are added to the images according to time and location. These data are integrated according to the proposed database model. The proposed system was tested and the databases were constructed for our experiment site, namely, the Capital Region First Ring Expressway in Goyang-si, Gyeonggi Province, South Korea. A total of 800 multi-sensorial images were collected from the expressway. The developed database can be used to train deep learning networks so that it will support detecting road signs or damage on asphalt surfaces.