A general surveillance camera with a near-infrared illuminator provides a night vision function, but it is difficult to take a picture under foggy or smoky conditions, in a heavy rainfall environment, or under direct exposure to the sun because of poor object temperature reflection. A thermal infrared (TIR) camera can have better imaging to reflect objects in bad environments, and they have many applications in safe driving and military and scientific fields for all-weather surveillance. However, TIR images are mainly presented in grayscale, which causes the applications of TIR images to be limited and used only for rough object recognition. In previous studies, auto-colorization by predicting luminance and chrominance from grayscale images at the same time was typically performed, but the results were always blurry and abnormally colorized images. This study proposes the Depthwise Separated Colorization Generative Adversarial Network (DSCGAN) to colorize TIR images and overcome these drawbacks. Initially, the preprocessing light channel convolutional autoencoder (PLCAE) is proposed to generate the predicted L channel of the International Commission on Illumination LAB color space (CIELAB) that is used to restore some lost luminance information. Then, this predicted L channel is used as input to the proposed Colorization Generative Adversarial Network (CGAN) to create the AB channel. Finally, the data from L, A, and B channels are converted to the RGB visible light image. The experimental results indicate that our proposed PLCAE can efficiently enhance luminance details and achieve an accuracy rate of 0.9773. The proposed CGAN advances colorization accuracy and improves the peak signal-to-noise ratio (PSNR) to more than 26 dB. The colorized TIR images have almost the same color as the visible light images and clearly maintain object textures and details.