Home security systems have been extensively used to protect our property and family, and these products are always equipped with some devices, including indoor and outdoor cameras, infrared motion sensors, human body temperature sensors, and smart locks. Security systems are designed to detect the presence of people or moving objects. However, they have certain limitations. Firstly, these systems are inactive when not turned on, rendering them ineffective during those times. Additionally, false alarms are common during system surveillance, posing a challenge to the system’s overall reliability. A smart surveillance camera has recently been added to the system, but it cannot distinguish between family members and intruders. A full-time facial recognition system has been proposed in this paper to address the drawbacks of the current security system. The Day Night Surveillance Neural Network (DNSNN), which is a face recognition network based on the optimized EfficientDet, is proposed. The DNSNN provides full-time recognition in this study and divides the system into day and night modes. It uses visible light images in day mode under good light conditions to perceive objects. Under poor light conditions, the camera automatically takes grayscale images with near IR. However, in these images, objects are challenging to recognize, and thus the accuracy rate is reduced. A proposed auto-coloring system is applied to colorize the grayscale images. The colorized images can have a similar hue to the visible light images and are equipped with the same vision. This method can improve system recognition capabilities under dim light. The experimental results show that our proposed approaches recognize family members and intruders under all light conditions and have an identification accuracy of more than 90%. This system can achieve full-time smart home surveillance.