The idea of Society 5.0 initiative has been proposed to solve various social problems by connecting virtual cyberspace and real physical space through information technology. When applying the idea to improve the work-life balance of physicians in the medical field, we must consider the increased stress owing to their long continuous working hours. Estimating the stress of physicians in their daily lives by the questionnaires is insufficient, because of the difficulty of accurate their activity recalling. By using bio-metric information such as heart rate, physical activity, and sleeping information, it was expected that the daily stress state of physicians with high accuracy. In this paper, we propose a method for estimating physician stress by analyzing bio-metric information acquired by wearing a wearable sensor device. The proposed method estimates the state of stress during daily activities by acquiring data on heart rate variability (HRV) during wakefulness as well as sleep depth during rapid eye movement (REM) and non-REM sleep. Up to seven physicians wore the wearable sensor device for the maximum of eight weeks and the sleep depth and low-/high-frequency (LF/HF) components of HRV were obtained. Our observation showed that physicians' root mean square of successive differences (rMSSDs) were constantly high in their healthy state. Therefore, the decreasing of this index can be used as an indicator of fatigue and stress. In addition, by combining LF/HF components to the rMSSDs, we may estimate the stress state of physicians and find personal stressors.