Heart rate (HR) measurement by a wrist-worn device suffers from noises owing to body movement. Even though many researchers have proposed sophisticated methods for the compensation of noises in measurement, such noises corrupt the sensor data itself, leading to difficulty in compensation. In this paper, we design a method for estimating the reliability of HR measurement. Our design principle is based on the fact that the change in HR is correlated with the magnitude of body movement and the current HR. To model the correlation, we construct a modified Kalman filter that estimates the displacement of the HR from the statistical data of the HR measured by a precise heart rate sensor and the variance of acceleration measured by a wrist-worn device. Then, we define the reliability of HR measurement as the absolute error between the output of a modified Kalman filter and the HR measured by the wrist-worn device. For evaluation, we compare our method with conventional outlier removal and smoothing after compensation using one of the state-of-the-art methods based on deep learning. Our method successfully removes 18.9% of the measurements with low reliability while achieving a mean absolute error of 6.25 bpm, a superior value to the conventional methods, for a single subject. For multiple subjects, our method decreases the mean absolute error by 13.1% on average.