To investigate the effect of music on emotion, the physiological responses to music were analyzed with respect to heart rate variability (HRV) using an electrocardiogram (ECG) sensor. For the experiment, music was categorized by its bits per minute (BPM) and played while monitoring the HRV of the participants. The obtained data were analyzed to obtain parameters in the frequency domain analysis. The indicators of the autonomic nervous system (ANS) changed with the BPM of music and represented the emotional state of the participants. In particular, total power (TP), high- and low-frequency powers, and high- and low-frequency power ratios (HFR and LFR, respectively) changed when listening to fast (120‒140 BPM), intermediate (60‒80 BPM), and slow (less than 40 BPM) music. Fast music enhanced the autonomic sympathetic nervous system, thus increasing LFR and TP, but HFR decreased owing to the reduced activity of the parasympathetic nerve. Intermediate music inhibited the activity of the autonomic sympathetic nerve, leading to decreased LFR and increased HRF owing to the enhanced activity of the parasympathetic nerve. Slow music did not change the activity of the autonomic sympathetic nerve. These results suggest that music can be used to lessen or prevent stress and enhance work performance.