In this study, the characteristics of a plastic optical fiber (POF) sensor based on an in-fiber rectangular hole for detecting mercury in water were investigated. The proposed sensor comprises a POF that includes an in-fiber rectangular hole partially filled with a synthesized rhodamine derivative acting as a sensing probe. The rhodamine derivative with maximum absorbance spectrum variations occurring at approximately 530 nm was fabricated from rhodamine 6G of 95% purity, and the in-fiber rectangular hole with dimensions of 3 × 0.65 mm2 was fabricated using an inexpensive drilling machine. The thickness of the rhodamine derivative attached to the rectangular hole was about 0.5 mm. The absorbance of the mercury-detecting POF sensor was proportional to the mercury ion concentration. The relationship between the absorbance and the mercury ion concentration was approximately linear in the mercury ion concentration range of 10 to 200 ppm. The sensitivity was about 0.00031 ppm−1 in the linear response region. The experimental results show that the proposed POF sensor can be effectively used to detect mercury ions in water.