The development of renewable energy has resulted in a gradual increase in the application of solar energy. However, hotspots have been identified as a significant failure of solar panels after prolonged usage. This phenomenon has resulted in declining power generation efficiency and poses a potential risk of fire hazards or environmental contamination. Consequently, the maintenance and inspection of solar panels have become imperative. In addressing this challenge, the utilization of drones, in addition to conventional inspection methods, offers a novel solution for large-scale power plants. However, this approach may not be as economically viable for small-scale or residential solar installations. In this study, we propose a novel hotspot inspection network with a locating matrix method for efficiently detecting hotspots on solar panels in small-scale plants. The proposed method utilizes an embedded device combined with an infrared camera and a lightweight You Only Look Once (YOLO) v5s model as the basis for the defect detection network implemented. The defect-locating matrix method utilizes image processing techniques, such as hue, saturation, value (HSV) color space transformation and morphology, to convert the panel configuration of a field into a locating matrix. The center of gravity of the detected hotspot area is subsequently mapped to the locating matrix. The location of the defective panel is then labeled. The proposed approach enables real-time monitoring and analysis of solar panel status. The experimental findings demonstrate that the proposed defect detection network model reaches a mAP of 85.1% and an F1 score of 84.6%, and performs better than other YOLO models. A graphical user interface was designed to facilitate a more intuitive display of inspection results and enable accuracy monitoring.