In this study, we extend the development of the laboratory’s smart wheelchair system by proposing a semantically aware solution tailored for elderly individuals with visual impairments and physical disabilities, aiming to alleviate the shortage of caregiving manpower in an aging society. The system integrates three core modules: semantic simultaneous localization and mapping (SLAM), path planning and navigation, and visual relationship detection (VRD). With an RGB-D camera sensor, the system performs oriented fast and rotated brief (ORB)-SLAM for localization and mapping, then applies semantic segmentation to label the point cloud and construct a 3D semantic map. For navigation, the Timed Elastic Band (TEB) algorithm is adopted to dynamically avoid obstacles and generate feasible paths. In terms of scene understanding, the system incorporates the Relation Transformer (RelTR) model to identify semantic relationships between objects and provides real-time feedback to the user through voice prompts, enhancing spatial awareness and decision-making. The complete system was integrated into a physical wheelchair for testing. Experimental results demonstrate its capability in stable mapping, effective obstacle avoidance, and semantic comprehension, contributing to greater mobility for visually impaired individuals while reducing caregiver workload.