In this paper, we present an end-to-end vision-based autonomous navigation system for robots using deep learning. The system integrates semantic segmentation with visual simultaneous localization and mapping (SLAM) to implement a semantic SLAM approach. The generated map not only contains geometric information but also recognizes and classifies objects in the environment, enhancing the robot’s perception capabilities. For semantic SLAM, a semantic segmentation neural network is first designed and trained using the SUNRGB-D dataset. Real-time data processing and node registration are carried out using the robot operating system. Depth images and camera intrinsics are used to generate point clouds, and semantic segmentation images are fused into the generated point clouds to create a 3D semantic map using Octomap, enriched with semantic information. For autonomous navigation, a test environment is set up in the Gazebo simulation, and expert data is collected. The autonomous navigation neural network is trained based on color images, depth images, and the robot’s position and orientation data, enabling the network to output linear and angular velocities based on visual data alone. Without relying on additional sensors, the robot is capable of path planning, obstacle avoidance, and autonomous navigation purely through visual input.