In this paper, we propose a new method to improve the autonomous navigation of mobile robots. The new method combines a multistep update method with a double deep Q-network (MS-DDQN) to realize reinforcement learning (RL) to enhance the navigation ability of mobile robots. The proposed MS-DDQN gives two types of rewards for taking actions: terminal and non-terminal rewards. These rewards are subdivided into several different ones including rewards for arrival and collision (terminal rewards) and rewards for distance, orientation, and danger (non-terminal rewards). With the subdivided reward system, the new method trains mobile robots more effectively to increase their autonomous navigation ability. In the experimental process of this study, a laser range finder was used as the sensor for the mobile robot to perceive the distance information of the obstacle. Experiment results validated the new method’s enhanced ability, showing higher success rates (97% on average) than those of other methods such as the double deep Q-network (DDQN), prioritized deep Q-network (DQN), and prioritized DDQN. The higher success rates originated from the sophisticated reward system as the total reward of the proposed method was 7–94% higher than those of the other methods in simulations in five different environments. The learning speed was also improved, reducing the learning time, as the new method completed the learning in fewer episodes. The results of the new model suggest that MS-DDQN enables mobile robots to have higher learning efficiency and generalization ability than conventional deep reinforcement (DRL)-based methods and allows them to navigate autonomously even in unknown complex environments.