Simultaneous localization and mapping (SLAM) is the key technology for achieving autonomous navigation and stable walking for robots. For addressing a dynamic and special environment indoors and outdoors, there are still some limitations in using a single sensor to estimate and locate a robot’s position and orientation. To further improve the accuracy of SLAM positioning in real time, in this study, we combine the advantages of the RGB-depth map (RGB-D) and light detection and ranging (LiDAR) and propose a model of a two-stage deep fusion framework named convolutional neural network (CNN)–LiDAR vision inertial measurement unit (CNN–LVI) for real-time pose estimation by a geometric method. Unlike existing methods that use either a two-stage framework or multistage pipelines, the proposed framework fuses image and raw 3D point cloud data after multisensor joint calibration, and then uses 3D point clouds as spatial anchors to predict the pose between two sequence frames. By using a CNN algorithm to identify and extract a 3D bounding box, the target object projection of an RGB image is tracked to obtain the target minimum bounding rectangle (MBR). Finally, the rotation angle and translation distance are calculated by a geometric method using the centroid of the target MBR, so as to combine an inertial measurement unit to perform joint optimization, achieve the pose estimation of a robot, and further improve the model’s location accuracy. Experiments show that the proposed model achieves significant performance improvement compared with many other methods in the car class and achieves the best trade-off between state-of-the-art performance and accuracy on the benchmark with the KITTI dataset.