The application of hyperspectral remote sensing technology to soil environment monitoring is a cost-effective and time-saving methodology. Owing to the complex multidimensional and nonlinear relationships of hyperspectral data, traditional machine learning models are limited in their ability to deal with such complex multidimensional and nonlinear relationships. Deep learning models have been proven to effectively handle this complex multidimensional and nonlinear relational data. We took 183 soil samples from Mojiang Hani Autonomous County, Pu’er City, Yunnan Province and selected them as the research subjects, and the concentration of arsenic in the soil was predicted on the basis of the results of visible and near-infrared (Vis–NIR) spectroscopy, and proposed a lightweight convolutional neural network (CNN) model, LMetal-ResNet, aiming to predict soil arsenic concentration quickly and accurately. In addition, we also constructed two traditional machine learning models, partial least squares regression and support vector regression, and a CNN model, GoogleNet7, to predict arsenic concentration. In all the models, the data obtained by Vis–NIR spectroscopy with Savitzky–Golay convolution smoothing, min-max normalization preprocessing, and Pearson correlation coefficient feature band selection were used as input, and the output was soil arsenic concentration. Finally, the modeling accuracies of these four models were compared and analyzed. The experimental results showed that the modeling accuracies of the CNN models were higher than those of the traditional machine learning models, and LMetal-ResNet achieved the highest prediction accuracy using 68.59% of the parameters of GoogleNet7. LMetal-ResNet in the validation dataset had a root mean square error of 106.8862 mg/kg, a coefficient of determination of 0.8744, and a relative analytical error of 2.8221. In addition, we also analyzed the top 20 characteristic bands that contribute to LMetal-ResNet in predicting arsenic concentration using soil Vis–NIR spectroscopy data. This study provides scientific theoretical guidance and technical support for the prediction of concentration in soil using deep learning CNNs combined with hyperspectral remote sensing technology.