Taste is one of the most important criteria for evaluating the quality of apples. In this study, a correlation model using hyperspectral images and quantitative taste information was built for the nondestructive detection of apple taste information. Firstly, the images of 90 sets of Aksu apples were collected by a hyperspectral image system and quantitative values of taste information (sourness and sweetness) were measured using an SA-402B electronic tongue. Secondly, to overcome the difficulties in obtaining the most representative wavelengths, a competitive adaptive reweighted sampling (CARS) algorithm was proposed to remove redundant information in the hyperspectral data. Then, 43 characteristic wavelengths corresponding to sourness and 22 characteristic wavelengths corresponding to sweetness were selected. Finally, particle swarm optimization (PSO) was used to dynamically optimize the kernel parameters and penalty factors of support vector regression (SVR). A PSO–SVR prediction model based on characteristic wavelengths was established. Upon comparing the performance of the prescreening and postscreening models, results showed that the CARS–PSO–SVR model achieved better prediction for apple taste information, for which the correlation coefficients (R2) of sourness and sweetness were 0.81 and 0.887, and the root mean square errors of the prediction set (RMSEP) were 0.03 and 0.018, respectively.