Establishing a reliable vision system is essential to developing table tennis robots as it provides vital information for accurately controlling the robot’s movement to hit a ping-pong ball. In this study, we present a vision system and its main algorithm, which can detect and track a ball, classify its spin, and estimate the contact point and hitting time on a predefined hitting region. To achieve this, we used two well-calibrated cameras to construct a binocular vision system for detecting and positioning the ball in 3D space. Then, two deep-learning models were separately deployed to classify the ball spin type and predict the ball’s trajectory. The contact point on the hitting plane and the hitting time were finally estimated. Experimental results showed that the proposed vision system performed well, with acceptable estimation errors for the contact point and hitting time, which are feasible for controlling our designed robot to strike a table tennis ball.