In this study, we develop a method that estimates the type of a sidewalk surface on which a user walks using three-axis acceleration sensor data measured by a user’s smartphone’s accelerometer. If the shape of the sidewalk surface can be estimated automatically, various sidewalk information, such as walkable sidewalks and sidewalks where pedestrians cannot easily walk, can be collected by simply having many users carry their smartphones. We, therefore, propose a method estimates the sidewalk surface by applying a convolutional neural network (CNN) based on the VGG16 architecture to sensor data. In addition, we combined VGG16 with hand-crafted features (HCFs) validated in preliminary experiments. During training, the model was pre-trained with the human activity sensing consortium (HASC) dataset, a large benchmark dataset of human activity recognition, as a source domain, and we applied fine-tuning (FT) for the sidewalk surfaces as a target domain. We conducted experiments on seven subjects and evaluated the accuracy of our proposed method using leave-one-subject-out cross-validation (LOSO-CV). The experimental results showed that our proposed method achieved the highest accuracy among all the compared methods. Specifically, our proposed method improved the accuracies of some subjects by more than 20% compared with the baseline method.