Brain tumors vary in size and location in magnetic resonance imaging (MRI), and rising patient volumes at imaging centers delay radiologist feedback owing to increased diagnostic workload. To address this issue, we established an unsupervised learning model with a convolutional autoencoder for the extraction of features in images and explored the application of masking technology within this approach. The proposed method classifies tumors as gliomas, meningiomas, or pituitary tumors by analyzing brain magnetic resonance sensor images. First, a shallow autoencoder network was used for image reconstruction. It has excellent feature dimensionality reduction, robustness, and noise suppression capabilities, and thus reduces the likelihood of overfitting. Subsequently, the features extracted from the encoder were fed into a single-layer dense neural network, and finally, classification was tested on a softmax layer. The experimental results demonstrated that the incorporation of masking technology enabled the essential feature information to be precisely captured and resulted in highly satisfactory generalizability for unlabeled image test datasets. The developed model was trained and evaluated on the contrast-enhanced (CE)-MRI and Kaggle datasets, and achieved accuracies of 95.59 and 97.01%, respectively.