We describe the detection of cell nuclei in oral cytology using artificial intelligence (AI). We focused on the detection of cell nuclei because the ratio of cell nuclei to cytoplasm increases with increasing cell malignancy. As an initial step in the development of AI-assisted cytology, we investigated two methods for the automatic detection of cell nuclei in blue-stained cells in cytopreparation images. We evaluated the usefulness of the sliding window method (SWM) and the mask region-based convolutional neural network (Mask-RCNN) method in identifying cell nuclei in oral cytopreparation images. Thirty cases of liquid-based oral cytology were analyzed. First, we performed the SWM by dividing each image into 96 × 96 pixels. Overall, 591 images with or without blue-stained cell nuclei were prepared as the training data and 197 as the test data (from among 1576 images in total). Next, we performed the Mask-RCNN method by preparing 130 images of Class II and III lesions and creating mask images showing cell regions based on these images. By the SWM method, the highest detection rate for blue-stained cells in the evaluation group was found to be 0.9314. For Mask-RCNN, 37 cell nuclei were identified, and 1 cell nucleus was identified as a non-nucleus after 40 epochs (error rate: 0.027). Mask-RCNN was shown to be more accurate than SWM in identifying the cell nuclei. If the blue-stained cell nuclei can be correctly identified automatically, the entire cell morphology can be grasped faster, and the diagnostic performance of cytology can be improved.