Filled tofu is considered hygienic and has a low environmental load during production, and its distribution has been rapidly increasing in recent years. The identification of product quality is a time- consuming and labor-intensive task, and there is a need for a technology that can automatically and accurately identify defective products on the production line. The purpose of this study was to classify the coagulation quality of filled tofu using deep learning with color images as input. An imaging system equipped with a polarizing filter to remove specular reflection was constructed, and color images of the film surface of filled tofu were captured on the actual production line. The images were visually labeled into three categories: A, B, and C classes, and classification models were constructed using eight different pre-trained networks. The highest accuracy on the test data was 95.84% with EfficientNet-b4. Visualization of the basis of judgment showed that the model correctly captured the defect features of the tofu surface, with a large weight on the location of bubbles on the tofu surface. These results suggest that color images of the tofu film surface and deep learning can be used to accurately identify the coagulation quality of tofu.