In snowy and cold regions, potholes occur frequently during the snowmelt season, requiring regular road patrols and systematic repairs based on road surface conditions. In recent years, in-vehicle cameras and AI have been introduced to improve the efficiency of road patrols . However, it is difficult to find a method to evaluate complex and significant road surface damage during the snowmelt season. In this study, we developed a method to detect potholes and further evaluate their road surface damage using image recognition AI (VGG16) and saliency map (Score-CAM). First, our method can detect potholes with an accuracy of more than 85% using AI trained on road surface images taken during the winter season. Next, we showed that the saliency map can evaluate road surface damage by extracting the shape of pothole areas in detail. Finally, we conducted periodic road patrols and planned network-level repairs using the proposed method. As a result, our findings can contribute to the efficiency and improvement of winter pavement maintenance and management.