We have developed a technology to inspect rail surface defects and corrosion on the rail base using image data captured by a line sensor camera and a deep learning model. The inspection model for rail surface defects can identify four types of defects shelling, head check, gauge corner cracks, and corrugation from images taken from the top of the rail. The inspection model for rail base corrosion can determine the presence of corrosion on the rail base from images taken at an angle of the rail's side. Both inspection models use a dataset created with soft labels checked by multiple people for the same image, allowing the model's predictions to reflect the variability in human judgment. This approach has enabled us to establish a high correlation between the model's prediction confidence and human judgment confidence.