To improve prediction accuracy in pattern recognition, many approaches using multiple classifiers are being presented nowadays. On the other hand, pattern recognition to time-series data such as video sequences are still challenging due to the real-time requirement. In this paper, we present a novel method for pattern recognition of video sequences using prediction probability calculated by a pattern classifier. Generally, in applying pattern classifier to video sequences, predicted classes are often partially fragmented. From the idea that prediction probabilities of the video sequences which have same recognition pattern would be similar to each other, the proposed method corrects the fragmented classes to correct one using the similarity of prediction probabilities. Evaluation experiments have shown that the proposed method works well to the system which estimates handlings for flexible cystoscope.