Proceedings of the Annual Conference of JSAI
Online ISSN : 2758-7347
38th (2024)
Session ID : 1B4-GS-2-01
Conference information

Automated Model Performance Evaluation via Contrastive Learning of Distilled Surrogate Model
*Makoto KAWANOKazuki KAWAMURA
Author information
CONFERENCE PROCEEDINGS FREE ACCESS

Details
Abstract

Real-world machine learning system operation suffers from performance degradation due to data distribution shift, which occurs during operation and leads to lower accuracy compared to model validation. Detecting this performance degradation enables appropriate measures such as model retraining or structural revision. However, continuous labeling of operational data is not realistic due to the high cost. Therefore, this study focuses on estimating the performance of a model on unlabeled test data. Since direct calculation of accuracy on test data is impossible without labels, previous studies have attempted to estimate test accuracy using distances or metrics correlated with it. One such study utilizes adversarial accuracy, but it requires simultaneous adversarial training with the model to be evaluated, rendering it inapplicable to pre-trained models. To address this, we propose CoLDS, a method that estimates the test performance of any model without labels by converting the model to be evaluated into a surrogate model using knowledge distillation and performing adversarial training on the surrogate model. This paper evaluates the effectiveness of CoLDS through experiments and reports the results.

Content from these authors
© 2024 The Japanese Society for Artificial Intelligence
Previous article Next article
feedback
Top