Prototypical Fine-tuning - Towards Robust Performance Under Varying Data Sizes

Preprint.

Authors

Yiqiao Jin¹, Xiting Wang², Yaru Hao², Yizhou Sun¹, Xing Xie²

¹University of California, Los Angeles, ²Microsoft Research Asia

Abstract

We move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the data complexity and the model’s inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the global optimum. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.

License

Copyright 2022-present Yiqiao Jin.

Released under the MIT license.