Prototypical Fine-tuning - Towards Robust Performance Under Varying Data Sizes

Preprint.
Authors
Yiqiao Jin1, Xiting Wang2, Yaru Hao2, Yizhou Sun1, Xing Xie2
1University of California, Los Angeles, 2Microsoft Research Asia
Abstract
We move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the data complexity and the model’s inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the global optimum. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.

License
Copyright 2022-present Yiqiao Jin.
Released under the MIT license.