Prototypical Fine-Tuning: Towards Robust Performance under Varying Data Sizes

Jan 1, 2023·
Yiqiao Jin
Yiqiao Jin
,
Xiting Wang
,
Yaru Hao
,
Yizhou Sun
,
Xing Xie
· 1 min read
Abstract
In this paper, we move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the number of data points and the model’s inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the optimal solution. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.
Type
Publication
AAAI Conference on Artificial Intelligence 2023

Abstract

In this paper, we move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the number of data points and the model’s inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the optimal solution. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.

Keywords

Fine-tuning, Few-shot Learning, Language Models, Prototypical Learning

Fine-Tuning Few-Shot Learning Language Models Prototypical Learning
Yiqiao Jin
Authors
Yiqiao Jin
Ph.D. Candidate in Computer Science
My research sits at the intersection of multimodal foundation models, intelligent agents, and spatial/physical intelligence. I build general-purpose models that perceive, reason, learn, and act in interactive virtual and physical environments.