Add to ORKGLanguage Models (LMs) pre-trained with self-supervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-training corpus has been assembled, all data samples in the corpus are treated with equal importance during LM pre-training. However, due to varying levels of relevance and quality of data, equal importance to all the data samples may not be the optimal choice. While data reweighting has been explored in the context of task-specific supervised learning and LM fine-tuning, model-driven reweighting for pre-training data has not been explored. We fill this important gap and propose PRESENCE, a method for jointly reweighting samples by leveraging self-influence (SI) scores...
Nowadays, owing to the superior capacity of the large pre-trained language models (PLM), the PLM-bas...
Large language models (LLMs) have achieved remarkable advancements in the field of natural language ...
The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural La...
Thesis (Ph.D.)--University of Washington, 2022A robust language processing machine should be able to...
The core of self-supervised learning for pre-training language models includes pre-training task des...
While pre-trained language model (PLM) fine-tuning has achieved strong performance in many NLP tasks...
The reusability of state-of-the-art Pre-trained Language Models (PLMs) is often limited by their gen...
Semi-supervised learning methods such as self-training are able to leverage unlabeled data, which is...
To mitigate the problem of having to traverse over the full vocabulary in the softmax normalization ...
Self-supervised representation learning (SSRL) has improved the performance on downstream phoneme re...
Adopting a two-stage paradigm of pretraining followed by fine-tuning, Pretrained Language Models (PL...
We present a new method LiST is short for Lite Prompted Self-Training for parameter-efficient fine-t...
In recent years, the development of accurate deep keyword spotting (KWS) models has resulted in KWS ...
The ability to identify influential training examples enables us to debug training data and explain ...
Large language models (LLMs) have recently shown great advances in a variety of tasks, including nat...
Nowadays, owing to the superior capacity of the large pre-trained language models (PLM), the PLM-bas...
Large language models (LLMs) have achieved remarkable advancements in the field of natural language ...
The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural La...
Thesis (Ph.D.)--University of Washington, 2022A robust language processing machine should be able to...
The core of self-supervised learning for pre-training language models includes pre-training task des...
While pre-trained language model (PLM) fine-tuning has achieved strong performance in many NLP tasks...
The reusability of state-of-the-art Pre-trained Language Models (PLMs) is often limited by their gen...
Semi-supervised learning methods such as self-training are able to leverage unlabeled data, which is...
To mitigate the problem of having to traverse over the full vocabulary in the softmax normalization ...
Self-supervised representation learning (SSRL) has improved the performance on downstream phoneme re...
Adopting a two-stage paradigm of pretraining followed by fine-tuning, Pretrained Language Models (PL...
We present a new method LiST is short for Lite Prompted Self-Training for parameter-efficient fine-t...
In recent years, the development of accurate deep keyword spotting (KWS) models has resulted in KWS ...
The ability to identify influential training examples enables us to debug training data and explain ...
Large language models (LLMs) have recently shown great advances in a variety of tasks, including nat...
Nowadays, owing to the superior capacity of the large pre-trained language models (PLM), the PLM-bas...
Large language models (LLMs) have achieved remarkable advancements in the field of natural language ...
The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural La...