Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing
  Models
  Semantic parsing is a key NLP task that maps natural language to structured
meaning representations. As in many other NLP tasks, SOTA performance in
semantic parsing is now attained by fine-tuning a large pretrained language
model (PLM). While effective, this approach is inefficient in the presence of
multiple downstream tasks, as a new set of values for all parameters of the PLM
needs to be stored for each task separately. Recent work has explored methods
for adapting PLMs to downstream tasks while keeping most (or all) of their
parameters frozen. We examine two such promising techniques, prefix tuning and
bias-term tuning, specifically on semantic parsing. We compare them against
each other on two different semantic parsing datasets, and we also compare them
against full and partial fine-tuning, both in few-shot and conventional data
settings. While prefix tuning is shown to do poorly for semantic parsing tasks
off the shelf, we modify it by adding special token embeddings, which results
in very strong performance without compromising parameter savings.
