{'fraction_non_alphanumeric': 0.027142857142857142, 'fraction_numerical': 0.0, 'mean_word_length': 5.095652173913043, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploring Few-Shot Fine-Tuning Strategies for Models of Visually Grounded Speech

In this paper, we study models of visually-grounded speech (VGS) in a few-shot setting. Beginning with a model that was pre-trained to associate natural images with speech waveforms describing the images, we probe the model’s ability to learn to recognize novel words and their visual referents from a limited number of additional examples. We define new splits for the SpokenCOCO dataset to facilitate few-shot word and object ac-quisition, explore various few-shot fine-tuning strategies in an effort to mitigate the catastrophic forgetting phenomenon, and identify several techniques that work well in this respect.