How can I pad the vocab to a set multiple?
Probably an easy one, but not having any luck in finding the solution, so thought I’d make a post.To use tensor cores effectively with mixed precision training aNVIDIA guiderecommends to “pad vocabulary to be a multiple of 8”.I’ve searched the tokenizersdocumentationfor answers but haven’t found much luck. The closest I could find is thepp_tokenizer.vocab_sizemethod that returns the current vocab size, but I can’t assign it a new value.Any idea how I can do this?
2021-11-26T10:33:15Z
Hi,You can provide the argumentpad_to_multiple_ofto a tokenizer in Transformers (this is supported both for fast and slow tokenizers):pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
>= 7.5 (Volta).
2021-11-27T02:50:56Z
Thanks for the reply. This method is good and useful for a sequence input, but I was more looking for something that could resize the matrices that depend on the vocab_size of a transformer.e.g. the embedding matrix of a transformer usually takes on dimensions something like (vocab_size, 1024), where the vocab_size might be something like 52153. A matrix of this size isn’t efficient to pass onto the tensor cores, so I was looking for a way to pad it so it was a multiple of 8 (e.g. to 52160).
2021-11-27T11:28:15Z
You can resize the embedding matrix of a Transformer model using theresize_token_embeddingsmethod (seedocs).
2021-11-28T08:39:19Z
thanks! that looks like what I’m looking for