Character-level tokenizer
Hi,I would like to use a character-level tokenizer to implement a use-case similar tominGPT play_charthat could be used in HuggingFace hub.My question is: is there an existing HFchar-level tokenizerthat can be used together with a HF autoregressive model (a.k.a. GPT-like model)?Thanks!
2021-12-02T10:22:31Z
Hi,We do have character-level tokenizers in the library, but those are not for decoder-only models.Current character-based tokenizers include:CANINE(encoder-only)ByT5(encoder-decoder)
2022-03-19T11:37:41Z
In order to have a HugginFace equivalent to minGPT, I ended-up using:gpt2(decoder-only)I enforced the gpt2 tokenizer to use single char tokens with some code like:chars = [ ‘a’, ‘b’, ‘c’, …]new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=len(chars), initial_alphabet=chars)The gpt2 tokenizer still contains extra tokens beyond those I wanted in theinitial_alphabet, but the gpt2 model performs reasonably well at char-level.
2022-10-18T14:21:46Z
In case anyone looking for a character tokenizer for hugging face transformers, you can check myrepo.
2024-04-29T10:11:46Z
Hi again everyone,I am working with bank transactions and some examples of the merchant names are“Tesco Super”, “Tesco Superma”, “Paypal * Tescosupermar”, “Zilch * tescosupermarket - 343” which have very similar meaning.I am currently using a sentence embedding based on average of words (universal sentence embedding) to classify these transactions into supermarkets, hotels, restaurants… it works ok, but it could be better, so I want to create a sentence embedding that is based in the average of character embeddings.Is there a language model in LLM that was trained on character tokens and from which I canextract a embedding combining the meaning of characters? I know there are some token tokenizers, but which models are compatible with a token tokenizer…I know there are is Char BERT and Character BERT, but those still use word tokenizer, just that the meaning of each word is based on characters… and for me the space " " is just another character, does not always mean a new word… and I want my embedding to consider the space is just another character with a meaning, also * is a character with an important meaning.Thanks a lot!
2024-05-07T22:16:35Z
Randomly stumbled on this as I’m looking for a character-level encoding model myself. However, for your case, I’d suggest trying a model from the MTEB leaderboard that does well with classification and is smaller (and hopefully faster). See if you actually get embeddings that are dissimilar for your examples. If you do, you can fine-tune your model. Feed it triples of (query, positive_example, random_negative_example). The query and positive examples would be like the ones you have. Do some googling on triplet loss function with pytorch for details. Good luck!
2024-05-08T02:01:42Z
Have you seen CANINE and Byt5?