dummy pr, do not merge! replace with dummy fast-tokenizer
#1
by
michaelfeil
- opened
No description provided.
For sample deployment with a Rust compatible fast tokenizer instead of the tokenizer.model which requires python code execution. Tokenizer is not guaranteed to be compatible, just adding the fees from Llama-3-70B
michaelfeil
changed pull request title from
replace with dummy fast-tokenizer
to dummy pr, do not merge! replace with dummy fast-tokenizer