Hugging Face
Models
Datasets
Spaces
Posts
Docs
Enterprise
Pricing
Log In
Sign Up
ocisd4
/
llama-2-tokenizer-dataprep
like
0
Follow
ocisd4
24
Model card
Files
Files and versions
Community
YAML Metadata Warning:
empty or missing yaml metadata in repo card (
https://huggingface.co/docs/hub/model-cards#model-card-metadata
)
關閉自動添加
<s>
,方便產生megatron-deepspeed訓練用檔案
指定pad token為
<unk>
,訓練時token數才會正確,以及finetune_t0.py才能正確pack_sample
Downloads last month
-
Downloads are not tracked for this model.
How to track
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no library tag.
Collection including
ocisd4/llama-2-tokenizer-dataprep
tokenizers
Collection
12 items
•
Updated
Jun 11, 2024