Simple example of Transformer from scratch?
Is there a full example of how to train an extremely small/simple transformer model (e.g.GPTNeowith only a hundred parameters) entirely from scratch? I’m trying to do this just for learning purposes but I keep getting CUDA errors. It can’t be that I’m filling the memory as the model and datasets are tiny, but it’s probably some kind of indexing error that might be due to my use of one or more parameters, and I’d like to start from a working example to figure it out. Any suggestions?
2023-12-25T20:49:14Z
The simplest example is the NanoGPT project from Karpathy:GitHub - karpathy/nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs..See also the training scripts of Hugging Face for causal language modeling, which are a bit more extensive/feature complete:transformers/examples/pytorch/language-modeling at main · huggingface/transformers · GitHub.
2023-12-25T21:18:54Z
Ah, sorry, I did not explain myself clearly since we’re in the Huggingface forums and gave it for granted - I meant simplest using thetransformerslibrary. I’m getting errors when launching the training with theTrainerclass, and am not sure I’m organizing everything right (datasets, tokenizer etc.). I guess your second link should be good for this, I’m interested in causal modeling so I guessrun_clm.pyit is for me.