Advice on model type
Hi all,I would like to know which initial model I could use in order to train a model capable of generating lists of numbers starting from a prompt in natural language.Examples:“Create a list of 3 prime numbers” → 1, 3, 5“Create a list of 5 random numbers” → 5, 7, 19, 22, 24“Create a list of 2 negative numbers” → -3, -5etc.From what I’ve read, the recommended models would be those based on “text-generation”, but before proceeding I would like to have further confirmation and maybe understand if eg. a GPT2 model might be fine or if there are others, even lighter.Thank you so much
2023-03-01T17:23:55Z
Hey@00ncBased on the examples you gave, where there is a clear separation between the input and the output, you can also consider “encoder-decoder” language models, like T5. These models have atext2text-generationtag.One more note: with tasks where you want the model to follow an instruction, consider starting from an instruction-tuned model likeFlan-T5orBLOOMZThese models have been fine-tuned to follow instructions and tend to behave better on this kind of tasks.
2023-03-01T20:40:03Z
Thank you very much!
2023-03-02T12:47:37Z
Hi Joao,I was able to load and query the “Flan-T5” model both locally and using the API.I was also able to write a code that queries locally the model from a CSV file built like that:input_text;target_text“Write a list of numbers”;“1, 2, 3”“Write 5 numbers”;“3, 6, 8, 2, 1”etc.But I wasn’t able to find any code sample to train the model, is there any?I am using Python on win10.Thank you.
2023-03-02T13:05:21Z
Hey@00nc– have a look at oursummarization transformers example, which can train a T5 model.
2023-03-02T16:17:24Z
Thank you again, I was able to run the script and to test it with the sample codepython examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generateNow my question is, how should I change this instruction to read the CSV file and use the FLAN-T5-SMALL model?Of course I will change the model name in flan-t5-small but how to load the CSV and also if a source prefix is needed.Thank you very much for your help!
2023-03-03T10:09:03Z
Ok, I was able to train the model using the following code:python run_summarization.py \
    --model_name_or_path flan-t5-small \
    --do_train \
    --do_eval \
    --train_file C:/mypath/train.csv \
    --validation_file C:/mypath/valid.csv \
    --source_prefix "summarize: " \
    --output_dir c:/myoutputpath \
    --overwrite_output_dir \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generateI have created a CSV training file of 15 lines like this:"write 3 numbers"; "6,12,18"
"write 3 numbers"; "4,8,12"
"write 3 numbers"; "17,34,51"
etc.the validation file is actually the same with different 3-number lists.The training finishes correctly but when I try to query the created model, this is what happens:Input: write 3 numbers
Expected output:  "7,14,21"
Generated output: <pad>a slam dunk a slam dunk a slam dunk a slam dunk a slam dunk a slWhy? Should I just train it on a lot more lists or am I doing anything wrong?Thank you!