Custom Distilbert does not use CUDA for predition
Hi,I am using the model classifier = pipeline(“text-classification”,model=‘bhadresh-savani/distilbert-base-uncased-emotion’, return_all_scores=True) for emotion classification of my datatset of 1.2m client feedbacks. I am not training the model, I am just doing a prediction. I noticed that the model uses CPU and does not use CUDA (I have RTX5000) and the prediction takes ages to compute.Can you explain why it is the case? Is there a way to use CUDA for this model predictions?Thank youKrzysztof
2021-07-27T15:42:46Z
I found a solution, add parameter device=0. But I have to classify in small batches as the GPU RAM is a serious limit, even my GPU has 16 GB
2021-07-27T18:17:29Z
very interesting, I had the same issue. How do you change the batch size with a pipeline?
2021-07-27T18:44:40Z
This is probably primitive but it works. I did it in the loop. It took six hours to classify 1.2 mln reviews, some quite long, on GPU with 16GB memory.classifier = pipeline(“zero-shot-classification”,model = “typeform/distilbert-base-uncased-mnli”,device = 0)emo_zs_labels = list()ind = 0lenss = len(sentences)while ind < lenss:bb = indee = bb+seq_len if bb+seq_len < lenss else lensspred = classifier(sentences[bb:ee],candidate_labels=[“sadness”, “joy”, “love”, “anger”, “fear”, “surprise”],)temp_labels = [x[‘labels’][0] for x in pred]emo_zs_labels.append(temp_labels)ind = eeprint(ind)I experimented with seq_len, caused CUDA out of memory for 1000 and 500, so I finally set it 100.
2021-07-27T18:48:35Z
got it, you created the batches yourself. I was curious to know whetherpipelinewould handle this but apparently it does not.Just a question: what is the advantage of using apipelinevs. using a tokenizer, loading the model, fine-tuning it and finally classifying? have you tried?
2021-07-27T19:21:12Z
Yes. Pipeline takes just few lines of code to train or predict. But with tokenizer etc. I feel I have more control over the hyperparameters.
2021-07-27T19:40:20Z
interesting, i thought pipeline had little hyperparameters… how do you access/change them in your example?
2021-07-27T20:01:39Z
That is what I am saying. If you do tokeknizer, model and trainer separately you have better control of parameters. And if you do it in native Pytorch, even more. But then instead of 5 lines of code you have few 100+