How to set batchsize of inference
I am a beginner and it seems that Transformers only supports processing one request at a time. Can we actually increase the parallel capability to process multiple requests in parallel like batchsize=n?
2024-10-17T10:00:32Z
hi@allenwang37I don’t know if this answers your question:github.com/huggingface/transformersHow to use transformers for batch inferenceopened07:41AM - 20 Aug 21 UTCclosed09:55AM - 20 Aug 21 UTCwangdong1992I use transformers to train text classification models，for a single text, it can…be inferred normally. The code is as follows

from transformers import BertTokenizer, TFAlbertForSequenceClassification
text = 'This is a sentence'
model_path ='../albert_chinese_tiny'
tokenizer = BertTokenizer.from_pretrained(model_path)
model = TFAlbertForSequenceClassification.from_pretrained('../model_tf/20210818')
encoding = tokenizer(text, truncation=True, padding=True, max_length=30, return_tensors="tf")
result = model(encoding)

When I predict more than one text at a time, an error will be reported. The code is as follows

texts = ['This is a sentence', 'This is another sentence']
encodings = []
model_path ='../albert_chinese_tiny'
tokenizer = BertTokenizer.from_pretrained(model_path)
model = TFAlbertForSequenceClassification.from_pretrained('../model_tf/20210818')
for text in texts:
    encoding = tokenizer(text, truncation=True, padding=True, max_length=30, return_tensors="tf")
    encodings.append(encoding)
result = model(np.array(encodings))

The error information is as follows:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Value for attr ‘Tindices’ of string is not in the list of allowed values: int32, int64
; NodeDef: {{node ResourceGather}}; Op<name=ResourceGather; signature=resource:resource, indices:Tindices → output:dtype; attr=batch_dims:int,default=0; attr=validate_indices:bool,default=true; attr=dtype:type; attr=Tindices:type,allowed=[DT_INT32, DT_INT64]; is_stateful=true> [Op:ResourceGather]If you have multiple GPUs:huggingface.coDistributed inferenceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.