Returning score associated with prediction_value from loaded_tokenizer
Hi. I’m trying to get the %age score corresponding to prediction_value using the following python code snippetpredict_input = loaded_tokenizer.encode(email_text, truncation=True, padding=True, return_tensors="tf")
output = loaded_model(predict_input)[0]

prediction_value = tf.argmax(output, axis=1).numpy()[0]loaded_model and loaded_tokenizer have been trained earlier using distilbert.I’ve tried using TextClassificationPipeline, but unfortunately “email_text” has too many tokens (more than 512) so doesn’t work. If I truncate email_text it returns incorrect prediction_value.Thanks in advance
2024-10-10T04:32:34Z
I searched to see if it was some kind of bug or configuration issue, but it seems to be a deep-rooted problem.By the way, you can pass options to the pipeline for the model and tokenizer in addition to the pipeline options. Check the model description to see what options are available. It would be easy to solve this problem.paperswithcode.comPapers with Code - Breaking the Token Barrier: Chunking and Convolution for...No code available yet.mercity.aiHow to Classify Long Documents and Texts with BERT ModelsMost of the applications need access to process long texts and documents, BERT models even though competent, might not perform very good for long contexts. In this blog, we show how to work with long texts and contexts with BERT.github.com/huggingface/transformersprocessing texts longer than 512 tokens with token-classification pipelineopened12:58PM - 17 Jan 22 UTCclosed03:07PM - 24 Feb 22 UTColiverguhr# 🚀 Feature request

Currently, the token-classification pipeline truncates in…put texts longer than 512 tokens. 
It would be great if the pipeline could process texts of any length.

## Motivation

This issue is a result of this feature request:
huggingface/hub-docs#11

As suggested I am tagging @Narsil here. 

## Your contribution

For my [punctuation prediction model](https://huggingface.co/oliverguhr/fullstop-punctuation-multilang-large), I wrote an inference code for this task.
This code segments a longer text into 512 tokens chunks with some overlap and 
reconstructs the final output string. The code however is rather complex and tailored to 
my punctuation prediction use case. So I am not sure if this is useful here.
2024-10-11T11:39:24Z
Thank you. It haddn’t occured to me that the input could be chunked. Although it’ll make the classification slower, if it makes it more accurate, that’ll be really helpful. I’ll look into it