Force decoder to avoid repetition between generated sentences
I would like to fine-tune T5 for diverse paraphrase generation.  For each original sentence, I would like to have several different paraphrases generated, but current results contain sentences very similar to each other.Example:Original Question ::What is the expected close date of the opportunityParaphrased Questions Generated  by T5::0: What will be the expected close date of the opportunity?1: What is the expected closing date for the opportunity that you are considering?2: What is the expected close date of the opportunity?3: What is the expected close date on the opportunity?4: When would be the expected close date of the opportunity?I tried to add diversity measure in the training but was notified it wouldn’t work.Thus, I want to directly force the decoder to avoid the repetition of ngrams between generated sentences during testing.  The‘generate’ functionhas two parameters:repetition_penalty, no_repeat_ngram_size. I checkthe paperand the source code, if I understand correctly,  they just avoid repetition along the beam rather than between the sentences.  No surprise: I  tried different values of the two parameters and there seems no effect.Thus, I was wondering if there is any simple way to penalize the repetition between sentences? My thought is, during beam search, to penalize the probabilities of repetitive words on different branches at the same/ nearby step. Is there open source code available for this? If not, is there anything I need to pay attention to when I modified the 'generate()’function for this?
2020-11-02T13:12:32Z
Hey@mengyahu- this sounds like a cool use case! You are right we the currentgenerate()method it is not really possible to avoid repetitions between sentences. It’s quite a special case so I’d suggest that after this PR is merged:Big `generate()` refactoryou to make a fork of thetransformersrepo and try to tweak thebeam_scorer.update()function (or theBeamSearchScorerclass in general to add a penalty as needed).
2021-04-16T10:59:15Z
Hey@mengyahu. I’m facing a similar issue where I’m getting repeated sentences in summaries I’m looking to produce. Did you get a chance to add this penalty for repeated sentences? Happy to help work on it if not.
2021-04-16T17:07:32Z
@kmfodaI have not found a way to add that penalty yet. As I moved on to other projects soon after I posted the question, I did check if PR is merged as mentioned by@patrickvonplaten.So it would be great if you continue working on this and post solutions if you find it.
2021-04-20T09:50:59Z
Thanks@mengyahu. Actually for my use case I found thatno_repeat_ngram_sizeworked great because I was looking to avoid sentence repetitions in the same single output. I’m guessing you want to avoid repetitions across the multiple outputs produced. Let me have a think about how that might be done and if I make some progress I’ll submit a PR.