Using custom models (not necessarily transformer based) with generate() and sampling
I’m comparing a number of Seq2Seq models for a particular (non NLP) task. I’ve got a version of this working in pure PyTorch, but I’m interested in porting the research to HuggingFace to take advantage of the existing sampling and beam search capabilities. Most of my existing models are autoregressive LMs, but I also have an autoregressive LSTM baseline that would need to be ported to use the same sampling infrastructure.Is this practical, or am I going to run into problems trying to use a non transformer model in Transformers?I assume that the way to do this is to derive fromPreTrainedModeland returnCausalLMOutput, but some of the optional parameters on the output (particularlyattentions) are transformer specific. Am I safe leaving these asNone, or are there going to be times when the generation infrastructure relies on these values? Is there a better way to make use of Transformers’ generation/sampling/beam capabilities with custom models?
2022-02-28T18:03:15Z
I think what you’re trying to do is feasible. As you mentioned, you could inheritPreTrainedModel, but could perhaps also just inheritGenerationMixin. You’d also want to make sure theforwardmethod of your custom class resembles theforwardmethods of Huggingface models.but some of the optional parameters on the output (particularlyattentions) are transformer specific. Am I safe leaving these asNone, or are there going to be times when the generation infrastructure relies on these values?You should be fine setting those toNone. Most of the things in the return objects are only there so users can access them if desired. In fact, theattentionsfield of the return objects is usuallyNone, because it’s controlled by theoutput_attentionsarg, which isFalseby default.
2022-03-01T04:59:55Z
Just inheritingGenerationMixinis a nice idea! I’ve got a version based onPreTrainedModelimplemented now, but I haven’t made it fully compatible with everything that class expects.I’ve opted encapsulate the wholePreTrainedModelclass inside ageneratefunction to hide the details and only expose the few features that I’ve actually implemented.