How can we test Transformer Models after converting it to TFLite format
I tried converting the Distilbert Model of Huggingface to TFLite format using this scriptReference ScriptI was able to convert it. After that, I wanted to test it in Python itself using tf.lite.Interpreter but I was not able to figure out the correct input dimension required for the model.  Can anyone help me with this?Colab Notebook
2020-10-21T15:23:50Z
Pinging@jplu,  as he’s working on TF
2020-10-22T12:52:46Z
Hello!Unfortunately I’m not really familiar with TFLite sorry. As a first guess I would suggest to get your features directly from the Tokenizers and see if it works.
2020-10-23T09:22:50Z
Hello@jpluI tested TFdistilbertModel but after conversion, it’s only expecting tensor of 3*5 dimension. The output of the tokenizer is a single-dimensional array.I also try to set input like thisinput_spec = tf.TensorSpec([1, 768], tf.int32)
model._set_inputs(input_spec, training=False)But when I checkmodel.inputsit wasNoneand  I was getting same error related to dimension.
2020-10-23T13:46:28Z
From what I see in your colab, your TFlite model is waiting only one input (input_ids) and you are giving two (input_ids + attention_mask). Also the method_set_inputsdoesn’t work with TF >= 2.2. You have to use:model._saved_model_inputs_spec = None
model._set_save_spec(input_spec)When doingTFDistilBertModel.from_pretrained('distilbert-base-uncased')the model is built with a fix input, a batch of three sentences of 3 tokens (then 5 with start and end tokens). So if you want it to fit your needs when creating your savedmodel/TFlite model you have to modify this beforehand as I specified above.
2020-10-26T15:46:18Z
Hi@jpluIs there a way to allow dynamic input size rather than setting a pre-fixed size? Thanks.
2020-10-26T16:56:29Z
Not for now.
2020-10-27T08:19:36Z
Thanks@jpluBy considering your suggestion by setting input shape, I was able to do inference on TFlite models converted from Huggingface.Here is myNotebookfor someone who is facing such an issue.
2024-02-10T16:23:40Z
@bhadresh-savanihello I could not access the notebook you provided. Could you send an updated link?
2024-03-26T23:41:08Z
github.combhadreshpsavani/UnderstandingNLP/blob/master/Notebooks/TFLite/TFLiteExperiments.ipynb{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TFLiteExperiments.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPA2eFqMnkEGqgxIQ3c0DXr",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "78a86cd1b2334981854a64e4a7aa6e1e": {
          "model_module": "@jupyter-widgets/controls",This file has been truncated.show original