Streamlit App Faster Model Loading
An open question for anyone that has used Transformer models in a Streamlit app.  I am using:::pipeline(“summarization”, model=“sshleifer/distilbart-cnn-6-6”, tokenizer=“sshleifer/distilbart-cnn-6-6”,framework=“pt”)::: to do summarization in the app. However, it takes about 55 seconds to create the summary, and it appears that 35 seconds or more of that time is spent downloading the model.  Is there another way to access the model quicker?  Perhaps by pre-loading the model to Streamlit Sharing (via the github repo the app sits in)?Also, the summary generation part of the app appears to work once or twice, but if done any more times the app crashes.  Has anyone else had this experience?
2021-01-15T08:26:57Z
No experience with Streamlit itself, but you can alwaysdownload the modellocally. Usage is a bit different then: you need to provide a directory to themodelargument instead of just the model name. So download all those files to a directory, and then use that directory as your arguments.
2021-01-15T09:40:48Z
You should wrap the loading of model/pipeline and add astreamlit.cachedecorator. That way, the loading/downloading part will be done only once.https://docs.streamlit.io/en/stable/api.html#optimize-performance
2021-01-21T21:57:01Z
Within thestreamlit.cache()decorator you’ll get better performance if you useallow_output_mutation=Truebecause this means Streamlit just uses the same copy of the model in memory, rather than reloading when it’s re-run.