Perplexity randomly failing due to missing cache file
Hi! I am new to the transformers and evaluate libraries but I am noticing that when trying to calculate perplexity my notebook will randomly fail with the error:image757×355 47.7 KBIt seems to happen randomly. When I callmodel.generateI am passing the following parameters:inputs, min_new_tokens=200, max_length=350, do_sample=do_sample, top_p=top_p, top_k=top_kAnd here is my function for calculating perplexity:from evaluate import load
def calc_perplexity(model_name, text):
  perplexity = load("perplexity", module_type="metric")
  results = perplexity.compute(predictions=text, model_id=model_name)
  return round(results.get('mean_perplexity'),2)The exact same parameters being passed on different runs will sometimes fail and sometimes not. Normally the first call to perplexity will work, but then the second call does not. Any ideas what could be happening here?EDIT: It fails after a few consecutive calls, generally about 3Solved: Well I am a goober. A string cleaning function I had was not returning in some edge scenarios and that caused a bunch of down stream issues. I’ll leave thi post up though in case anyone runs into something similiar.
2024-01-27T06:33:54Z
Here is a copy of my notebook if it helps:Google ColabDon’t judge the code too harshly I’m still learning the library