Inconsistent model inference time
Recently, I want to measure the inference time, I design two cases, 
First I read model input from files, 
Second, each model input from the standard input.

While I found first case inference time is about 70 ms except the first call, the second case starts to be highly inconsistent, ranging from 72 ms to 483 ms. 

Why this happened ? How can I make the second case inference time keep consistent ?

```python
model = TokenClassificationModel.restore_from(restore_path=checkpoint_path)
# first case code
for line in open(txt_file, 'r'):
    torch.cuda.synchronize()
    start = time.perf_counter()
    result = model.infer_one_text(text)
    torch.cuda.synchronize()
    elapsed_time = time.perf_counter() - start
    print(f'elapsed_time:{elapsed_time:.6f}', flush=True)

# first case log
elapsed time:0.289745
elapsed time:0.072184
elapsed time:0.060471
elapsed time:0.062084
elapsed time:0.063837
```

```python
# second case code
model = TokenClassificationModel.restore_from(restore_path=checkpoint_path)
while True:
    text = input("text:")
    torch.cuda.synchronize()
    start = time.perf_counter()
    result = model.infer_one_text(text)
    torch.cuda.synchronize()
    end = time.perf_counter() - start
    print(f'elapsed time: {end:.6f}', flush=True)

# second case log
elapsed time:0.483897
elapsed time:0.089466
elapsed time:0.450116
elapsed time:0.072568
elapsed time:0.469203
```
