Context shift constantly failing?
Is that only me, or with Q8_0, context shift randomly failing, causing full reprocess of context? Looks like i have this issue only with this model, other nemo-based models are fine. Is there some problem with tokenizer maybe?
I'm using koboldcpp_rocm, and tried Alpaca, Mistral and ChatML format - it's still the same. It can be fine for a few swipes, or even for a few dozen swipes, but then it fails randomly and reprocess everything again.
This model is composed of 3 fine tuned models. It is possible one or more is causing the issue and/or the way they were merged.
This model is design for mistral instruct template ; however it should also work with alpaca, and may work ok with chatml.
The other option: disable context shift.
"Context shift" is known to cause issues.
I see. Well, that's a shame. I have to use context shift anyway, because my GPU can hold up to 32k, and some chats are longer. Thanks anyway.