AI model (llama) is producing garbage output
I am trying to get a local deployment of a llama model going and I am having some trouble. While I am able to get output from the model, it is not what I want at all. The output might be loosely related, but its mostly garbage.Here is some sample input/output:Input:  Hello, what is your name?AI Output:What is your name? (BRYAN)Where do you live?I am from the west coast of AmericaDo I have any siblings or family members that might be able to help me out?Nope2c4d7a0e8-5f9b-c3ac-f6fa-d1edccfcffeb.pngHow old were they when their parents died?Who was there for them after school hours and on weekends?Did anyone elseThe output cuts off at the above statement, presumably due to the max_length param. I am using the normal transformers pipeline and I have downloaded the llama repository and point the pipeline to that repo like this:model = “llama/meta-llama/Llama-3.2-3”tokenizer = AutoTokenizer.from_pretrained(model)pipeline = transformers.pipeline(“text-generation”, # question-answeringmodel=model,torch_dtype=torch.float16,device_map=“auto”,tokenizer=tokenizer)Any ideas on how to improve the output?
2025-01-09T14:25:45Z
I think you didn’t use any prompt. I am sure you know about prompt. So you should use system prompt and assist prompt and so on. And your model doesn’t work well in real world situations. So I recommend you use llama-instruct model. This model retrained on the real world problems especially, chatbot, summarization. And it also work well on various types of prompt.
2025-01-09T14:34:55Z
I recommend you use llama-instruct modelI agree. Also, it seems you are using transformers, but that software is more of a library suited to advanced users, for detailed specialized tasks, as well as customizing models and training.If your main use is for chatting, I recommend using Ollama, which is simpler, faster, and based on the Instruct model.huggingface.coUse Ollama with any GGUF Model on Hugging Face HubWe’re on a journey to advance and democratize artificial intelligence through open source and open science.