Getting repetative response using ConversationalRetrievalChain + HugginFaace
Hello I’m Trying To move from Chat GPT api to Hugging Face using ConversationalRetrievalChainWhen Getting final answer From llm I’m getting repeated same message which is very weirdHere is example of the codeasync def create_crc_chain(self, user_id, repo: UserRepo):“”“Creatates Chain For User and stores it in SimpleChainManager”“”# Creating session# async with self.session_pool() as session:#     repo = RequestsRepo(session)#     await repo.basic.get_or_create_session(user_id)# ---Old tamplate solution----
    # prompt = PromptTemplate(
    #     input_variables=["question"],
    #     # , "twitter_information"],
    #     template=SYSTEM_PROMPT,

    # )

    prompt = PromptTemplate.from_template(SYSTEM_PROMPT)

    ######  Setting Multiquery retriever as base retriver ######
    QUERY_PROMPT = PromptTemplate(
        input_variables=["question"],
        template="""You are an AI language model assistant.\n
    Your task is to generate 3 different versions of the given user question to retrieve relevant documents from a vector  database.\n 
    By generating multiple perspectives on the user question, your goal is to help the user overcome some of the limitations of distance-based similarity search.\n
    Provide these alternative questions separated by newlines.\n
    Original question: {question}"""
    )

    document_prompt = PromptTemplate(
        input_variables=["page_content", "source"],
        template="Context:\nContent:{page_content}\n Source file name:{source}",
    )

    condense_question_prompt = PromptTemplate.from_template(QUESTION_PROMPT_TEMPLATE)
    model = "meta-llama/Meta-Llama-3-70B-Instruct"
    hb_token = os.getenv('HUGGINGFACEHUB_API_TOKEN')
    llm = HuggingFaceEndpoint(
        repo_id=model,
        huggingfacehub_api_token=hb_token,
        temperature=0.4,
    )
    document_llm_chain = LLMChain(
        llm=llm,
        prompt=prompt,
        callbacks=None,
        verbose=True
    )

    combine_documents_chain = StuffDocumentsChain(
        llm_chain=document_llm_chain,
        document_variable_name="context",
        document_prompt=document_prompt,
        callbacks=None,
        verbose=True
    )

    base_retriever = MultiQueryRetriever.from_llm(
        retriever=self.docsearch.as_retriever(
            search_kwargs={"k": 20}
        ),
        prompt=QUERY_PROMPT,
        llm=llm
    )

    compressor = CohereRerank(top_n=1, cohere_api_key='T8j5GzGyw0SPCm8wv8ewwEYDsCALVU7lw3EnmwdY')
    # compression_retriever = ContextualCompressionRetriever(
    #     base_compressor=compressor, base_retriever=self.docsearch.as_retriever(search_kwargs={"k": 20})
    # )
    compression_retriever = ContextualCompressionRetriever(
        base_compressor=compressor, base_retriever=base_retriever
    )

    question_generator_chain = LLMChain(
        llm=llm,
        prompt=condense_question_prompt,
        verbose=True)

    memory = await self.create_memory_with_history(
        repo, user_id,
        llm=llm,
    )

    qa = ConversationalRetrievalChain(
        combine_docs_chain=combine_documents_chain,
        question_generator=question_generator_chain,
        callbacks=None,
        verbose=True,
        retriever=compression_retriever,
        return_source_documents=True,
        memory=memory
    )Here is example of input outputinput : hioutput : Hi! I’m here to help you navigate the world of dating billionaire men. I can assist you in having efficient conversations with them and potentially leading to successful relationships. What would you like to know or discuss? Do you have a specific question or situation in mind? Feel free to ask, and I’ll do my best to provide you with personalized guidance and advice.assistantHi! I’m here to help you navigate the world of dating billionaire men. I can assist you in having efficient conversations with them and potentially leading to successful relationships. What would you like to know or discuss? Do you have a specific question or situation in mind? Feel free to ask, and I’ll do my best to provide you with personalized guidance and advice.assistantHi! I’m here to help you navigate the world of dating billionaire men. I can assist you in having efficient conversations with them and potentially leading to successful relationships. What would you like to know or discuss? Do you have a specific question or situation in mind? Feel free to ask, and I’ll do my best to provide you with personalized guidance and advice.assistantetc…What can be issue and maybe you have some thought how I can fix it