RAG isnt working as expected
This is my code:import faissfrom sentence_transformers import SentenceTransformerfrom transformers import AutoTokenizer, AutoModelForCausalLMimport torchimport pickleimport osLoad pickle filesdata =for filename in os.listdir(‘.’):  # replace with your directoryif filename.endswith(‘.pickle’):with open(os.path.join(‘./’, filename), ‘rb’) as f:datapoint = pickle.load(f)if datapoint != “nothing”:data.append(datapoint)Load pre-trained model for encoding text to vectorsmodel_data_encoding = SentenceTransformer(‘all-MiniLM-L6-v2’)Encode all data to vectorsdata_vectors = [model_data_encoding.encode(f"{datapoint[‘names’]}: {datapoint[‘axioms’]}") for datapoint in data]Build a FAISS indexdimension = data_vectors[0].shape[0]index = faiss.IndexFlatL2(dimension)for vector in data_vectors:index.add(vector.reshape(1, -1))Load pre-trained LLMtokenizer = AutoTokenizer.from_pretrained(“gpt2”)model = AutoModelForCausalLM.from_pretrained(“gpt2”)tokenizer.pad_token_id = tokenizer.eos_token_iddef answer_question(question):# Encode question to vectorquestion_vector = model_data_encoding.encode(question)# Find the most similar documents in the database
distances, indices = index.search(question_vector.reshape(1, -1), 5)

# Get the corresponding data
relevant_data = [data[index] for index in indices[0]]

# Format the data for the LLM
inputs = tokenizer("Given these retrieved axioms from a mathematical database:\n" +" ".join([f"{datapoint['names'][0]}: {datapoint['axioms']}" for datapoint in relevant_data]) + "\nAnswer this question:\n" + question + "\nAnswer: ", return_tensors="pt")

print("Given these retrieved axioms from a mathematical database:\n" +"\n".join([f"{datapoint['names'][0]}: {datapoint['axioms']}" for datapoint in relevant_data]) + "\nAnswer this question:\n" + question + "\nAnswer: ")
# Generate a response
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.7)

# Decode the response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

# Parsing out the first sentence
first_sentence = response.split("Answer:")[1].split(".")[0]

return first_sentencequestion = “what is the LA-semihypergroup axiom?”response = answer_question(question)print(response)The response I am getting is:The LA-semihypergroup axiom is a generalization of the axiom of the inverse semigroupInstead of the “obvious” answer:(x*y)z = (zy)*xWhy could this be happening and how can I improve it?
2024-05-02T15:55:01Z
To get this , formula based answer you should have this answer in expernal data (pickle , I guess). And embedding model with tokenizer that support mathematics language . I am afraid all-MiniLM-L6-v2 is not like this. Still hard to give precise answer , but these two direction I would check.
2024-05-02T16:06:26Z
the retrieval is working perfectly fine though, the problem is really the generation. If i put he prompt with the retrieved axioms into chatgpt it works fine. Could this just be a problem with the gpt2 model?
2024-05-02T18:10:28Z
Very likely. If retrieve part works , the problem with synthesizing the answer by GPT2 . Try to run your Retriever with GPT4 API with llama_index and see if your get the same you get in chatGPT . You have tones of examples on the llma_index doc site.