How do I create Datasets from PDF files?
Has anyone managed to extract text from PDF files and create a good question-and-answer dataset from the extracted text? Or generally create datasets from given PDF files?
2025-01-09T12:30:10Z
There are several libraries for Python.github.comGitHub - VikParuchuri/marker: Convert PDF to markdown + JSON quickly with high...Convert PDF to markdown + JSON quickly with high accuracygithub.comGitHub - pdfminer/pdfminer.six: Community maintained fork of pdfminer - we fathom...Community maintained fork of pdfminer - we fathom PDFpymupdf.readthedocs.ioPyMuPDF4LLM - PyMuPDF 1.25.1 documentationPyMuPDF is a high-performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents.github.comGitHub - CambioML/uniflow-llm-based-pdf-extraction-text-cleaning-data-clustering: LLM-based text extraction from unstructured data...LLM-based text extraction from unstructured data like PDFs, Words and HTMLs. Transform and cluster the text into your desired format. Less information loss, more interpretation, and faster R&D!Edit:Code sample byHugging Chat.pip install PyMuPDF
pip install spacy
python -m spacy download en_core_web_sm
pip install datasetsimport fitz  # PyMuPDF
import spacy
from datasets import Dataset

# Load the spaCy model
nlp = spacy.load("en_core_web_sm")

def extract_text_from_pdf(pdf_path):
    document = fitz.open(pdf_path)
    text = ""
    for page_num in range(len(document)):
        page = document.load_page(page_num)
        text += page.get_text("text")
    return text

def identify_questions(text):
    doc = nlp(text)
    questions = []
    for sent in doc.sents:
        if sent[-1].text == '?':
            questions.append(sent.text)
    return questions

def simple_qa_pairs(text):
    questions = identify_questions(text)
    qa_pairs = []
    doc = nlp(text)
    sentences = list(doc.sents)
    for question in questions:
        question_idx = [i for i, sent in enumerate(sentences) if sent.text == question][0]
        if question_idx > 0:
            answer = sentences[question_idx - 1].text
            qa_pairs.append({"question": question, "answer": answer})
    return qa_pairs

def create_qa_dataset(pdf_path):
    text = extract_text_from_pdf(pdf_path)
    qa_pairs = simple_qa_pairs(text)
    dataset = Dataset.from_list(qa_pairs)
    return dataset

# Example usage
pdf_path = "example.pdf"
dataset = create_qa_dataset(pdf_path)

# Save the dataset to disk
dataset.save_to_disk("qa_dataset")

# Load the dataset to verify
loaded_dataset = Dataset.load_from_disk("qa_dataset")
for example in loaded_dataset:
    print("Question:", example["question"])
    print("Answer:", example["answer"])
2025-01-16T17:49:34Z
I will suggest you combine a RAG system with a good amount of prompt engineering.The PDF file will be embedded and stored in a vector DB, which you will query via a vector similarity_search with score. The results will be the contents of the document that have closer vector values compared to the vector of the search query.You save each query and its search result with the highest similarity_score in a dataFrame.PS: You can use a good prompt to get a lot of questions related to your type of document and its content.  A for loop can handle the rest (from querying the vector db for each question) and appending the query and result (with highest similarity_score) into a dataFrame.
2025-01-16T23:36:06Z
Luli1409:Has anyone managed to extract text from PDF files and create a good question-and-answer dataset from theBusanbextracted text? Or generally create datasets from given PDF files?Yes, it’s possible to extract text from PDFs and create Q&A datasets. Tools like PyPDF2 or PDFMiner can extract text, while NLP models can generate questions and answers. This process often involves cleaning the text, extracting key information, and using models for question generation.
2025-01-17T01:33:40Z
not sure if it’s totally ready for use but nvidia seems to be doing something similar to what you’re looking forgithub.comGitHub - NVIDIA/nv-ingest: NVIDIA Ingest is an early access set of...NVIDIA Ingest is an early access set of microservices for parsing hundreds of thousands of complex, messy unstructured PDFs and other enterprise documents into metadata and text to embed into retrieval systems.
2025-01-17T02:30:06Z
This extracts data from pdfs to jsongithub.comGitHub - DocumindHQ/documind: Open-source platform for extracting structured...Open-source platform for extracting structured data from documents using AI.