license:cc-by-4.0 size_categories:1K<n<10K format:json modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc-by-4.0
---

Two types of questions are in this dataset. 

* Synthetic generated questions and answers from Chatgpt 3.5 Turbo on US History.
  * This included a python script to first randomly select a date between 1607-2020. Then generate a historical question
    from that date. Repeated 3000 times.
* This is along with a curated 2048 character chunked American Yawp and Openstax American History,
that were feed into Chatgpt 3.5 Turbo one chunk at a time to generate the context, then a question from 
that context, then an answer from that question.


Python script used to generate synthetic history questions and answer from ChatGPT Turbo-Instruct.

```
import pandas as pd
import random
import openai
from datetime import datetime

# Set your OpenAI API key
openai.api_key = "API KEY"

# Function to generate synthetic historical questions using OpenAI GPT-3
def generate_question():
    # Define a prompt to get a historical question from GPT-3
    prompt = "Generate a historical question related to American History from 1607-2020."

    # Call the OpenAI API to get a synthetic question
    response = openai.Completion.create(
        engine="gpt-3.5-turbo-instruct",
        prompt=prompt,
        max_tokens=50,
        n=1,
        stop=None
    )

    # Extract the generated question from the API response
    question = response['choices'][0]['text'].strip()

    # Generate input and output based on the selected question
    context = f"Make sure the answer does not include any form of list, and that it completes the prompt with a complete sentence."
    input_text = ""  # Set input_text to an empty string

    # Instead of using the question as the answer, let's generate a concise response
    response_prompt = f"{context} Answer the following question: {question} "
    answer_response = openai.Completion.create(
        engine="gpt-3.5-turbo-instruct",
        prompt=response_prompt,
        max_tokens=50,
        n=1,
        stop=None
    )

    # Check the token count and adjust the prompt if needed
    remaining_tokens = 100 - answer_response['usage']['total_tokens']
    if remaining_tokens < 10:
        # If there are fewer than 10 tokens left, reduce the max tokens in the response
        answer_response = openai.Completion.create(
            engine="gpt-3.5-turbo-instruct",
            prompt=response_prompt,
            max_tokens=answer_response['usage']['total_tokens'] + 10,
            n=1,
            stop=None
        )

    output_text = f"{answer_response['choices'][0]['text'].strip()}"

    return question, input_text, output_text

# Create a DataFrame to store the synthetic dataset
data = {'instruction': [], 'input': [], 'output': []}

# Generate 10 entries
for _ in range(100):
    question, input_text, output_text = generate_question()
    data['instruction'].append(question)
    data['input'].append(input_text)
    data['output'].append(output_text)

# Convert the dictionary to a DataFrame
df = pd.DataFrame(data)

# Generate a timestamp for the unique filename
timestamp = datetime.now().strftime("%Y%m%d%H%M%S")

# Save the DataFrame to a CSV file with a unique filename
file_path = f'/content/data_sets/data_set_{timestamp}.csv'
df.to_csv(file_path, index=False)

print(f"Synthetic dataset generated and saved to '{file_path}'")

```

This is the python script to segment books or text files into chunks, then generate a question and answer into a csv. This does produce some 
misalignments between cells. So make sure to look over the data when completed for any data issues. This was generated through colab, so !pip 
instruction are present.

## Generating Chunks

Chunking the sizes helps depending if you want to modify the llm to something with a larger token window. I tried to keep mine within the
token limit of ChatGPT 3.5 Turbo 1106 for the larger context window.

```
# Commented out IPython magic to ensure Python compatibility.
!pip install openai==0.28.0
!pip install pandas
# %mkdir textfiles
# %mkdir chunks

import os

input_file = "/content/textfiles/text.txt"
chunk_size = 8192
output_dir = "chunks"

if not os.path.exists(output_dir):
    os.makedirs(output_dir)

with open(input_file) as f:
    text = f.read()

print(f"Original text length: {len(text)}")

chunks = []
for i in range(0, len(text), chunk_size):
    chunks.append(text[i:i+chunk_size])

last_chunk = text[i+chunk_size:]
if last_chunk:
    chunks.append(last_chunk)

total_chunks = "".join(chunks)
assert total_chunks == text, "Text does not match chunks"

print(f"Total chunks length: {len(total_chunks)}")
print(f"Num chunks: {len(chunks)}")

for i, chunk in enumerate(chunks):
    chunk_file = os.path.join(output_dir, f"chunk{i}.txt")
    with open(chunk_file, "w") as f:
        f.write(chunk)

print(f"Saved {len(chunks)} chunks to {output_dir}")

```

## Generating text (questions and answers) and formating

Using the chatgpt to generate the context of the chunk, then having it create a historical question from that text. Then having it create
the answer. (At time it wanted to put the answers into a summarized list, so I add to the instruction NOT to include it in a list. This 
has seemed to work a little better.)

The last part is to bring it all into a csv file from the format generated from the code above. Make sure that you have made backup to the
chunks gereated by the first python script. If you wish to keep them. The last part of this code deletes the last chunk that it created
an answer and question for. This helped me figure out where I need to stop and start it next time.

```
import os
import openai
import pandas as pd

# Set your OpenAI API key
openai.api_key = 'API KEY'

chunk_dir = "/content/chunks"
qa_dir = "/content/qa_chunks"

# Ensure the output directory exists
if not os.path.exists(qa_dir):
    os.makedirs(qa_dir)

# Function to get historical context and Q&A
def process_chunk(chunk):
    # Get historical context
    context_response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo-1106",
        messages=[{"role": "system", "content": "Determine the historical context of the following text."},
                  {"role": "user", "content": chunk}]
    )
    context = context_response['choices'][0]['message']['content']

    # Generate Q&A
    questions = []
    answers = []
    for i in range(5):
        question_response = openai.ChatCompletion.create(
            model="gpt-3.5-turbo-1106",
            messages=[{"role": "system", "content": "Generate a different question about the historical context of the text."},
                      {"role": "user", "content": chunk}]
        )
        question = question_response['choices'][0]['message']['content']
        questions.append(question)

        answer_response = openai.ChatCompletion.create(
            model="gpt-3.5-turbo-1106",
            messages=[{"role": "system", "content": f"Answer this question based on the text, do not put the answer into a list: {question}"},
                      {"role": "user", "content": chunk}]
        )
        answer = answer_response['choices'][0]['message']['content']
        answers.append(answer)

    return questions, answers

# Function to create a CSV file
def create_csv(questions, answers, filename):
    df = pd.DataFrame({'Question': questions, 'Answer': answers})
    df.to_csv(filename, index=False)

# Read and process each chunk with a pause after every 50 chunks
chunk_counter = 0

# Retrieve and sort chunk file names
chunk_files = sorted([f for f in os.listdir(chunk_dir) if f.startswith("chunk") and f.endswith(".txt")], key=lambda x: int(x[5:-4]))

for chunk_file in chunk_files:
    chunk_path = os.path.join(chunk_dir, chunk_file)
    with open(chunk_path, 'r') as f:
        chunk = f.read()

    questions, answers = process_chunk(chunk)
    csv_file = os.path.join(qa_dir, f"qa_{chunk_file[:-4]}.csv")
    create_csv(questions, answers, csv_file)
    print(f"Processed {chunk_file}, questions and answers saved to {csv_file}")

    # Delete the chunk file after processing
    os.remove(chunk_path)
    print(f"Deleted processed chunk file: {chunk_file}")

    chunk_counter += 1
    if chunk_counter % 50 == 0:
        input("Processed 50 chunks. Press Enter to continue...")

print("Processing complete.")

```