task_categories:question-answering language:en license:mit size_categories:10K<n<100K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: mit
dataset_info:
  features:
  - name: Q
    dtype: string
  - name: A
    dtype: string
  - name: C
    dtype: string
  - name: source
    dtype: string
  splits:
  - name: train
    num_bytes: 32789385
    num_examples: 39383
  download_size: 7105730
  dataset_size: 32789385
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
task_categories:
- question-answering
language:
- en
pretty_name: j
---
### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->
The dataset consist in a set of Question, Answer and Context. 

Question "Q" and answer "A" are both synthetic and have been created using the model 'meta-llama/Meta-Llama-3.1-8B-Instruct'.<br>
"Context" is the context that has been passed in the prompt to the model.

- **Curated by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Language(s) (NLP):** English
- **License:** [More Information Needed]


## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->
Question and answer are both synthetic and have been created using the model 'meta-llama/Meta-Llama-3.1-8B-Instruct'.
Context is the context that has been passed to the prompt in the question and answer creation.
Source indicates what of the implementation guides contains the context used for this Q&A pair generation.


### Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->
Oficial JD Edwards EnterpriseOne 9.2 implementation guides:

'sales-order-management-implementation-guide.pdf', <br>
 'forecast-management-implementation-guide.pdf',<br>
 'demand-scheduling-execution-implementation-guide.pdf',<br>
 'product-data-management-implementation-guide.pdf',<br>
 'transportation-management-implementation-guide.pdf',<br>
 'requisition-self-service-implementation-guide.pdf',<br>
 'bulk-stock-inventory-implementation-guide.pdf',<br>
 'engineer-to-order-implementation-guide.pdf',<br>
 'general-accounting-implementation-guide.pdf',<br>
 'requirements-planning-implementation-guide.pdf',<br>
 'foundation-guide.pdf',<br>
 'data-interface-for-electronic-data-interchange-implementation-guide.pdf',<br>
 'configurator-implementation-guide.pdf',<br>
 'grower-management-implementation-guide.pdf',<br>
 'advanced-pricing-implementation-guide.pdf',<br>
 'shop-floor-management-implementation-guide.pdf',<br>
 'advanced-cost-accounting-implementation-guide.pdf',<br>
 'inventory-management-implementation-guide.pdf',<br>
 'warehouse-management-implementation-guide.pdf',<br>
 'product-costing-and-manufacturing-accounting-implementation-guide.pdf',<br>
 'address-book-implementation-guide.pdf',<br>
 'grower-pricing-and-payments-implementation-guide.pdf',<br>
 'demand-flow-manufacturing-implementation-guide.pdf'<br>

#### Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->
Guides downloaded from https://docs.oracle.com/

### Annotations
<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->
    
    Each guide was splited into chunks with:
    CharacterTextSplitter(chunk_size=700, chunk_overlap=200, separator="\n",)
    
    For each chunk we prompted the model 

    model_id = 'meta-llama/Meta-Llama-3.1-8B-Instruct'

    pipeline = transformers.pipeline(
      "text-generation",
      model=model_id,
      model_kwargs={"torch_dtype": torch.bfloat16},
      device_map="auto",
    )

    with the next prompt.
    
    prompt= [
    {"role": "system", "content": """You are a chatbot assistant that is trying his best to came up with al least 2 questions suitable to be answered with the text given by the user. 
     You must also provide the answers to those questions. The answers must be grounded in the text and in the text only. 
     You should answer in JSONL format where each line correspond to one question, use Q as the key, and its respective answer, use A as the key."""},
    {"role": "user", "content": f""" Text : {content}"""},

    and converted the answer to a JSON. 

    -aprox 4% of the answers failed to be converted to a json
    
    -Once we have the json with questions "Q" and answers "A" we added C whit the context previously passed to the prompt as "content".

    Cleaning process for the obtained Q & A dataset:
    -removed rows containing answers that were just an integer number.
    -removed rows with "duplicated" answers.
      for this we got the embedding for each answer and built a cosine similarity matrix. We kept one answer for each set having cosine similarity higher than 0.99
      embedding_model = 'paraphrase-multilingual-mpnet-base-v2'
    -remove rows with answers that referred to page numbers    .
    -remove rows with questions that referred to examples given in the guides.
    -remove rows with answers containg web urls.
     
      