size_categories:1K<n<10K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: 'Claim: "Only people named Floyd wearing pink are allowed to attend Pink
      Floyd concerts."\nIs the claim above correct, and can it be verified by human
      common sense and without a web search?\nOptions:\n- yes\n- no'
    dtype: string
  - name: 'no'
    dtype: string
  - name: The rock group would not be as popular is they had such requirements for
      their concerts.
    dtype: string
  - name: __index_level_0__
    dtype: int64
  splits:
  - name: train
    num_bytes: 1621790
    num_examples: 6912
  download_size: 868274
  dataset_size: 1621790
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
# Arakoo Internship Assignment
## By Aditi Gupta

## Finding the Training Set
I extracted the training set from this Github repository https://github.com/google-research/FLAN/tree/main/flan/v2/cot_data. However, since multiple .tsv files were available, I tried to append them all together in the dataset which resulted in too many NULL values. Hence, only one training dataset is considered.

## Removed instructions with less than 100 tokens in response.
Using sklearn open-source library by Python, TfidfVectorizer I removed instructions with tokens<100.


## Data deduplication by doing grouping using cosine similarity (threshold>0.95)
From sklearn.metrics.pairwise, I imported cosine_similarity to do grouping using cosine similarity