region:us
---
dataset_info:
  features:
  - name: id
    dtype: string
  - name: submitter
    dtype: string
  - name: authors
    dtype: string
  - name: title
    dtype: string
  - name: comments
    dtype: string
  - name: journal-ref
    dtype: string
  - name: doi
    dtype: string
  - name: report-no
    dtype: string
  - name: categories
    dtype: string
  - name: license
    dtype: string
  - name: orig_abstract
    dtype: string
  - name: versions
    list:
    - name: created
      dtype: string
    - name: version
      dtype: string
  - name: update_date
    dtype: string
  - name: authors_parsed
    sequence:
      sequence: string
  - name: abstract
    dtype: string
  - name: prompt
    dtype: string
  - name: y_true
    dtype: string
  - name: comp_Llama-3.1-8B
    dtype: string
  - name: preds_Llama-3.1-8B
    dtype: string
  - name: comp_s1-L-3.1-8B-base
    dtype: string
  - name: preds_s1-L-3.1-8B-base
    dtype: string
  - name: comp_s2-L-3.1-8B-base
    dtype: string
  - name: preds_s2-L-3.1-8B-base
    dtype: string
  - name: comp_s3-L-3.1-8B-base_v3
    dtype: string
  - name: preds_s3-L-3.1-8B-base_v3
    dtype: string
  - name: comp_s4-L-3.1-8B-base
    dtype: string
  - name: preds_s4-L-3.1-8B-base
    dtype: string
  - name: comp_s5-L-3.1-8B-base
    dtype: string
  - name: preds_s5-L-3.1-8B-base
    dtype: string
  - name: comp_s6-L-3.1-8B-base
    dtype: string
  - name: preds_s6-L-3.1-8B-base
    dtype: string
  splits:
  - name: test
    num_bytes: 675556620
    num_examples: 45195
  download_size: 311429772
  dataset_size: 675556620
configs:
- config_name: default
  data_files:
  - split: test
    path: data/test-*
---


Takes the LLMsForHepth/hep-th_hep-ph_gr-qc_primary_v3 dataset and appends columns `prompt` and `y_true`. 
Various models are then run, they take as inputs the data in the `prompt` column and generate text which completes each prompt.
Each models' output is stored in a column starting with 'comp' or 'preds', for example `comp_s3-L-3.1-8B-base_v3` and `preds_s3-L-3.1-8B-base_v3`,
which respectively hold the full completions (including prompt) and predictions (excluding prompt) generated by the model LLMsForHepth/s3-L-3.1-8B-base_v3.

The generation is done with the following parameters

- temperature: 0.7
- max_new_tokens: 1024
- min_new_tokens: 1
- do_sample: True

Each model is loaded using:

model_cfg = {"attn_implementation": "sdpa",
             "device_map": "auto",
             "torch_dtype": torch.float16
             }