license:cc-by-4.0 size_categories:10M<n<100M format:json modality:text library:datasets library:dask library:mlcroissant region:us
---
license: cc-by-4.0
---
# Synthetic mass spectra dataset No. 2
This is the second iteration of the GC-EI-MS synthetic dataset generated by [NEIMS] and [RASSP] models from 4.8 million structures downloaded from [ZINC 20].
The overlap with version 1 is approximately 8k structures (negligible), so it's usable as an extension to synth1. 


## Spectra generators 
For generating this dataset, we used custom-trained NEIMS and RASSP models. We did this step because RASSP model is not publicly available and we wanted to 
have more control over potential dataleaks in the context of further models stemming from this dataset. For better reproducibility we make the NIST splits 
used for training NEIMS and RASSP models available in our [GitHub repository] (#TODO)


## Dataset creation

To uniformly cover the chemical space, we first scraped 1.8 billion SMILES strings from the ZINC20 library using the 2D-standard-annotated-druglike query. 
From this dataset, we extracted a random sample of 30 million noncorrupted SMILES strings shorter than 100 characters. Further, we canonicalized, deduplicated,
and stripped the SMILES of stereochemical information, as stereochemistry is not reliably translated into spectra and cannot be accurately detected.
Additionally, we removed all NIST 20 compounds to prevent data leakage during the testing of our trained model in the future.

To evaluate the impact of RASSP and NEIMS respective spectrum prediction capabilities on SpecTUS performance, we created two identical sets of spectra generated
by each model. RASSP's strict input filters reduced the dataset to approximately 4.8 million compounds. In total, the pretraining datasets comprises 4.8 million
unique compounds and 9.6 million unique spectra.

Lastly, we split each synthetic dataset into training, validation, and test sets using a 0.9:0.05:0.05 ratio. The splitting process was random, but corresponding splits
(training, validation, and test sets) for the NEIMS and RASSP-generated spectra contained the same compounds.

## Size

```text
rassp_custom_gen/train.jsonl    4364744
rassp_custom_gen/valid.jsonl    242486
rassp_custom_gen/test.jsonl     242486
TOTAL          4849716

neims_custom_gen/train.jsonl    4364748
neims_custom_gen/valid.jsonl    242483
neims_custom_gen/test.jsonl     242485
TOTAL          4849716
```

<!-- 
## Dataset choice
If you're choosing between synth1 and synth2 (and don't need both), choose synth2. The origin of synth2 is better documented in the 3_datapreprocessing.ipynb notebook in our [GitHub repository], 
even though the creation process is in essence the same for both datasets. 
There are the same compound in the splits of both RASSP and NEIMS generated datasets but some
duplicities emerged, so the sets are not identical (!the scale of the leak is less than 4 compounds in almost 5M set, and the test and validation sets are not used for
the final testing and validation in our paper, so we call it negligible!)

 -->
## Data format

Each line of every file is a `json` comprising three items:

```text
{"intensity":[0.01,0.08,0.06,0.05 ... 0.02,0.79,0.34,0.12],
"mz":[18,27,28,29,38,39,40,41,42,43, ... 202,203,270,271,272,299,300,301],
"smiles":"CCC(C)C1CCCCN1C(=O)CNc1cccc(C#N)c1"}
```

From the nature of the NEIMS spectra predictor, all NEIMS-generated peak intensities are in 2 decimals precision. RASSP generated intensities are not restricted in this way,
so to save storage we rounded them up to 6 decimal places which is beyond recognition for our model in the downstream task.

Our [preprint] (TODO) provides more information about the task background, the final finetuned model, and the experiments.



[NEIMS]: https://github.com/brain-research/deep-molecular-massspec
[RASSP]: https://github.com/thejonaslab/rassp-public
[ZINC 20]: https://zinc20.docking.org/
[GitHub repository]: TODO