license:cc-by-sa-4.0 size_categories:10K<n<100K format:csv modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc-by-sa-4.0
---

Downloaded from https://archive.org/download/stackexchange

Used [TexSoup](https://pypi.org/project/TexSoup/) to replace all text in math environments with [UNK].  For instance the text:

"The integral $\int_a^b f(x) \textrm{ d}x$ is easy to evaluate if..."

was replaced with

"The integral [UNK] is easy to evaluate if..."

Note:  There is still some "ascii math".  For instance, people sometimes write things like f: X --> Y.  This is retained. 

Concatenated title and body.

Some of these are "answer" posts rather than "question" posts.  In the original data these are untagged.  
I tagged each "answer" post with the tags of the question they are responding to.

I only retained posts which used at least one of the 32 arxiv tags ('ac.commutative-algebra',
 'ag.algebraic-geometry', ..., 'st.statistics').

I only retained posts which had >5 upvotes.

The train/valid/test split was accomplished using [MultilabelStratifiedShuffleSplit](https://github.com/trent-b/iterative-stratification).
This does a better job of respecting multilabel co-occurance statistics than a purely random split.