language:war language:ceb language:min language:vie language:ilo language:tgl language:lao language:khm language:mya language:jav language:ind language:tha language:sun language:zlm license:cc0-1.0 arxiv:2406.10118 region:us self-supervised-pretraining

---
license: cc0-1.0
language: 
- war
- ceb
- min
- vie
- ilo
- tgl
- lao
- khm
- mya
- jav
- ind
- tha
- sun
- zlm
pretty_name: Oscar 2201
task_categories: 
- self-supervised-pretraining
tags: 
- self-supervised-pretraining
---

OSCAR or Open Super-large Crawled Aggregated coRpus is a huge multilingual corpus
obtained by language classification and filtering of the Common Crawl corpus using
the ungoliant architecture. Data is distributed by language in both original and
deduplicated form.


## Languages

war, ceb, min, vie, ilo, tgl, lao, khm, mya, jav, ind, tha, sun, zlm

## Supported Tasks

Self Supervised Pretraining

## Dataset Usage
### Using `datasets` library
```
from datasets import load_dataset
dset = datasets.load_dataset("SEACrowd/oscar_2201", trust_remote_code=True)
```
### Using `seacrowd` library
```import seacrowd as sc
# Load the dataset using the default config
dset = sc.load_dataset("oscar_2201", schema="seacrowd")
# Check all available subsets (config names) of the dataset
print(sc.available_config_names("oscar_2201"))
# Load the dataset using a specific config
dset = sc.load_dataset_by_config_name(config_name="<config_name>")
```

More details on how to load the `seacrowd` library can be found [here](https://github.com/SEACrowd/seacrowd-datahub?tab=readme-ov-file#how-to-use).


## Dataset Homepage

[https://huggingface.co/datasets/oscar-corpus/OSCAR-2201](https://huggingface.co/datasets/oscar-corpus/OSCAR-2201)

## Dataset Version

Source: 2022.1.0. SEACrowd: 2024.06.20.

## Dataset License

Creative Commons Zero v1.0 Universal (cc0-1.0)

## Citation

If you are using the **Oscar 2201** dataloader in your work, please cite the following:
```
@inproceedings{abadji2022cleaner,
    author    = {Julien Abadji and
                Pedro Javier Ortiz Su{'{a}}rez and
                Laurent Romary and
                Beno{\^{\i}}t Sagot},
    title     = {Towards a Cleaner Document-Oriented Multilingual Crawled Corpus},
    booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference,
                {LREC} 2022, Marseille, France, 20-25 June 2022},
    pages     = {4344--4355},
    publisher = {European Language Resources Association},
    year      = {2022},
    url       = {https://aclanthology.org/2022.lrec-1.463},
}

@inproceedings{abadji2021ungoliant,
    author    = {Julien Abadji and
                Pedro Javier Ortiz Su{'a}rez and
                Laurent Romary and
                Beno{\^i}t Sagot},
    title     = {Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus},
    series    = {Proceedings of the Workshop on Challenges in the Management of Large Corpora
                (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event)},
    editor    = {Harald L{"u}ngen and
                Marc Kupietz and
                Piotr Bański and
                Adrien Barbaresi and
                Simon Clematide and
                Ines Pisetta},
    publisher = {Leibniz-Institut f{"u}r Deutsche Sprache},
    address   = {Mannheim},
    doi       = {10.14618/ids-pub-10468},
    url       = {https://nbn-resolving.org/urn:nbn:de:bsz:mh39-104688},
    pages     = {1 -- 9},
    year      = {2021},
    abstract  = {Since the introduction of large language models in Natural Language
    Processing, large raw corpora have played a crucial role in Computational Linguistics.
    However, most of these large raw corpora are either available only for English or not
    available to the general public due to copyright issues. Nevertheless, there are some
    examples of freely available multilingual corpora for training Deep Learning NLP
    models, such as the OSCAR and Paracrawl corpora. However, they have quality issues,
    especially for low-resource languages. Moreover, recreating or updating these corpora
    is very complex. In this work, we try to reproduce and improve the goclassy pipeline
    used to create the OSCAR corpus. We propose a new pipeline that is faster, modular,
    parameterizable, and well documented. We use it to create a corpus similar to OSCAR
    but larger and based on recent data. Also, unlike OSCAR, the metadata information is
    at the document level. We release our pipeline under an open source license and
    publish the corpus under a research-only license.},
    language  = {en}
}

@article{kreutzer2022quality,
    title     = {Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets},
    author    = {Kreutzer, Julia  and
                Caswell, Isaac  and
                Wang, Lisa  and
                Wahab, Ahsan  and
                van Esch, Daan  and
                Ulzii-Orshikh, Nasanbayar  and
                Tapo, Allahsera  and
                Subramani, Nishant  and
                Sokolov, Artem  and
                Sikasote, Claytone  and
                Setyawan, Monang  and
                Sarin, Supheakmungkol  and
                Samb, Sokhar  and
                Sagot, Beno{\^\i}t  and
                Rivera, Clara  and
                Rios, Annette  and
                Papadimitriou, Isabel  and
                Osei, Salomey  and
                Suarez, Pedro Ortiz  and
                Orife, Iroro  and
                Ogueji, Kelechi  and
                Rubungo, Andre Niyongabo  and
                Nguyen, Toan Q.  and
                M{"u}ller, Mathias  and
                M{"u}ller, Andr{'e}  and
                Muhammad, Shamsuddeen Hassan  and
                Muhammad, Nanda  and
                Mnyakeni, Ayanda  and
                Mirzakhalov, Jamshidbek  and
                Matangira, Tapiwanashe  and
                Leong, Colin  and
                Lawson, Nze  and
                Kudugunta, Sneha  and
                Jernite, Yacine  and
                Jenny, Mathias  and
                Firat, Orhan  and
                Dossou, Bonaventure F. P.  and
                Dlamini, Sakhile  and
                de Silva, Nisansa  and
                {\c{C}}abuk Ball{\i}, Sakine  and
                Biderman, Stella  and
                Battisti, Alessia  and
                Baruwa, Ahmed  and
                Bapna, Ankur  and
                Baljekar, Pallavi  and
                Azime, Israel Abebe  and
                Awokoya, Ayodele  and
                Ataman, Duygu  and
                Ahia, Orevaoghene  and
                Ahia, Oghenefego  and
                Agrawal, Sweta  and
                Adeyemi, Mofetoluwa},
    editor    = {Roark, Brian  and
                Nenkova, Ani},
    journal   = {Transactions of the Association for Computational Linguistics},
    volume    = {10},
    year      = {2022},
    address   = {Cambridge, MA},
    publisher = {MIT Press},
    url       = {https://aclanthology.org/2022.tacl-1.4},
    doi       = {10.1162/tacl_a_00447},
    pages     = {50--72},
    abstract  = {With the success of large-scale pre-training and multilingual modeling in
    Natural Language Processing (NLP), recent years have seen a proliferation of large,
    Web-mined text datasets covering hundreds of languages. We manually audit the quality
    of 205 language-specific corpora released with five major public datasets (CCAligned,
    ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At
    least 15 corpora have no usable text, and a significant fraction contains less than
    50{\%} sentences of acceptable quality. In addition, many are mislabeled or use
    nonstandard/ambiguous language codes. We demonstrate that these issues are easy to
    detect even for non-proficient speakers, and supplement the human audit with automatic
    analyses. Finally, we recommend techniques to evaluate and improve multilingual
    corpora and discuss potential risks that come with low-quality data releases.},
}

@inproceedings{ortizsuarez2020monolingual,
    title     = {A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages},
    author    = {Ortiz Su{'a}rez, Pedro Javier  and
                Romary, Laurent  and
                Sagot, Benoit},
    booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
    month     = {jul},
    year      = {2020},
    address   = {Online},
    publisher = {Association for Computational Linguistics},
    url       = {https://www.aclweb.org/anthology/2020.acl-main.156},
    pages     = {1703--1714},
    abstract  = {We use the multilingual OSCAR corpus, extracted from Common Crawl via
    language classification, filtering and cleaning, to train monolingual contextualized
    word embeddings (ELMo) for five mid-resource languages. We then compare the
    performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on
    the part-of-speech tagging and parsing tasks. We show that, despite the noise in the
    Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than
    monolingual embeddings trained on Wikipedia. They actually equal or improve the
    current state of the art in tagging and parsing for all five languages. In particular,
    they also improve over multilingual Wikipedia-based contextual embeddings
    (multilingual BERT), which almost always constitutes the previous state of the art,
    thereby showing that the benefit of a larger, more diverse corpus surpasses the
    cross-lingual benefit of multilingual embedding architectures.},
}

@inproceedings{ortizsuarez2019asynchronous,
    author    = {Pedro Javier {Ortiz Su{'a}rez} and
                Benoit Sagot and
                Laurent Romary},
    title     = {Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures},
    series    = {Proceedings of the Workshop on Challenges in the Management of Large Corpora
                (CMLC-7) 2019. Cardiff, 22nd July 2019},
    editor    = {Piotr Bański and
                Adrien Barbaresi and
                Hanno Biber and
                Evelyn Breiteneder and
                Simon Clematide and
                Marc Kupietz and
                Harald L{"u}ngen and
                Caroline Iliadi},
    publisher = {Leibniz-Institut f{"u}r Deutsche Sprache},
    address   = {Mannheim},
    doi       = {10.14618/ids-pub-9021},
    url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},
    pages     = {9 -- 16},
    year      = {2019},
    abstract  = {Common Crawl is a considerably large, heterogeneous multilingual corpus
    comprised of crawled documents from the internet, surpassing 20TB of data and
    distributed as a set of more than 50 thousand plain text files where each contains
    many documents written in a wide variety of languages. Even though each document has a
    metadata block associated to it, this data lacks any information about the language in
    which each document is written, making it extremely difficult to use Common Crawl for
    monolingual applications. We propose a general, highly parallel, multithreaded
    pipeline to clean and classify Common Crawl by language; we specifically design it so
    that it runs efficiently on medium to low resource infrastructures where I/O speeds
    are the main constraint. We develop the pipeline so that it can be easily reapplied to
    any kind of heterogeneous corpus and so that it can be parameterised to a wide range
    of infrastructures. We also distribute a 6.3TB version of Common Crawl, filtered,
    classified by language, shuffled at line level in order to avoid copyright issues, and
    ready to be used for NLP applications.},
    language  = {en}
}


@article{lovenia2024seacrowd,
    title={SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages}, 
    author={Holy Lovenia and Rahmad Mahendra and Salsabil Maulana Akbar and Lester James V. Miranda and Jennifer Santoso and Elyanah Aco and Akhdan Fadhilah and Jonibek Mansurov and Joseph Marvin Imperial and Onno P. Kampman and Joel Ruben Antony Moniz and Muhammad Ravi Shulthan Habibi and Frederikus Hudi and Railey Montalan and Ryan Ignatius and Joanito Agili Lopo and William Nixon and Börje F. Karlsson and James Jaya and Ryandito Diandaru and Yuze Gao and Patrick Amadeus and Bin Wang and Jan Christian Blaise Cruz and Chenxi Whitehouse and Ivan Halim Parmonangan and Maria Khelli and Wenyu Zhang and Lucky Susanto and Reynard Adha Ryanda and Sonny Lazuardi Hermawan and Dan John Velasco and Muhammad Dehan Al Kautsar and Willy Fitra Hendria and Yasmin Moslem and Noah Flynn and Muhammad Farid Adilazuarda and Haochen Li and Johanes Lee and R. Damanhuri and Shuo Sun and Muhammad Reza Qorib and Amirbek Djanibekov and Wei Qi Leong and Quyet V. Do and Niklas Muennighoff and Tanrada Pansuwan and Ilham Firdausi Putra and Yan Xu and Ngee Chia Tai and Ayu Purwarianti and Sebastian Ruder and William Tjhi and Peerat Limkonchotiwat and Alham Fikri Aji and Sedrick Keh and Genta Indra Winata and Ruochen Zhang and Fajri Koto and Zheng-Xin Yong and Samuel Cahyawijaya},
    year={2024},
    eprint={2406.10118},
    journal={arXiv preprint arXiv: 2406.10118}
}

```