size_categories:10K<n<100K format:parquet library:datasets library:dask library:mlcroissant library:polars region:us
---
dataset_info:
- config_name: p=160
  features:
  - name: n_samples
    sequence: int32
  - name: noisy_covariances
    sequence:
      sequence:
        sequence: float32
  - name: clean_covariance
    sequence:
      sequence: float32
  - name: clean_precision
    sequence:
      sequence: float32
  splits:
  - name: train
    num_bytes: 4637160000
    num_examples: 5000
  - name: validation
    num_bytes: 463716000
    num_examples: 500
  - name: test
    num_bytes: 927432000
    num_examples: 1000
  download_size: 5367984948
  dataset_size: 6028308000
- config_name: p=40
  features:
  - name: n_samples
    sequence: int32
  - name: noisy_covariances
    sequence:
      sequence:
        sequence: float32
  - name: clean_covariance
    sequence:
      sequence: float32
  - name: clean_precision
    sequence:
      sequence: float32
  splits:
  - name: train
    num_bytes: 361240000
    num_examples: 5000
  - name: validation
    num_bytes: 36124000
    num_examples: 500
  - name: test
    num_bytes: 72248000
    num_examples: 1000
  download_size: 396306458
  dataset_size: 469612000
configs:
- config_name: p=160
  data_files:
  - split: train
    path: p=160/train-*
  - split: validation
    path: p=160/validation-*
  - split: test
    path: p=160/test-*
- config_name: p=40
  data_files:
  - split: train
    path: p=40/train-*
  - split: validation
    path: p=40/validation-*
  - split: test
    path: p=40/test-*
---

# Covariance Denoising Dataset

This dataset is designed for training and evaluating machine learning models for covariance denoising. It consists of synthetically generated data, including noisy and clean covariance matrices. You can access the dataset on Hugging Face: [https://huggingface.co/datasets/rhd008/covariance_denoising](https://huggingface.co/datasets/rhd008/covariance_denoising)

## Data Description

The dataset contains the following features for each sample:

* **n_samples:** An array indicating the number of samples used to generate the noisy covariance matrices.
* **noisy_covariances:** A stack of noisy covariance matrices generated with different numbers of samples.
* **clean_covariance:** The ground truth (clean) covariance matrix.
* **clean_precision:** The ground truth (clean) precision matrix, which is the inverse of the covariance matrix.

## Dataset Splits

The dataset is divided into three splits:

* **Train:** Used for training the model.
* **Validation:** Used for evaluating the model during training.
* **Test:** Used for final evaluation of the model.

## Data Generation

The data is generated using the following steps:

1. Generate sparse symmetric positive definite matrices using `make_sparse_spd_matrix` from `sklearn.datasets`.
2. Calculate the corresponding covariance and precision matrices.
3. Generate noisy covariance matrices by sampling from a multivariate normal distribution with the clean covariance matrix and varying numbers of samples.

## Use Cases

This dataset can be used for various tasks, including:

* **Covariance Denoising:** Training models to estimate the clean covariance matrix from noisy observations.
* **Precision Matrix Estimation:** Training models to estimate the clean precision matrix.
* **Graphical Model Inference:** Inferring the structure of graphical models from the estimated precision matrix.

## Citation

If you use this dataset in your research, please cite it as follows:

