Dreem Open Datasets: Multi-Scored Sleep Datasets to compare Human and
  Automated sleep staging
  Sleep stage classification constitutes an important element of sleep disorder
diagnosis. It relies on the visual inspection of polysomnography records by
trained sleep technologists. Automated approaches have been designed to
alleviate this resource-intensive task. However, such approaches are usually
compared to a single human scorer annotation despite an inter-rater agreement
of about 85 % only. The present study introduces two publicly-available
datasets, DOD-H including 25 healthy volunteers and DOD-O including 55 patients
suffering from obstructive sleep apnea (OSA). Both datasets have been scored by
5 sleep technologists from different sleep centers. We developed a framework to
compare automated approaches to a consensus of multiple human scorers. Using
this framework, we benchmarked and compared the main literature approaches. We
also developed and benchmarked a new deep learning method, SimpleSleepNet,
inspired by current state-of-the-art. We demonstrated that many methods can
reach human-level performance on both datasets. SimpleSleepNet achieved an F1
of 89.9 % vs 86.8 % on average for human scorers on DOD-H, and an F1 of 88.3 %
vs 84.8 % on DOD-O. Our study highlights that using state-of-the-art automated
sleep staging outperforms human scorers performance for healthy volunteers and
patients suffering from OSA. Consideration could be made to use automated
approaches in the clinical setting.
