title: GitLab Errno::ENOSPC: No space left on device @ dir_s_mkdir
body: <p>After running a very long job my GitLab (on docker container) became unavailable, I believe it is full of tmp files or metadata, or something else. When I tried to do a backup I realized that the disk was full and I got the error below.</p>
  <p>The way that I worked this around to keep working, was to double the disk size, now at least I can work.</p>
  <p>I ran this command inside the container <code>gitlab-ctl registry-garbage-collect -m</code> (<a href="https://docs.gitlab.com/ee/raketasks/cleanup.html" rel="nofollow noreferrer">https://docs.gitlab.com/ee/raketasks/cleanup.html</a>) and I could free up 3GB.</p>
  <p>I thought it was an issue with Linux inodes, but as you can see below it wasn't. The official GitLab documentation sometimes is excellent but sometimes is going in circles.</p>
  <p>I don't know what I supposed to delete to gain space, and overall I don't know what to do to avoid this issue next time, searched on Gitlab forums but somehow not clear tips or troubleshooting.</p>
  <p>I'm affraid that if I ran that job one more time, I will collapse the disk again.</p>
  <p>Any ideas on what to delete, what to add on <code>/etc/gitlab/gitlab.rb</code> to avoid this, or how to implement some housekeeping or alerts for this purpose?, thank you.</p>
  <pre><code>docker exec -t gitlab gitlab-backup create
  rake aborted!
  Errno::ENOSPC: No space left on device @ dir_s_mkdir - /tmp/prometheus-mmap20201110-4631-1eio26e
  /opt/gitlab/embedded/service/gitlab-rails/config/initializers/7_prometheus_metrics.rb:18:in `&lt;top (required)&gt;'
  /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `&lt;top (required)&gt;'
  /opt/gitlab/embedded/bin/bundle:23:in `load'
  /opt/gitlab/embedded/bin/bundle:23:in `&lt;main&gt;'
  Tasks: TOP =&gt; gitlab:backup:create =&gt; gitlab_environment =&gt; environment
  (See full trace by running task with --trace)
  </code></pre>
  <pre><code>[gitlab ~] df -h
  Filesystem      Size  Used Avail Use% Mounted on
  devtmpfs        7.8G   80K  7.8G   1% /dev
  tmpfs           7.8G     0  7.8G   0% /dev/shm
  /dev/nvme1n1p1   59G   56G  3.0G  96% /
  
  [gitlab ~] df -iH
  Filesystem     Inodes IUsed IFree IUse% Mounted on
  devtmpfs         2.1M   516  2.1M    1% /dev
  tmpfs            2.1M     1  2.1M    1% /dev/shm
  /dev/nvme1n1p1   4.0M  160k  3.8M    5% /
  </code></pre>
  <p><strong>Gitlab version</strong></p>
  <pre><code>root@gitlab:/# gitlab-rake gitlab:env:info
  
  System information
  System:
  Proxy:      no
  Current User:   git
  Using RVM:  no
  Ruby Version:   2.6.6p146
  Gem Version:    2.7.10
  Bundler Version:1.17.3
  Rake Version:   12.3.3
  Redis Version:  5.0.9
  Git Version:    2.28.0
  Sidekiq Version:5.2.9
  Go Version: unknown
  
  GitLab information
  Version:    13.4.4-ee
  Revision:   4196ccb4738
  Directory:  /opt/gitlab/embedded/service/gitlab-rails
  DB Adapter: PostgreSQL
  DB Version: 11.9
  URL:        https://
  HTTP Clone URL: https://
  SSH Clone URL:  git@
  Elasticsearch:  no
  Geo:        no
  Using LDAP: no
  Using Omniauth: yes
  Omniauth Providers:
  
  GitLab Shell
  Version:    13.7.0
  Repository storage paths:
  - default:  /var/opt/gitlab/git-data/repositories
  GitLab Shell path:      /opt/gitlab/embedded/service/gitlab-shell
  Git:        /opt/gitlab/embedded/bin/git
  </code></pre>