title: kubernetes: Unable to attach or mount volumes timed out waiting for the condition
body: <p>my question is about this:</p>
  
  <pre><code>Type     Reason                  Age        From                                  Message
  ----     ------                  ----       ----                                  -------
  Normal   Scheduled               &lt;unknown&gt;  default-scheduler                     Successfully assigned git/gitlab-gitaly-0 to node00b.workspace.domain.com
  Normal   SuccessfulAttachVolume  8m4s       attachdetach-controller               AttachVolume.Attach succeeded for volume "pvc-b5b268b6-4a99-4cda-9fd2-bf6b6c186053"
  Warning  FailedMount             6m21s      kubelet, node00b.workspace.domain.com  Unable to attach or mount volumes: unmounted volumes=[repo-data], unattached volumes=[custom-ca-certificates default-token-nsd89 gitaly-config init-gitaly-secrets gitaly-secrets repo-data etc-ssl-certs]: timed out waiting for the condition
  Warning  FailedMount             4m6s       kubelet, node00b.workspace.domain.com  Unable to attach or mount volumes: unmounted volumes=[repo-data], unattached volumes=[gitaly-secrets repo-data etc-ssl-certs custom-ca-certificates default-token-nsd89 gitaly-config init-gitaly-secrets]: timed out waiting for the condition
  Warning  FailedMount             112s       kubelet, node00b.workspace.domain.com  Unable to attach or mount volumes: unmounted volumes=[repo-data], unattached volumes=[default-token-nsd89 gitaly-config init-gitaly-secrets gitaly-secrets repo-data etc-ssl-certs custom-ca-certificates]: timed out waiting for the condition
  Normal   Pulled                  42s        kubelet, node00b.workspace.domain.com  Container image "registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3" already present on machine
  Normal   Created                 42s        kubelet, node00b.workspace.domain.com  Created container certificates
  Normal   Started                 42s        kubelet, node00b.workspace.domain.com  Started container certificates
  Normal   Pulling                 40s        kubelet, node00b.workspace.domain.com  Pulling image "busybox:latest"
  Normal   Pulled                  38s        kubelet, node00b.workspace.domain.com
  
  ---------------------------
  
  Successfully pulled image "busybox:latest"
  Normal   Created                 38s        kubelet, node00b.workspace.domain.com  Created container configure
  Normal   Started                 38s        kubelet, node00b.workspace.domain.com  Started container configure
  Normal   Pulled                  37s        kubelet, node00b.workspace.domain.com  Container image "registry.gitlab.com/gitlab-org/build/cng/gitaly:v12.9.3" already present on machine
  Normal   Created                 37s        kubelet, node00b.workspace.domain.com  Created container gitaly
  Normal   Started                 36s        kubelet, node00b.workspace.domain.com  Started container gitaly
  </code></pre>
  
  <pre><code>May 26 15:28:56 node00b.workspace.domain.com kubelet[64419]: E0526 15:28:56.862824   64419 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock" failed. No retries permitted until 2020-05-26 15:30:58.86276487 +0200 CEST m=+9429.926617333 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
  May 26 15:29:16 node00b.workspace.domain.com kubelet[64419]: E0526 15:29:16.411411   64419 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/csi/rbd.csi.ceph.com^0001-0024-16042995-ff80-4561-8d47-70d5a80ff4ea-0000000000000010-5ddd037e-74b1-11ea-9d12-f2eedd7cc4d2\"" failed. No retries permitted until 2020-05-26 15:29:16.911291367 +0200 CEST m=+9327.975143870 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-b5b268b6-4a99-4cda-9fd2-bf6b6c186053\" (UniqueName: \"kubernetes.io/csi/rbd.csi.ceph.com^0001-0024-16042995-ff80-4561-8d47-70d5a80ff4ea-0000000000000010-5ddd037e-74b1-11ea-9d12-f2eedd7cc4d2\") pod \"gitlab-gitaly-0\" (UID: \"4c4a25eb-361c-418f-a08e-814757e0e62c\") "
  May 26 15:29:16 node00b.workspace.domain.com kubelet[64419]: I0526 15:29:16.411538   64419 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-nsd89" (UniqueName: "kubernetes.io/secret/4c4a25eb-361c-418f-a08e-814757e0e62c-default-token-nsd89") pod "gitlab-gitaly-0" (UID: "4c4a25eb-361c-418f-a08e-814757e0e62c")
  May 26 15:29:16 node00b.workspace.domain.com kubelet[64419]: I0526 15:29:16.411695   64419 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "gitaly-secrets" (UniqueName: "kubernetes.io/empty-dir/4c4a25eb-361c-418f-a08e-814757e0e62c-gitaly-secrets") pod "gitlab-gitaly-0" (UID: "4c4a25eb-361c-418f-a08e-814757e0e62c")
  </code></pre>
  
  <p>Every time a pod wants to use a volume there is an undefined time until it succeeds.</p>
  
  <p>Then everything works as expected.</p>
  
  <p>What is that time?</p>
  
  <p>Who establishes it?</p>
  
  <p>Is there a way to parameterize it?</p>
  
  <p>Environment:</p>
  
  <pre><code>k8s v1.16
  docker v18.6.3
  csi-rbd v2.1.1
  </code></pre>