title: Hadoop node taking a long time to decommission
body: <p>EDIT: I finally figured out what the issue was. Some files had very high replication factor set, and I was reducing my cluster to 2 nodes. Once I reduced my replication factor on those files, the decommissioning successfully ended quickly.</p>
  
  <p>I've added the node to be decommissioned in the <code>dfs.hosts.exclude</code> and <code>mapred.hosts.exclude</code> files, and executed this command:</p>
  
  <p><code>bin/hadoop dfsadmin -refreshNodes</code>.</p>
  
  <p>In the NameNode UI, I see this node under <code>Decommissioning Nodes</code>, but it's taking too long time, and I don't have much data on the node being decommissioned.</p>
  
  <p>Does it always take a very long time to decommision nodes or is there some place I should be looking? I'm not sure what is exactly going on.</p>
  
  <p>I don't see any corrupted blocks also on this node:</p>
  
  <pre><code>$ ./hadoop/bin/hadoop fsck -blocks /
   Total size:    157254687 B
   Total dirs:    201
   Total files:   189 (Files currently being written: 6)
   Total blocks (validated):      140 (avg. block size 1123247 B) (Total open file blocks (not validated): 1)
   Minimally replicated blocks:   140 (100.0 %)
   Over-replicated blocks:        6 (4.285714 %)
   Under-replicated blocks:       12 (8.571428 %)
   Mis-replicated blocks:         0 (0.0 %)
   Default replication factor:    2
   Average block replication:     1.9714285
   Corrupt blocks:                0
   Missing replicas:              88 (31.884058 %)
   Number of data-nodes:          3
   Number of racks:               1
  FSCK ended at Mon Jul 22 14:42:45 IST 2013 in 33 milliseconds
  
  
  The filesystem under path '/' is HEALTHY
  
  $ ./hadoop/bin/hadoop dfsadmin -report
  Configured Capacity: 25357025280 (23.62 GB)
  Present Capacity: 19756299789 (18.4 GB)
  DFS Remaining: 19366707200 (18.04 GB)
  DFS Used: 389592589 (371.54 MB)
  DFS Used%: 1.97%
  Under replicated blocks: 14
  Blocks with corrupt replicas: 0
  Missing blocks: 0
  
  -------------------------------------------------
  Datanodes available: 3 (3 total, 0 dead)
  
  Name: 10.40.11.107:50010
  Decommission Status : Decommission in progress
  Configured Capacity: 8452341760 (7.87 GB)
  DFS Used: 54947840 (52.4 MB)
  Non DFS Used: 1786830848 (1.66 GB)
  DFS Remaining: 6610563072(6.16 GB)
  DFS Used%: 0.65%
  DFS Remaining%: 78.21%
  Last contact: Mon Jul 22 14:29:37 IST 2013
  
  
  Name: 10.40.11.106:50010
  Decommission Status : Normal
  Configured Capacity: 8452341760 (7.87 GB)
  DFS Used: 167412428 (159.66 MB)
  Non DFS Used: 1953377588 (1.82 GB)
  DFS Remaining: 6331551744(5.9 GB)
  DFS Used%: 1.98%
  DFS Remaining%: 74.91%
  Last contact: Mon Jul 22 14:29:37 IST 2013
  
  
  Name: 10.40.11.108:50010
  Decommission Status : Normal
  Configured Capacity: 8452341760 (7.87 GB)
  DFS Used: 167232321 (159.49 MB)
  Non DFS Used: 1860517055 (1.73 GB)
  DFS Remaining: 6424592384(5.98 GB)
  DFS Used%: 1.98%
  DFS Remaining%: 76.01%
  Last contact: Mon Jul 22 14:29:38 IST 2013
  </code></pre>