{"action":"created","issue":{"url":"https://api.github.com/repos/cloudera/clusterdock/issues/27","repository_url":"https://api.github.com/repos/cloudera/clusterdock","labels_url":"https://api.github.com/repos/cloudera/clusterdock/issues/27/labels{/name}","comments_url":"https://api.github.com/repos/cloudera/clusterdock/issues/27/comments","events_url":"https://api.github.com/repos/cloudera/clusterdock/issues/27/events","html_url":"https://github.com/cloudera/clusterdock/issues/27","id":219703309,"number":27,"title":"Data node has DFS Used%: 100.00%","user":{"login":"Crandel","id":4005032,"avatar_url":"https://avatars0.githubusercontent.com/u/4005032?v=3","gravatar_id":"","url":"https://api.github.com/users/Crandel","html_url":"https://github.com/Crandel","followers_url":"https://api.github.com/users/Crandel/followers","following_url":"https://api.github.com/users/Crandel/following{/other_user}","gists_url":"https://api.github.com/users/Crandel/gists{/gist_id}","starred_url":"https://api.github.com/users/Crandel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Crandel/subscriptions","organizations_url":"https://api.github.com/users/Crandel/orgs","repos_url":"https://api.github.com/users/Crandel/repos","events_url":"https://api.github.com/users/Crandel/events{/privacy}","received_events_url":"https://api.github.com/users/Crandel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-04-05T20:21:18Z","updated_at":"2017-04-05T21:43:26Z","closed_at":null,"body":"I have Intel Core i7 and 20 Gb Ram on my laptop. \r\nWhen I run \r\n\r\n``` \r\nclusterdock_run ./bin/start_cluster -n hadoop cdh --include-service-type=HDFS,YARN,HIVE,HUE,OOZIE,SPARK --primary-node=node-1 --secondary-nodes=node-2\r\n```\r\n\r\nI always have this error. \r\n\r\n```\r\nINFO:clusterdock.cluster:Successfully started node-1.hadoop (IP address: 192.168.124.2).\r\nINFO:clusterdock.cluster:Successfully started node-2.hadoop (IP address: 192.168.124.3).\r\nINFO:clusterdock.cluster:Started cluster in 19.06 seconds.\r\nINFO:clusterdock.topologies.cdh.actions:Changing server_host to node-1.hadoop in /etc/cloudera-scm-agent/config.ini...\r\nINFO:clusterdock.topologies.cdh.actions:Restarting CM agents...\r\ncloudera-scm-agent is already stopped\r\ncloudera-scm-agent is already stopped\r\nStarting cloudera-scm-agent: [  OK  ]\r\nStarting cloudera-scm-agent: [  OK  ]\r\nINFO:clusterdock.topologies.cdh.actions:Waiting for Cloudera Manager server to come online...\r\nINFO:clusterdock.topologies.cdh.actions:Detected Cloudera Manager server after 44.05 seconds.\r\nINFO:clusterdock.topologies.cdh.actions:CM server is now accessible at http://cradlemanl.localdomain:32771\r\nINFO:clusterdock.topologies.cdh.cm:Detected CM API v13.\r\nINFO:clusterdock.topologies.cdh.cm_utils:Updating database configurations...\r\nINFO:clusterdock.topologies.cdh.cm:Updating NameNode references in Hive metastore...\r\nINFO:clusterdock.topologies.cdh.actions:Removing service ks_indexer from Cluster 1 (clusterdock)...\r\nINFO:clusterdock.topologies.cdh.actions:Removing service impala from Cluster 1 (clusterdock)...\r\nINFO:clusterdock.topologies.cdh.actions:Removing service hbase from Cluster 1 (clusterdock)...\r\nINFO:clusterdock.topologies.cdh.actions:Removing service solr from Cluster 1 (clusterdock)...\r\nINFO:clusterdock.topologies.cdh.actions:Removing service spark_on_yarn from Cluster 1 (clusterdock)...\r\nINFO:clusterdock.topologies.cdh.actions:Removing service zookeeper from Cluster 1 (clusterdock)...\r\nINFO:clusterdock.topologies.cdh.actions:Once its service starts, Hue server will be accessible at http://cradlemanl.localdomain:32770\r\nINFO:clusterdock.topologies.cdh.actions:Deploying client configuration...\r\nINFO:clusterdock.topologies.cdh.actions:Starting cluster...\r\nINFO:clusterdock.topologies.cdh.actions:Starting Cloudera Management service...\r\nINFO:clusterdock.topologies.cdh.cm:Beginning service health validation...\r\nTraceback (most recent call last):\r\n  File \"./bin/start_cluster\", line 70, in <module>\r\n    main()\r\n  File \"./bin/start_cluster\", line 63, in main\r\n    actions.start(args)\r\n  File \"/root/clusterdock/clusterdock/topologies/cdh/actions.py\", line 159, in start\r\n    deployment.validate_services_started()\r\n  File \"/root/clusterdock/clusterdock/topologies/cdh/cm.py\", line 91, in validate_services_started\r\n    \"(at fault: {1}).\").format(timeout_min, at_fault_services))\r\nException: Timed out after waiting 10 minutes for services to start (at fault: [[u'hdfs', \"Failed health checks: [u'HDFS_CANARY_HEALTH', u'HDFS_DATA_NODES_HEALTHY', u'HDFS_FREE_SPACE_REMAINING', u'HDFS_HA_NAMENODE_HEALTH']\"], [u'yarn', \"Failed health checks: [u'YARN_JOBHISTORY_HEALTH', u'YARN_NODE_MANAGERS_HEALTHY', u'YARN_RESOURCEMANAGERS_HEALTH']\"], [u'hive', \"Failed health checks: [u'HIVE_HIVEMETASTORES_HEALTHY', u'HIVE_HIVESERVER2S_HEALTHY']\"], [u'oozie', \"Failed health checks: [u'OOZIE_OOZIE_SERVERS_HEALTHY']\"], [u'hue', \"Failed health checks: [u'HUE_HUE_SERVERS_HEALTHY']\"], [u'mgmt', \"Failed health checks: [u'MGMT_ALERT_PUBLISHER_HEALTH', u'MGMT_EVENT_SERVER_HEALTH', u'MGMT_HOST_MONITOR_HEALTH', u'MGMT_SERVICE_MONITOR_HEALTH']\"]]).\r\n```\r\n\r\nI could connect to node-1 and node-2 and see all directories inside HDFS, but when I try to copy files from local to hdfs I always have this error\r\n\r\n```\r\ncopyFromLocal: File blabla._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.\r\n```\r\n\r\nThis command\r\n\r\n`sudo -u hdfs hdfs dfsadmin -report`\r\n\r\ngive me this result\r\n\r\n```Configured Capacity: 0 (0 B)\r\nPresent Capacity: 586440704 (559.27 MB)\r\nDFS Remaining: 0 (0 B)\r\nDFS Used: 586440704 (559.27 MB)\r\nDFS Used%: 100.00%\r\nUnder replicated blocks: 2\r\nBlocks with corrupt replicas: 0\r\nMissing blocks: 0\r\nMissing blocks (with replication factor 1): 0\r\n\r\n-------------------------------------------------\r\nLive datanodes (1):\r\n\r\nName: 192.168.124.3:50010 (node-2.hadoop)\r\nHostname: node-2.hadoop\r\nRack: /default\r\nDecommission Status : Normal\r\nConfigured Capacity: 0 (0 B)\r\nDFS Used: 586440704 (559.27 MB)\r\nNon DFS Used: 0 (0 B)\r\nDFS Remaining: 0 (0 B)\r\nDFS Used%: 100.00%\r\nDFS Remaining%: 0.00%\r\nConfigured Cache Capacity: 4294967296 (4 GB)\r\nCache Used: 0 (0 B)\r\nCache Remaining: 4294967296 (4 GB)\r\nCache Used%: 0.00%\r\nCache Remaining%: 100.00%\r\nXceivers: 2\r\nLast contact: Wed Apr 05 20:20:33 GMT 2017```"},"comment":{"url":"https://api.github.com/repos/cloudera/clusterdock/issues/comments/292006273","html_url":"https://github.com/cloudera/clusterdock/issues/27#issuecomment-292006273","issue_url":"https://api.github.com/repos/cloudera/clusterdock/issues/27","id":292006273,"user":{"login":"dimaspivak","id":6088908,"avatar_url":"https://avatars0.githubusercontent.com/u/6088908?v=3","gravatar_id":"","url":"https://api.github.com/users/dimaspivak","html_url":"https://github.com/dimaspivak","followers_url":"https://api.github.com/users/dimaspivak/followers","following_url":"https://api.github.com/users/dimaspivak/following{/other_user}","gists_url":"https://api.github.com/users/dimaspivak/gists{/gist_id}","starred_url":"https://api.github.com/users/dimaspivak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dimaspivak/subscriptions","organizations_url":"https://api.github.com/users/dimaspivak/orgs","repos_url":"https://api.github.com/users/dimaspivak/repos","events_url":"https://api.github.com/users/dimaspivak/events{/privacy}","received_events_url":"https://api.github.com/users/dimaspivak/received_events","type":"User","site_admin":false},"created_at":"2017-04-05T21:43:26Z","updated_at":"2017-04-05T21:43:26Z","body":"Run `clusterdock_run ./bin/housekeeping nuke` to clean up the existing container clusters on your machine (as well as the `/etc/hosts` file) and then try again."}}
{"actor":{"display_login":"dimaspivak"}}