{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/WALinuxAgent/issues/763","repository_url":"https://api.github.com/repos/Azure/WALinuxAgent","labels_url":"https://api.github.com/repos/Azure/WALinuxAgent/issues/763/labels{/name}","comments_url":"https://api.github.com/repos/Azure/WALinuxAgent/issues/763/comments","events_url":"https://api.github.com/repos/Azure/WALinuxAgent/issues/763/events","html_url":"https://github.com/Azure/WALinuxAgent/issues/763","id":236376242,"number":763,"title":"walinuxagent Failed to run 'run-exthandlers':","user":{"login":"tungripe","id":10226184,"avatar_url":"https://avatars1.githubusercontent.com/u/10226184?v=3","gravatar_id":"","url":"https://api.github.com/users/tungripe","html_url":"https://github.com/tungripe","followers_url":"https://api.github.com/users/tungripe/followers","following_url":"https://api.github.com/users/tungripe/following{/other_user}","gists_url":"https://api.github.com/users/tungripe/gists{/gist_id}","starred_url":"https://api.github.com/users/tungripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tungripe/subscriptions","organizations_url":"https://api.github.com/users/tungripe/orgs","repos_url":"https://api.github.com/users/tungripe/repos","events_url":"https://api.github.com/users/tungripe/events{/privacy}","received_events_url":"https://api.github.com/users/tungripe/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-06-16T04:20:29Z","updated_at":"2017-06-16T16:48:58Z","closed_at":"2017-06-16T16:48:58Z","body":"Hi,\r\nWe have dozen servers running Ubuntu 14.04.5 LTS on Azure. We recently upgrade waagent to the latest versions. Our azure vm backup fails since upgrading waagent to version 2.2.12. The strange thing is azure VM backup  works on all of them except for 3 different VMs\r\n\r\nHere is the waagent version:\r\n```\r\n# waagent -version\r\nWALinuxAgent-2.2.12 running on ubuntu 14.04\r\nPython: 3.4.3\r\nGoal state agent: 2.2.12\r\n```\r\n\r\nIn the waagent log file, this error appears every 3 seconds: \r\n```\r\n# less /var/log/waagent.log\r\n2017/06/16 04:02:34.893131 ERROR Command: 'None                     # If \"None\", the system default version is used. cms -decrypt -in /var/lib/waagent/Certificates.p7m -inkey /var/lib/waagent/TransportPrivate.pem -recip /var/lib/waagent/TransportCert.pem | None                     # If \"None\", the system default version is used. pkcs12 -nodes -password pass: -out /var/lib/waagent/Certificates.pem'\r\n2017/06/16 04:02:34.905336 ERROR Return code: 127\r\n2017/06/16 04:02:34.907113 ERROR Result: /bin/sh: 1: None: not found\r\n\r\n2017/06/16 04:02:34.912356 ERROR Command: 'None                     # If \"None\", the system default version is used. rsa -in /var/lib/waagent/0.prv -pubout 2>/dev/null'\r\n2017/06/16 04:02:34.919377 ERROR Return code: 127\r\n2017/06/16 04:02:34.922083 ERROR Result: /bin/sh: 1: None: not found\r\n\r\n2017/06/16 04:02:34.926843 ERROR Command: 'None                     # If \"None\", the system default version is used. x509 -in /var/lib/waagent/1.crt -pubkey -noout'\r\n2017/06/16 04:02:34.932286 ERROR Return code: 127\r\n2017/06/16 04:02:34.934056 ERROR Result: /bin/sh: 1: None: not found\r\n\r\n2017/06/16 04:02:34.938140 ERROR Command: 'None                     # If \"None\", the system default version is used. x509 -in /var/lib/waagent/1.crt -fingerprint -noout'\r\n2017/06/16 04:02:34.944696 ERROR Return code: 127\r\n2017/06/16 04:02:34.946497 ERROR Result: /bin/sh: 1: None: not found\r\n\r\n2017/06/16 04:02:34.950498 ERROR Failed to run 'run-exthandlers': Traceback (most recent call last):\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/agent.py\", line 147, in main\r\n    agent.run_exthandlers()\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/agent.py\", line 117, in run_exthandlers\r\n    update_handler.run()\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/ga/update.py\", line 236, in run\r\n    get_monitor_handler().run()\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/ga/monitor.py\", line 96, in run\r\n    self.init_sysinfo()\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/ga/monitor.py\", line 121, in init_sysinfo\r\n    protocol = self.protocol_util.get_protocol()\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/protocol/util.py\", line 261, in get_protocol\r\n    self.protocol = self._detect_protocol(protocols)\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/protocol/util.py\", line 183, in _detect_protocol\r\n    return self._detect_wire_protocol()\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/protocol/util.py\", line 152, in _detect_wire_protocol\r\n    protocol.detect()\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/protocol/wire.py\", line 99, in detect\r\n    self.client.update_goal_state(forced=True)\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/protocol/wire.py\", line 729, in update_goal_state\r\n    self.update_certs(goal_state)\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/protocol/wire.py\", line 688, in update_certs\r\n    self.certs = Certificates(self, xml_text)\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/protocol/wire.py\", line 1147, in __init__\r\n    self.parse(xml_text)\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/protocol/wire.py\", line 1203, in parse\r\n    thumbprint = cryptutil.get_thumbprint_from_crt(tmp_file)\r\n  File \"/usr/lib/python3/dist-packages/azurelinuxagent/common/utils/cryptutil.py\", line 55, in get_thumbprint_from_crt\r\n    thumbprint = thumbprint.rstrip().split('=')[1].replace(':', '').upper()\r\nIndexError: list index out of range\r\n\r\n2017/06/16 04:02:35.662751 INFO Agent WALinuxAgent-2.2.12 launched with command 'python3 -u /usr/sbin/waagent -run-exthandlers' is successfully running\r\n2017/06/16 04:02:35.665392 INFO Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.2.12 launched with command 'python3 -u /usr/sbin/waagent -run-exthandlers' is successfully running\r\n2017/06/16 04:02:35.668840 WARNING Too many files under: /var/lib/waagent/events, removing oldest\r\n2017/06/16 04:02:35.670446 INFO Installed Agent WALinuxAgent-2.2.12 is the most current agent\r\n2017/06/16 04:02:35.829161 INFO Agent WALinuxAgent-2.2.12 is running as the goal state agent\r\n2017/06/16 04:02:35.831243 INFO Detect protocol endpoints\r\n2017/06/16 04:02:35.832317 INFO Clean protocol\r\n2017/06/16 04:02:35.833313 INFO WireServer endpoint is not found. Rerun dhcp handler\r\n2017/06/16 04:02:35.834788 INFO Test for route to 168.63.129.16\r\n2017/06/16 04:02:35.837044 INFO Route to 168.63.129.16 exists\r\n2017/06/16 04:02:35.837840 INFO Wire server endpoint:168.63.129.16\r\n2017/06/16 04:02:35.846969 INFO Fabric preferred wire protocol version:2015-04-05\r\n2017/06/16 04:02:35.848083 INFO Wire protocol version:2012-11-30\r\n2017/06/16 04:02:35.849052 WARNING Server preferred version:2015-04-05\r\n2017/06/16 04:02:35.856200 ERROR Command: 'None                     # If \"None\", the system default version is used. req -x509 -nodes -subj /CN=LinuxTransport -days 730 -newkey rsa:2048 -keyout /var/lib/waagent/TransportPrivate.pem -out /var/lib/waagent/TransportCert.pem'\r\n2017/06/16 04:02:35.857008 ERROR Return code: 127\r\n2017/06/16 04:02:35.857873 ERROR Result: /bin/sh: 1: None: not found\r\n```\r\n\r\nI restarted waagent services few times, and it did not help. I reinstalled waagent but it did not help either, the errors are still there and backups still failed.\r\n\r\nI also notice that there are waagent and walinuxagent  services on the machine. If I start one service, the other one stops automatically. Can you please clarify this for me?\r\n```\r\n# service waagent status\r\nwaagent stop/waiting\r\n# service walinuxagent status\r\nwalinuxagent start/running, process 11680\r\n```\r\n\r\nOur main issue is the VM backups are failing on our production server. So we would like to find out the root cause and bring it back to normal.\r\n\r\nLet us know if you need further details. Appreciate your helps.\r\n\r\n**Edit:** \r\nThe error that we got in azure console is: VM agent is unable to communicate with the Azure Backup Service.\r\n\r\nI also followed the instruction to troubleshoot here but did not work:\r\nhttps://docs.microsoft.com/en-us/azure/backup/backup-azure-troubleshoot-vm-backup-fails-snapshot-timeout\r\n\r\nCheers,\r\nTung"},"comment":{"url":"https://api.github.com/repos/Azure/WALinuxAgent/issues/comments/309076807","html_url":"https://github.com/Azure/WALinuxAgent/issues/763#issuecomment-309076807","issue_url":"https://api.github.com/repos/Azure/WALinuxAgent/issues/763","id":309076807,"user":{"login":"brendandixon","id":305205,"avatar_url":"https://avatars2.githubusercontent.com/u/305205?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandixon","html_url":"https://github.com/brendandixon","followers_url":"https://api.github.com/users/brendandixon/followers","following_url":"https://api.github.com/users/brendandixon/following{/other_user}","gists_url":"https://api.github.com/users/brendandixon/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandixon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandixon/subscriptions","organizations_url":"https://api.github.com/users/brendandixon/orgs","repos_url":"https://api.github.com/users/brendandixon/repos","events_url":"https://api.github.com/users/brendandixon/events{/privacy}","received_events_url":"https://api.github.com/users/brendandixon/received_events","type":"User","site_admin":false},"created_at":"2017-06-16T16:48:58Z","updated_at":"2017-06-16T16:48:58Z","body":"@tungripe @lovepreetkaul The agent uses an exceedingly simple parse for its configuration file: It only accepts <key>=<value> pairs without additional material on the line (e.g., no comments). Comments must live on their own lines.\r\n\r\nThe version of /etc/waagent.conf we ship with the agent (e.g., https://github.com/Azure/WALinuxAgent/blob/master/config/waagent.conf) honors this restriction."}}
{"actor":{"display_login":"brendandixon"}}