title: "Dataproc Agent Startup Failed" with Initialization Script
body: <p>The below initialization script is intended to install Anaconda, some additional packages, and the genomic analysis toolkit Hail, on a Dataproc cluster. However, upon launching a cluster with it, I am receiving an error "Dataproc Agent Startup Failed." Oddly enough, copying/pasting this code into the terminal as root runs the entire script perfectly fine.</p>
  
  <pre><code>#!/bin/bash
  
  ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
  
  if [[ "${ROLE}" == 'Master' ]]; then
  
      # Download and install Anaconda in /home/anaconda2/
      wget -P /home/anaconda2/ https://repo.continuum.io/archive/Anaconda2-5.0.1-Linux-x86_64.sh
      bash /home/anaconda2/Anaconda2-5.0.1-Linux-x86_64.sh -b -f -p /home/anaconda2
      chmod -R 0777 /home/anaconda2/
  
      # Additional packages to install.
      /home/anaconda2/bin/pip install lxml
      /home/anaconda2/bin/pip install jupyter-spark
      /home/anaconda2/bin/pip install jgscm
  
      # Download current version of Hail for Spark 2.1.0. Currently set to use a default version.
      wget -P /home https://storage.googleapis.com/hail-common/distributions/0.1/Hail-0.1-5306854d2213-Spark-2.1.0.zip
  
      # Unzip hail and create resources.
      unzip /home/Hail-0.1-5306854d2213-Spark-2.1.0.zip -d /home
      zip -r /home/hail/python/pyhail.zip /home/hail/python/hail
  
      # Create Jupyter kernel spec file.
      mkdir /home/anaconda2/share/jupyter/kernels/
      mkdir /home/anaconda2/share/jupyter/kernels/hail
      echo '{"display_name": "Hail", "language": "python", "argv": ["/home/anaconda2/bin/python", "-m", "ipykernel", "-f", "{connection_file}"], "env": {"SPARK_HOME": "/usr/lib/spark/", "PYTHONHASHSEED": "0", "SPARK_CONF_DIR": "/home/hail/conf/", "PYTHONPATH": "/usr/lib/spark/python/:/usr/lib/spark/python/lib/py4j-0.10.3-src.zip:/home/hail/hail/python/pyhail.zip"}}' &gt; /home/anaconda2/share/jupyter/kernels/hail/kernel.json
  
      # Copy the Spark configuration files to a custom directory.
      mkdir /home/hail/conf
      cp /etc/spark/conf/spark-defaults.conf /home/hail/conf/spark-defaults.conf
      cp /etc/spark/conf/spark-env.sh /home/hail/conf/spark-env.sh
  
      # Modify custom Spark conf file to reference Hail jar and zip
      echo 'spark.files=/home/hail/lib/hail-all-spark.jar' &gt;&gt; /home/hail/conf/spark-defaults.conf
      echo 'spark.submit.pyFiles=/home/hail/lib/hail-all-spark.jar' &gt;&gt; /home/hail/conf/spark-defaults.conf
      echo 'spark.driver.extraClassPath=./hail-all-spark.jar' &gt;&gt; /home/hail/conf/spark-defaults.conf
      echo 'spark.executor.extraClassPath=./hail-all-spark.jar' &gt;&gt; /home/hail/conf/spark-defaults.conf
  
      # Add Spark variable designating Anaconda Python executable as the default on driver, in both custom and default conf files.
      echo 'PYSPARK_DRIVER_PYTHON=/home/anaconda2/bin/python' &gt;&gt; /home/hail/conf/spark-env.sh
      echo 'PYSPARK_DRIVER_PYTHON=/home/anaconda2/bin/python' &gt;&gt; /etc/spark/conf/spark-env.sh
  
      # Create Jupyter configuration file.
      mkdir /home/anaconda2/etc/jupyter/
      echo 'c.Application.log_level = "DEBUG"' &gt;&gt; /home/anaconda2/etc/jupyter/jupyter_notebook_config.py
      echo 'c.NotebookApp.ip = "*"' &gt;&gt; /home/anaconda2/etc/jupyter/jupyter_notebook_config.py
      echo 'c.NotebookApp.open_browser = False' &gt;&gt; /home/anaconda2/etc/jupyter/jupyter_notebook_config.py
      echo 'c.NotebookApp.port = 8123' &gt;&gt; /home/anaconda2/etc/jupyter/jupyter_notebook_config.py
      echo 'c.NotebookApp.token = ""' &gt;&gt; /home/anaconda2/etc/jupyter/jupyter_notebook_config.py
      echo 'c.NotebookApp.contents_manager_class = "jgscm.GoogleStorageContentManager"' &gt;&gt; /home/anaconda2/etc/jupyter/jupyter_notebook_config.py
  
      # Setup jupyter-spark extension.
      /home/anaconda2/bin/jupyter serverextension enable --user --py jupyter_spark
      /home/anaconda2/bin/jupyter nbextension install --user --py jupyter_spark
      /home/anaconda2/bin/jupyter nbextension enable --user --py jupyter_spark
      /home/anaconda2/bin/jupyter nbextension enable --user --py widgetsnbextension
  
      # Create the systemd service file for Jupyter notebook server process.
      echo '[Unit]' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'Description=Jupyter Notebook' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'After=hadoop-yarn-resourcemanager.service' &gt;&gt; /lib/systemd/system/jupyter.service
      echo '[Service]' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'Type=simple' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'User=root' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'Group=root' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'WorkingDirectory=/home/hail/' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'ExecStart=/home/anaconda2/bin/python /home/anaconda2/bin/jupyter notebook --allow-root' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'Restart=always' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'RestartSec=1' &gt;&gt; /lib/systemd/system/jupyter.service
      echo '[Install]' &gt;&gt; /lib/systemd/system/jupyter.service
      echo 'WantedBy=multi-user.target' &gt;&gt; /lib/systemd/system/jupyter.service
  
  
      # Add Jupyter service to autorun and start it.
      systemctl daemon-reload
      systemctl enable jupyter
      service jupyter start
  
      # Sleep for 30 seconds to allow Jupyter notebook server to start.
      sleep 30
  
  fi
  </code></pre>
  
  <p>Has anyone encountered this error previously? Normally, Google cloud would tell you where the error logs are, but there is no  additional information regarding this message.</p>
  
  <p>Exceptions in <code>/var/log/google-dataproc-agent.0.log</code>:</p>
  
  <pre><code># FIRST:
  WARNING: exception thrown while executing request
  java.net.SocketTimeoutException: connect timed out
          at java.net.PlainSocketImpl.socketConnect(Native Method)
          at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
          at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
          at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
          at java.net.Socket.connect(Socket.java:589)
          at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
          at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
          at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
          at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
          at sun.net.www.protocol.https.HttpsClient.&lt;init&gt;(HttpsClient.java:264)
          at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
          at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
          at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
          at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
          at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
          at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
          at com.google.cloud.hadoop.services.repackaged.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
          at com.google.cloud.hadoop.services.repackaged.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:965)
          at com.google.cloud.hadoop.services.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
          at com.google.cloud.hadoop.services.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
          at com.google.cloud.hadoop.services.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
          at com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getObject(GoogleCloudStorageImpl.java:1727)
          at com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getItemInfo(GoogleCloudStorageImpl.java:1618)
          at com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.open(GoogleCloudStorageImpl.java:550)
          at com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.open(GoogleCloudStorageImpl.java:531)
          at com.google.cloud.hadoop.services.agent.ClusterPropertiesModule.provideClusterProperties(ClusterPropertiesModule.java:52)
          at com.google.cloud.hadoop.services.agent.ClusterPropertiesModule$$FastClassByGuice$$52db3948.invoke(&lt;generated&gt;)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:402)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
          at com.google.cloud.hadoop.services.agent.ClusterPropertiesModule.provideFallbackProperties(ClusterPropertiesModule.java:89)
          at com.google.cloud.hadoop.services.agent.ClusterPropertiesModule$$FastClassByGuice$$52db3948.invoke(&lt;generated&gt;)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:776)
          at com.google.cloud.hadoop.services.agent.protocol.$Proxy17.getAgentApiClientWithEndpoint(Unknown Source)
          at com.google.cloud.hadoop.services.agent.protocol.AgentServiceClientFactory.get(AgentServiceClientFactory.java:60)
          at com.google.cloud.hadoop.services.agent.protocol.AgentServiceClientFactory.get(AgentServiceClientFactory.java:20)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
          at com.google.cloud.hadoop.services.repackaged.com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
          at com.google.cloud.hadoop.services.agent.AgentMain.boot(AgentMain.java:58)
          at com.google.cloud.hadoop.services.agent.AgentMain.main(AgentMain.java:46)
  
  
  
  # SECOND
  java.io.IOException: Cannot run program "/etc/google-dataproc/startup-scripts/dataproc-initialization-script-0": error=2, No such file or directory
          at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
          at com.google.cloud.hadoop.services.agent.util.NativeAsyncProcessWrapperFactory.startAndWrap(NativeAsyncProcessWrapperFactory.java:33)
          at com.google.cloud.hadoop.services.agent.util.NativeAsyncProcessWrapperFactory.startAndWrap(NativeAsyncProcessWrapperFactory.java:27)
          at com.google.cloud.hadoop.services.agent.BootstrapActionRunner.runScriptAndPipeOutputToGcs(BootstrapActionRunner.java:252)
          at com.google.cloud.hadoop.services.agent.BootstrapActionRunner.runSingleCustomInitializationScriptWithTimeout(BootstrapActionRunner.java:127)
          at com.google.cloud.hadoop.services.agent.BootstrapActionRunner.runCustomInitializationActions(BootstrapActionRunner.java:114)
          at com.google.cloud.hadoop.services.agent.AbstractAgentRunner.runCustomInitializationActionsIfFirstRun(AbstractAgentRunner.java:150)
          at com.google.cloud.hadoop.services.agent.MasterAgentRunner.initialize(MasterAgentRunner.java:165)
          at com.google.cloud.hadoop.services.agent.AbstractAgentRunner.start(AbstractAgentRunner.java:68)
          at com.google.cloud.hadoop.services.agent.MasterAgentRunner.start(MasterAgentRunner.java:36)
          at com.google.cloud.hadoop.services.agent.AgentMain.lambda$boot$0(AgentMain.java:63)
          at com.google.cloud.hadoop.services.agent.AgentStatusReporter.runWith(AgentStatusReporter.java:52)
          at com.google.cloud.hadoop.services.agent.AgentMain.boot(AgentMain.java:59)
          at com.google.cloud.hadoop.services.agent.AgentMain.main(AgentMain.java:46)
  Caused by: java.io.IOException: error=2, No such file or directory
          at java.lang.UNIXProcess.forkAndExec(Native Method)
          at java.lang.UNIXProcess.&lt;init&gt;(UNIXProcess.java:247)
          at java.lang.ProcessImpl.start(ProcessImpl.java:134)
          at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
          ... 13 more
  </code></pre>