{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/922","repository_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/922/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/922/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/922/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/922","id":202209074,"number":922,"title":"An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD","user":{"login":"aginpatrick","id":5891272,"avatar_url":"https://avatars1.githubusercontent.com/u/5891272?v=4","gravatar_id":"","url":"https://api.github.com/users/aginpatrick","html_url":"https://github.com/aginpatrick","followers_url":"https://api.github.com/users/aginpatrick/followers","following_url":"https://api.github.com/users/aginpatrick/following{/other_user}","gists_url":"https://api.github.com/users/aginpatrick/gists{/gist_id}","starred_url":"https://api.github.com/users/aginpatrick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aginpatrick/subscriptions","organizations_url":"https://api.github.com/users/aginpatrick/orgs","repos_url":"https://api.github.com/users/aginpatrick/repos","events_url":"https://api.github.com/users/aginpatrick/events{/privacy}","received_events_url":"https://api.github.com/users/aginpatrick/received_events","type":"User","site_admin":false},"labels":[{"id":119737425,"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/:Spark","name":":Spark","color":"009800","default":false},{"id":207945587,"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/feedback-requested","name":"feedback-requested","color":"ededed","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-01-20T18:25:11Z","updated_at":"2017-09-12T16:00:11Z","closed_at":"2017-09-12T16:00:11Z","author_association":"NONE","body":"I run pyspark with:\r\n\r\n    ] ./bin/pyspark --jars jars/elasticsearch-hadoop-2.2.0.jar \r\n\r\n    Python 2.7.9 (default, Apr  2 2015, 15:33:21) \r\n    [GCC 4.9.2] on linux2\r\n    Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\r\n    Setting default log level to \"WARN\".\r\n    To adjust logging level use sc.setLogLevel(newLevel).\r\n    17/01/20 12:54:52 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n    Welcome to\r\n          ____              __\r\n         / __/__  ___ _____/ /__\r\n        _\\ \\/ _ \\/ _ `/ __/  '_/\r\n       /__ / .__/\\_,_/_/ /_/\\_\\   version 2.0.1\r\n          /_/\r\n\r\n    Using Python version 2.7.9 (default, Apr  2 2015 15:33:21)\r\n    SparkSession available as 'spark'.\r\n\r\nWhen I try to define an elasticsearch rdd (ES 2.4.3), I get the following error:\r\n\r\n    >>> es_read_conf = { \"es.resource\" : \"test/docs\" , \"es.nodes\" : \"xx.xx.xx.xx\",\"es.port\" : \"9200\", \"es.net.http.auth.user\": \"myusername\", \"es.net.http.auth.pass\": \"mypassword\"}\r\n    >>> es_rdd = sc.newAPIHadoopRDD(inputFormatClass=\"org.elasticsearch.hadoop.mr.EsInputFormat\",keyClass=\"org.apache.hadoop.io.NullWritable\", valueClass=\"org.elasticsearch.hadoop.mr.LinkedMapWritable\", conf=es_read_conf)\r\n\r\n    Traceback (most recent call last):\r\n      File \"<stdin>\", line 1, in <module>\r\n      File \"/home/patrick/spark-2.0.1-bin-hadoop2.7/python/pyspark/context.py\", line 646, in newAPIHadoopRDD\r\n        jconf, batchSize)\r\n      File \"/home/patrick/spark-2.0.1-bin-hadoop2.7/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py\", line 1133, in __call__\r\n      File \"/home/patrick/spark-2.0.1-bin-hadoop2.7/python/pyspark/sql/utils.py\", line 63, in deco\r\n    return f(*a, **kw)\r\n      File \"/home/patrick/spark-2.0.1-bin-hadoop2.7/python/lib/py4j-0.10.3-src.zip/py4j/protocol.py\", line 319, in get_return_value\r\n    py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD.\r\n    : org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [GET] on [_nodes/http] failed; server[xx.xx.xx.yy:9201] returned [502|Bad Gateway:]\r\n    (...other messages)\r\n\r\nNotice the server address in the error message, it's not the same as the host that I specified in the es.nodes key in es_read_conf dict. In this dict, I defined the host as the master node address (xx.xx.xx.**xx**) and the error refers to one of the data node (xx.xx.xx.**yy**). Actually, when I run the command again, it can raise the same error but with another data node (xx.xx.xx.**zz**). If I run the command multiple times, sometimes no error occurs (I suppose that it's when command is sent to master node only). \r\n\r\n    >>> es_rdd = sc.newAPIHadoopRDD(inputFormatClass=\"org.elasticsearch.hadoop.mr.EsInputFormat\",keyClass=\"org.apache.hadoop.io.NullWritable\", valueClass=\"org.elasticsearch.hadoop.mr.LinkedMapWritable\", conf=es_read_conf)\r\n    17/01/20 13:19:36 WARN EsInputFormat: Cannot determine task id...\r\n    >>> for field in es_rdd.first()[1]:\r\n                print field\r\n\r\n    17/01/20 13:21:03 WARN EsInputFormat: Cannot determine task id...\r\n    signaletic\r\n    sourceinfo\r\n    document\r\n\r\nAny hint will be very appreciated.\r\nPatrick\r\n\r\n    "},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/328899155","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/922#issuecomment-328899155","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/922","id":328899155,"user":{"login":"jbaiera","id":875779,"avatar_url":"https://avatars1.githubusercontent.com/u/875779?v=4","gravatar_id":"","url":"https://api.github.com/users/jbaiera","html_url":"https://github.com/jbaiera","followers_url":"https://api.github.com/users/jbaiera/followers","following_url":"https://api.github.com/users/jbaiera/following{/other_user}","gists_url":"https://api.github.com/users/jbaiera/gists{/gist_id}","starred_url":"https://api.github.com/users/jbaiera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbaiera/subscriptions","organizations_url":"https://api.github.com/users/jbaiera/orgs","repos_url":"https://api.github.com/users/jbaiera/repos","events_url":"https://api.github.com/users/jbaiera/events{/privacy}","received_events_url":"https://api.github.com/users/jbaiera/received_events","type":"User","site_admin":false},"created_at":"2017-09-12T16:00:11Z","updated_at":"2017-09-12T16:00:11Z","author_association":"CONTRIBUTOR","body":"Closing this as the conversation has trailed off. We can always re-open this if new information comes to light."}}
{"actor":{"display_login":"jbaiera"}}