{"action":"opened","issue":{"url":"https://api.github.com/repos/locationtech-labs/geopyspark/issues/350","repository_url":"https://api.github.com/repos/locationtech-labs/geopyspark","labels_url":"https://api.github.com/repos/locationtech-labs/geopyspark/issues/350/labels{/name}","comments_url":"https://api.github.com/repos/locationtech-labs/geopyspark/issues/350/comments","events_url":"https://api.github.com/repos/locationtech-labs/geopyspark/issues/350/events","html_url":"https://github.com/locationtech-labs/geopyspark/issues/350","id":242363967,"number":350,"title":"Methods For Certain TiledRasterLayers Do Not Work","user":{"login":"jbouffard","id":11041248,"avatar_url":"https://avatars0.githubusercontent.com/u/11041248?v=3","gravatar_id":"","url":"https://api.github.com/users/jbouffard","html_url":"https://github.com/jbouffard","followers_url":"https://api.github.com/users/jbouffard/followers","following_url":"https://api.github.com/users/jbouffard/following{/other_user}","gists_url":"https://api.github.com/users/jbouffard/gists{/gist_id}","starred_url":"https://api.github.com/users/jbouffard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbouffard/subscriptions","organizations_url":"https://api.github.com/users/jbouffard/orgs","repos_url":"https://api.github.com/users/jbouffard/repos","events_url":"https://api.github.com/users/jbouffard/events{/privacy}","received_events_url":"https://api.github.com/users/jbouffard/received_events","type":"User","site_admin":false},"labels":[{"id":501436410,"url":"https://api.github.com/repos/locationtech-labs/geopyspark/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":{"login":"jbouffard","id":11041248,"avatar_url":"https://avatars0.githubusercontent.com/u/11041248?v=3","gravatar_id":"","url":"https://api.github.com/users/jbouffard","html_url":"https://github.com/jbouffard","followers_url":"https://api.github.com/users/jbouffard/followers","following_url":"https://api.github.com/users/jbouffard/following{/other_user}","gists_url":"https://api.github.com/users/jbouffard/gists{/gist_id}","starred_url":"https://api.github.com/users/jbouffard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbouffard/subscriptions","organizations_url":"https://api.github.com/users/jbouffard/orgs","repos_url":"https://api.github.com/users/jbouffard/repos","events_url":"https://api.github.com/users/jbouffard/events{/privacy}","received_events_url":"https://api.github.com/users/jbouffard/received_events","type":"User","site_admin":false},"assignees":[{"login":"jbouffard","id":11041248,"avatar_url":"https://avatars0.githubusercontent.com/u/11041248?v=3","gravatar_id":"","url":"https://api.github.com/users/jbouffard","html_url":"https://github.com/jbouffard","followers_url":"https://api.github.com/users/jbouffard/followers","following_url":"https://api.github.com/users/jbouffard/following{/other_user}","gists_url":"https://api.github.com/users/jbouffard/gists{/gist_id}","starred_url":"https://api.github.com/users/jbouffard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbouffard/subscriptions","organizations_url":"https://api.github.com/users/jbouffard/orgs","repos_url":"https://api.github.com/users/jbouffard/repos","events_url":"https://api.github.com/users/jbouffard/events{/privacy}","received_events_url":"https://api.github.com/users/jbouffard/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/locationtech-labs/geopyspark/milestones/2","html_url":"https://github.com/locationtech-labs/geopyspark/milestone/2","labels_url":"https://api.github.com/repos/locationtech-labs/geopyspark/milestones/2/labels","id":2543315,"number":2,"title":"0.2","description":"","creator":{"login":"jbouffard","id":11041248,"avatar_url":"https://avatars0.githubusercontent.com/u/11041248?v=3","gravatar_id":"","url":"https://api.github.com/users/jbouffard","html_url":"https://github.com/jbouffard","followers_url":"https://api.github.com/users/jbouffard/followers","following_url":"https://api.github.com/users/jbouffard/following{/other_user}","gists_url":"https://api.github.com/users/jbouffard/gists{/gist_id}","starred_url":"https://api.github.com/users/jbouffard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbouffard/subscriptions","organizations_url":"https://api.github.com/users/jbouffard/orgs","repos_url":"https://api.github.com/users/jbouffard/repos","events_url":"https://api.github.com/users/jbouffard/events{/privacy}","received_events_url":"https://api.github.com/users/jbouffard/received_events","type":"User","site_admin":false},"open_issues":12,"closed_issues":29,"state":"open","created_at":"2017-05-25T16:48:27Z","updated_at":"2017-07-12T12:32:10Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2017-07-12T12:32:10Z","updated_at":"2017-07-12T12:32:10Z","closed_at":null,"body":"There seems to be an issue when working with certain `TiledRasterLayer`s that causes them to fail when using certain methods. The problems that arise seem to only happen for `TiledRasterLayer`s which were made from Python `RDD`s.\r\n\r\nFor example, using this `TiledRasterLayer`:\r\n\r\n```python\r\ndata = np.array([[\r\n    [1.0, 1.0, 1.0, 1.0, 1.0],\r\n    [1.0, 1.0, 1.0, 1.0, 1.0],\r\n    [1.0, 1.0, 1.0, 1.0, 1.0],\r\n    [1.0, 1.0, 1.0, 1.0, 1.0],\r\n    [1.0, 1.0, 1.0, 1.0, 0.0]]], dtype=float)\r\n\r\nlayer = [(SpatialKey(row=0, col=0), Tile(cells=data, cell_type='FLOAT', no_data_value=-1.0)),\r\n         (SpatialKey(row=1, col=0), Tile(cells=data, cell_type='FLOAT', no_data_value=-1.0)),\r\n         (SpatialKey(row=0, col=1), Tile(cells=data, cell_type='FLOAT', no_data_value=-1.0)),\r\n         (SpatialKey(row=1, col=1), Tile(cells=data, cell_type='FLOAT', no_data_value=-1.0))]\r\n\r\nrdd = pysc.parallelize(layer)\r\n\r\nextent = Extent(0.0, 0.0, 33.0, 33.0)\r\nlayout = TileLayout(2, 2, 5, 5)\r\nbounds = Bounds(SpatialKey(col=0, row=0), SpatialKey(col=1, row=1))\r\nlayout_definition = LayoutDefinition(extent, layout)\r\n\r\nmetadata = Metadata(\r\n    bounds=bounds,\r\n    crs='+proj=longlat +datum=WGS84 +no_defs ',\r\n    cell_type='float32ud-1.0',\r\n    extent=extent,\r\n    layout_definition=layout_definition)\r\n\r\ntiled_layer = TiledRasterLayer.from_numpy_rdd(pysc=pysc, layer_type=LayerType.SPATIAL, numpy_rdd=rdd, metadata=metadata)\r\n```\r\n\r\nWill throw an error when trying to use `reproject`:\r\n\r\n```python\r\ntiled_layer.reproject(target_crs=3857)\r\n\r\nPy4JJavaError: An error occurred while calling o40.reproject.\r\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 4.0 failed 1 times, most recent failure: Lost task 1.0 in stage 4.0 (TID 7, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):\r\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 163, in main\r\n    func, profiler, deserializer, serializer = read_command(pickleSer, infile)\r\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 54, in read_command\r\n    command = serializer._read_with_length(file)\r\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 169, in _read_with_length\r\n    return self.loads(obj)\r\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 451, in loads\r\n    return pickle.loads(obj, encoding=encoding)\r\nImportError: No module named 'geopyspark.geotrellis.protobufserializer'\r\n\r\n\tat org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)\r\n\tat org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)\r\n\tat org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)\r\n\tat org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat geotrellis.spark.ContextRDD.compute(ContextRDD.scala:39)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\nThis does not happen though when the `TiledRasterLayer` is created from a `RasterLayer`:\r\n\r\n```python\r\nlayer = geotiff.get(pysc, \"spatial\", \"/home/jacob/econic.tif\")\r\nt_layer = layer.to_tiled_layer()\r\nt_layer.reproject(target_crs=3857) \r\n```\r\n\r\nFrom the above error message, this appears to be a serialization error of some sort."}}
{"actor":{"display_login":"jbouffard"}}