{"action":"opened","issue":{"url":"https://api.github.com/repos/gettyimages/docker-spark/issues/24","repository_url":"https://api.github.com/repos/gettyimages/docker-spark","labels_url":"https://api.github.com/repos/gettyimages/docker-spark/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/gettyimages/docker-spark/issues/24/comments","events_url":"https://api.github.com/repos/gettyimages/docker-spark/issues/24/events","html_url":"https://github.com/gettyimages/docker-spark/issues/24","id":219097715,"number":24,"title":"spark-submit job.py can't use sparkConf to pass configuration parameters","user":{"login":"bizarte","id":11860384,"avatar_url":"https://avatars2.githubusercontent.com/u/11860384?v=3","gravatar_id":"","url":"https://api.github.com/users/bizarte","html_url":"https://github.com/bizarte","followers_url":"https://api.github.com/users/bizarte/followers","following_url":"https://api.github.com/users/bizarte/following{/other_user}","gists_url":"https://api.github.com/users/bizarte/gists{/gist_id}","starred_url":"https://api.github.com/users/bizarte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bizarte/subscriptions","organizations_url":"https://api.github.com/users/bizarte/orgs","repos_url":"https://api.github.com/users/bizarte/repos","events_url":"https://api.github.com/users/bizarte/events{/privacy}","received_events_url":"https://api.github.com/users/bizarte/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-04-03T23:51:38Z","updated_at":"2017-04-03T23:51:38Z","closed_at":null,"body":"I'm submitting my python job using the bin/spark-submit script. I want to configure the parameter in the python code through SparkConf & SparkContext class. I tried to set appName, master, spark.cores.max and spark.scheduler.mode in SparkConf object and pass it as an argument(conf) to the SparkContext. It turned out that the job was not sent to the master of the standalone cluster I set up(it just ran locally). Through the print statements below, it's clearly to see the SparkConf I passed to the SparkContext has all 4 configurations() but the conf variable from the SparkContext object doesn't have any of my updates despite of the default conf.\r\n\r\nFor other ways, I tried using conf/spark-defaults.conf or --properties-file my-spark.conf --master spark://spark-master:7077. It works.\r\nI also tried to set master using master parameter seperately  `sc = pyspark.SparkContext(master=\"spark://spark-master:7077\", conf=conf)` and it worked as well! \r\n\r\nSo it seems that only the conf parameter cannot be congested by the SparkContext correctly.\r\n\r\nYou can find the master got updated.\r\n```\r\nSparkContextConf: [('spark.app.name', 'test.py'), ('spark.rdd.compress', 'True'), ('spark.driver.port', '41147'), ('spark.app.id', 'app-20170403234627-0001'), ('spark.master', 'spark://spark-master:7077'), ('spark.serializer.objectStreamReset', '100'), ('spark.executor.id', 'driver'), ('spark.submit.deployMode', 'client'), ('spark.files', 'file:/mnt/scratch/yidi/docker-volume/test.py'), ('spark.driver.host', '10.0.0.5')]\r\n\r\n```\r\n\r\n\r\n\r\nThis is my python job code. \r\n\r\n```\r\nimport operator\r\nimport pyspark\r\n\r\ndef main():\r\n    '''Program entry point'''\r\n\r\n    import socket\r\n    print(socket.gethostbyname(socket.gethostname()))\r\n    print('!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!')\r\n    num_cores = 4\r\n    conf = pyspark.SparkConf()\r\n    conf.setAppName(\"word-count\").setMaster(\"spark://spark-master:7077\")#setExecutorEnv(\"CLASSPATH\", path)\r\n    conf.set(\"spark.scheduler.mode\", \"FAIR\")\r\n    conf.set(\"spark.cores.max\", num_cores)\r\n    print(\"Conf: {}\".format(conf.getAll()))\r\n    sc = pyspark.SparkContext(conf=conf)\r\n    print(\"SparkContextConf: {}\".format(sc.getConf().getAll()))\r\n\r\n    #Intialize a spark context\r\n    # with pyspark.SparkContext(conf=conf) as sc:\r\n    #     #Get a RDD containing lines from this script file  \r\n    #     lines = sc.textFile(\"/mnt/scratch/yidi/docker-volume/war_and_peace.txt\")\r\n    #     #Split each line into words and assign a frequency of 1 to each word\r\n    #     words = lines.flatMap(lambda line: line.split(\" \")).map(lambda word: (word, 1))\r\n    #     #count the frequency for words\r\n    #     counts = words.reduceByKey(operator.add)\r\n    #     #Sort the counts in descending order based on the word frequency\r\n    #     sorted_counts =  counts.sortBy(lambda x: x[1], False)\r\n    #     #Get an iterator over the counts to print a word and its frequency\r\n    #     for word,count in sorted_counts.toLocalIterator():\r\n    #         print(\"{} --> {}\".format(word.encode('utf-8'), count))\r\n    # print \"Spark conf: \", conf.getAll()\r\nif __name__ == \"__main__\":\r\n    main()\r\n```\r\n\r\nConsole output running the job above:\r\n```\r\nroot@bdd1ba99bf4f:/usr/spark-2.1.0# bin/spark-submit /mnt/scratch/yidi/docker-volume/test.py \r\n10.0.0.5\r\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\r\nConf: dict_items([('spark.scheduler.mode', 'FAIR'), ('spark.app.name', 'word-count'), ('spark.master', 'spark://spark-master:7077'), ('spark.cores.max', '4')])\r\n17/04/03 23:24:27 INFO spark.SparkContext: Running Spark version 2.1.0\r\n17/04/03 23:24:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n17/04/03 23:24:27 INFO spark.SecurityManager: Changing view acls to: root\r\n17/04/03 23:24:27 INFO spark.SecurityManager: Changing modify acls to: root\r\n17/04/03 23:24:27 INFO spark.SecurityManager: Changing view acls groups to: \r\n17/04/03 23:24:27 INFO spark.SecurityManager: Changing modify acls groups to: \r\n17/04/03 23:24:27 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()\r\n17/04/03 23:24:27 INFO util.Utils: Successfully started service 'sparkDriver' on port 38193.\r\n17/04/03 23:24:27 INFO spark.SparkEnv: Registering MapOutputTracker\r\n17/04/03 23:24:27 INFO spark.SparkEnv: Registering BlockManagerMaster\r\n17/04/03 23:24:27 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information\r\n17/04/03 23:24:27 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up\r\n17/04/03 23:24:27 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-7b545c86-9344-4f73-a3db-e891c562714d\r\n17/04/03 23:24:27 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB\r\n17/04/03 23:24:27 INFO spark.SparkEnv: Registering OutputCommitCoordinator\r\n17/04/03 23:24:27 INFO util.log: Logging initialized @1090ms\r\n17/04/03 23:24:27 INFO server.Server: jetty-9.2.z-SNAPSHOT\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3950ddbb{/jobs,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@286430d5{/jobs/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@404eb034{/jobs/job,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@79e3feec{/jobs/job/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4661596e{/stages,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4f8a3bef{/stages/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7a70fd3a{/stages/stage,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1c826806{/stages/stage/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@50e4e8d1{/stages/pool,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4e2fe461{/stages/pool/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@33cb59b3{/storage,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3c06c594{/storage/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4b5300a5{/storage/rdd,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7a6ef942{/storage/rdd/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1301217d{/environment,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@19217cec{/environment/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a241145{/executors,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@470d45aa{/executors/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5d9d8eff{/executors/threadDump,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4fc94fbc{/executors/threadDump/json,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@258dd139{/static,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@880f037{/,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@395b7dae{/api,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3cea7196{/jobs/job/kill,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@77257b2b{/stages/stage/kill,null,AVAILABLE}\r\n17/04/03 23:24:27 INFO server.ServerConnector: Started ServerConnector@788dbc45{HTTP/1.1}{0.0.0.0:4040}\r\n17/04/03 23:24:27 INFO server.Server: Started @1154ms\r\n17/04/03 23:24:27 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.\r\n17/04/03 23:24:27 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://localhost:4040\r\n17/04/03 23:24:27 INFO spark.SparkContext: Added file file:/mnt/scratch/yidi/docker-volume/test.py at file:/mnt/scratch/yidi/docker-volume/test.py with timestamp 1491261867734\r\n17/04/03 23:24:27 INFO util.Utils: Copying /mnt/scratch/yidi/docker-volume/test.py to /tmp/spark-5bb22b93-172d-40d6-8a37-6e3da5dc15a6/userFiles-ed0a6692-b96a-4b3d-be16-215a21cf836b/test.py\r\n17/04/03 23:24:27 INFO executor.Executor: Starting executor ID driver on host localhost\r\n17/04/03 23:24:27 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44223.\r\n17/04/03 23:24:27 INFO netty.NettyBlockTransferService: Server created on 10.0.0.5:44223\r\n17/04/03 23:24:27 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy\r\n17/04/03 23:24:27 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.0.5, 44223, None)\r\n17/04/03 23:24:27 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.0.0.5:44223 with 366.3 MB RAM, BlockManagerId(driver, 10.0.0.5, 44223, None)\r\n17/04/03 23:24:27 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.0.5, 44223, None)\r\n17/04/03 23:24:27 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.0.0.5, 44223, None)\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@c06133a{/metrics/json,null,AVAILABLE}\r\nSparkContextConf: [('spark.app.name', 'test.py'), ('spark.rdd.compress', 'True'), ('spark.serializer.objectStreamReset', '100'), ('spark.master', 'local[*]'), ('spark.executor.id', 'driver'), ('spark.submit.deployMode', 'client'), ('spark.files', 'file:/mnt/scratch/yidi/docker-volume/test.py'), ('spark.driver.host', '10.0.0.5'), ('spark.app.id', 'local-1491261867756'), ('spark.driver.port', '38193')]\r\n17/04/03 23:24:27 INFO spark.SparkContext: Invoking stop() from shutdown hook\r\n17/04/03 23:24:27 INFO server.ServerConnector: Stopped ServerConnector@788dbc45{HTTP/1.1}{0.0.0.0:4040}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@77257b2b{/stages/stage/kill,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3cea7196{/jobs/job/kill,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@395b7dae{/api,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@880f037{/,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@258dd139{/static,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4fc94fbc{/executors/threadDump/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5d9d8eff{/executors/threadDump,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@470d45aa{/executors/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4a241145{/executors,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@19217cec{/environment/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1301217d{/environment,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7a6ef942{/storage/rdd/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4b5300a5{/storage/rdd,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3c06c594{/storage/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@33cb59b3{/storage,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4e2fe461{/stages/pool/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@50e4e8d1{/stages/pool,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1c826806{/stages/stage/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7a70fd3a{/stages/stage,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4f8a3bef{/stages/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4661596e{/stages,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@79e3feec{/jobs/job/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@404eb034{/jobs/job,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@286430d5{/jobs/json,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3950ddbb{/jobs,null,UNAVAILABLE}\r\n17/04/03 23:24:27 INFO ui.SparkUI: Stopped Spark web UI at http://localhost:4040\r\n17/04/03 23:24:27 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!\r\n17/04/03 23:24:27 INFO memory.MemoryStore: MemoryStore cleared\r\n17/04/03 23:24:27 INFO storage.BlockManager: BlockManager stopped\r\n17/04/03 23:24:27 INFO storage.BlockManagerMaster: BlockManagerMaster stopped\r\n17/04/03 23:24:27 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!\r\n17/04/03 23:24:27 INFO spark.SparkContext: Successfully stopped SparkContext\r\n17/04/03 23:24:27 INFO util.ShutdownHookManager: Shutdown hook called\r\n17/04/03 23:24:27 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-5bb22b93-172d-40d6-8a37-6e3da5dc15a6/pyspark-50d75419-36b8-4dd2-a7c6-7d500c7c5bca\r\n17/04/03 23:24:27 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-5bb22b93-172d-40d6-8a37-6e3da5dc15a6\r\n\r\n```\r\n"}}
{"actor":{"display_login":"bizarte"}}