The current version on Qubole is 3.5.3, and some packages, like PyMC3 and future XGBoost need higher versions. How do I upgrade? And would that affect other clusters' settings?

error message


There are a couple of options:

If you are using Environments, you could reach out to Qubole support and get updated to the latest version which comes with Python 3.7. However, this would affect the whole cluster and would not be limited to your job.

In case you are managing your packages on your own, then you could install miniconda with Python 3.7 via node bootstrap and then point your job to use that environment.

pushd /tmp
chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -bfp /media/ephemeral0/py_37_env

Then you can use Spark property spark.pyspark.python to point you job to use this miniconda environment. You might have to reinstall dependencies accordingly.

  • Thank you!! Yeah, I'm using Environments, I contacted Qubole Support for that. There is an option to upgrade to Python 3.7 from a drop-down menu in Edit button in Environments, but for some reason it's greyed out. – HT. Mar 12 '20 at 15:22

Your Answer

