 as it provides up to 4 vCPUs and 16 GB of memory.Docker image for the Spark drivers and executors

When you want to run your steps on a Kubernetes cluster, Spark will require you to choose a base image for the driver and executor pods. Normally, for this purpose, you can either use one of the base images in Spark’s dockerhub or create an image using the docker-image-tool which will use your own Spark installation and build an image.

When using Spark in EKS, you need to use the latter and utilize the docker-image-tool. However, before the build process, you also need to download the following packages

hadoop-aws = 3.3.1

aws-java-sdk-bundle = 1.12.150

and put them in the jars folder within your Spark installation. Once that is set up, you can build the image as follows:

cd $SPARK_HOME # If this empty for you then you need to set the SPARK_HOME variable which points to your Spark installation

SPARK_IMAGE_TAG=<SPARK_IMAGE_TAG>

./bin/docker-image-tool.sh -t $SPARK_IMAGE_TAG -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile -u 0 build

BASE_IMAGE_NAME=spark-py:$SPARK_IMAGE_TAG

If you are working on an M1 Mac, you will need to build the image for the amd64 architecture, by using the prefix -X on the previous command. For example:

./bin/docker-image-tool.sh -X -t $SPARK_IMAGE_TAG -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile -u 0 build

Configuring RBAC

Additionally, you may need to create the several resources in Kubernetes in order to give Spark access to edit/manage your driver executor pods.

To do so, create a file called rbac.yaml with the following content:

apiVersion: v1

kind: Namespace

metadata:

name: spark-namespace

---

apiVersion: v1

kind: ServiceAccount

metadata:

name: spark-service-account

namespace: spark-namespace

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

name: spark-role

namespace: spark-namespace

subjects:

kind: ServiceAccount

name: spark-service-account

namespace: spark-namespace

roleRef:

kind: ClusterRole