pace: spark-namespace

roleRef:

kind: ClusterRolename: edit

apiGroup: rbac.authorization.k8s.io

---

And then execute the following command to create the resources:

aws eks --region=$REGION update-kubeconfig --name=$EKS_CLUSTER_NAME

kubectl create -f rbac.yaml

Lastly, note down the namespace and the name of the service account since you will need them when registering the stack component in the next step.

How to use it

To use the KubernetesSparkStepOperator, you need:

the ZenML spark integration. If you haven't installed it already, runCopyzenml integration install spark

Docker installed and running.

A remote artifact store as part of your stack.

A remote container registry as part of your stack.

A Kubernetes cluster deployed.

We can then register the step operator and use it in our active stack:

zenml step-operator register spark_step_operator \

--flavor=spark-kubernetes \

--master=k8s://$EKS_API_SERVER_ENDPOINT \

--namespace=<SPARK_KUBERNETES_NAMESPACE> \

--service_account=<SPARK_KUBERNETES_SERVICE_ACCOUNT>

# Register the stack

zenml stack register spark_stack \

o default \

s spark_step_operator \

a spark_artifact_store \

c spark_container_registry \

i local_builder \

--set

Once you added the step operator to your active stack, you can use it to execute individual steps of your pipeline by specifying it in the @step decorator as follows:

from zenml import step

@step(step_operator=<STEP_OPERATOR_NAME>)

def step_on_spark(...) -> ...:

"""Some step that should run with Spark on Kubernetes."""

...

After successfully running any step with a KubernetesSparkStepOperator, you should be able to see that a Spark driver pod was created in your cluster for each pipeline step when running kubectl get pods -n $KUBERNETES_NAMESPACE.

Instead of hardcoding a step operator name, you can also use the Client to dynamically use the step operator of your active stack:

from zenml.client import Client

step_operator = Client().active_stack.step_operator

@step(step_operator=step_operator.name)