ome additional important configuration parameters:namespace is the namespace under which the driver and executor pods will run.

service_account is the service account that will be used by various Spark components (to create and watch the pods).

Additionally, the _backend_configuration method is adjusted to handle the Kubernetes-specific configuration.

When to use it

You should use the Spark step operator:

when you are dealing with large amounts of data.

when you are designing a step that can benefit from distributed computing paradigms in terms of time and resources.

How to deploy it

To use the KubernetesSparkStepOperator you will need to setup a few things first:

Remote ZenML server: See the deployment guide for more information.

Kubernetes cluster: There are many ways to deploy a Kubernetes cluster using different cloud providers or on your custom infrastructure. For AWS, you can follow the Spark EKS Setup Guide below.

Spark EKS Setup Guide

The following guide will walk you through how to spin up and configure a Amazon Elastic Kubernetes Service with Spark on it:

EKS Kubernetes Cluster

Follow this guide to create an Amazon EKS cluster role.

Follow this guide to create an Amazon EC2 node role.

Go to the IAM website, and select Roles to edit both roles.

Attach the AmazonRDSFullAccess and AmazonS3FullAccess policies to both roles.

Go to the EKS website.

Make sure the correct region is selected on the top right.

Click on Add cluster and select Create.

Enter a name and select the cluster role for Cluster service role.

Keep the default values for the networking and logging steps and create the cluster.

Note down the cluster name and the API server endpoint:

EKS_CLUSTER_NAME=<EKS_CLUSTER_NAME>

EKS_API_SERVER_ENDPOINT=<API_SERVER_ENDPOINT>

After the cluster is created, select it and click on Add node group in the Compute tab.

Enter a name and select the node role.

For the instance type, we recommend t3a.xlarge, as it provides up to 4 vCPUs and 16 GB of memory.