ts, and appends them to the overall configuration.Once the configuration is completed, _launch_spark_job comes into play. This takes the completed configuration and runs a Spark job on the given master URL with the specified deploy_mode. By default, this is achieved by creating and executing a spark-submit command.

Warning

In its first iteration, the pre-configuration with _io_configuration method is only effective when it is paired with an S3ArtifactStore (which has an authentication secret). When used with other artifact store flavors, you might be required to provide additional configuration through the submit_args.

Stack Component: KubernetesSparkStepOperator

The KubernetesSparkStepOperator is implemented by subclassing the base SparkStepOperator and uses the PipelineDockerImageBuilder class to build and push the required Docker images.

from typing import Optional

from zenml.integrations.spark.step_operators.spark_step_operator import (

SparkStepOperatorConfig

class KubernetesSparkStepOperatorConfig(SparkStepOperatorConfig):

"""Config for the Kubernetes Spark step operator."""

namespace: Optional[str] = None

service_account: Optional[str] = None

from pyspark.conf import SparkConf

from zenml.utils.pipeline_docker_image_builder import PipelineDockerImageBuilder

from zenml.integrations.spark.step_operators.spark_step_operator import (

SparkStepOperator

class KubernetesSparkStepOperator(SparkStepOperator):

"""Step operator which runs Steps with Spark on Kubernetes."""

def _backend_configuration(

self,

spark_config: SparkConf,

step_config: "StepConfiguration",

) -> None:

"""Configures Spark to run on Kubernetes."""

# Build and push the image

docker_image_builder = PipelineDockerImageBuilder()

image_name = docker_image_builder.build_and_push_docker_image(...)

# Adjust the spark configuration

spark_config.set("spark.kubernetes.container.image", image_name)

...

For Kubernetes, there are also some additional important configuration parameters: