Spark

Executing individual steps on Spark

The spark integration brings two different step operators:

Step Operator: The SparkStepOperator serves as the base class for all the Spark-related step operators.

Step Operator: The KubernetesSparkStepOperator is responsible for launching ZenML steps as Spark applications with Kubernetes as a cluster manager.

Step Operators: SparkStepOperator

A summarized version of the implementation can be summarized in two parts. First, the configuration:

from typing import Optional, Dict, Any

from zenml.step_operators import BaseStepOperatorConfig

class SparkStepOperatorConfig(BaseStepOperatorConfig):

"""Spark step operator config.

Attributes:

master: is the master URL for the cluster. You might see different

schemes for different cluster managers which are supported by Spark

like Mesos, YARN, or Kubernetes. Within the context of this PR,

the implementation supports Kubernetes as a cluster manager.

deploy_mode: can either be 'cluster' (default) or 'client' and it

decides where the driver node of the application will run.

submit_kwargs: is the JSON string of a dict, which will be used

to define additional params if required (Spark has quite a

lot of different parameters, so including them, all in the step

operator was not implemented).

"""

master: str

deploy_mode: str = "cluster"

submit_kwargs: Optional[Dict[str, Any]] = None

and then the implementation:

from typing import List

from pyspark.conf import SparkConf

from zenml.step_operators import BaseStepOperator

class SparkStepOperator(BaseStepOperator):

"""Base class for all Spark-related step operators."""

def _resource_configuration(

self,

spark_config: SparkConf,

resource_configuration: "ResourceSettings",

) -> None:

"""Configures Spark to handle the resource configuration."""

def _backend_configuration(

self,

spark_config: SparkConf,

step_config: "StepConfiguration",

) -> None: