nf,

step_config: "StepConfiguration",

) -> None:"""Configures Spark to handle backends like YARN, Mesos or Kubernetes."""

def _io_configuration(

self,

spark_config: SparkConf

) -> None:

"""Configures Spark to handle different input/output sources."""

def _additional_configuration(

self,

spark_config: SparkConf

) -> None:

"""Appends the user-defined configuration parameters."""

def _launch_spark_job(

self,

spark_config: SparkConf,

entrypoint_command: List[str]

) -> None:

"""Generates and executes a spark-submit command."""

def launch(

self,

info: "StepRunInfo",

entrypoint_command: List[str],

) -> None:

"""Launches the step on Spark."""

Under the base configuration, you will see the main configuration parameters:

master is the master URL for the cluster where Spark will run. You might see different schemes for this URL with varying cluster managers such as Mesos, YARN, or Kubernetes.

deploy_mode can either be 'cluster' (default) or 'client' and it decides where the driver node of the application will run.

submit_args is the JSON string of a dictionary, which will be used to define additional parameters if required ( Spark has a wide variety of parameters, thus including them all in a single class was deemed unnecessary.).

In addition to this configuration, the launch method of the step operator gets additional configuration parameters from the DockerSettings and ResourceSettings. As a result, the overall configuration happens in 4 base methods:

_resource_configuration translates the ZenML ResourceSettings object to Spark's own resource configuration.

_backend_configuration is responsible for cluster-manager-specific configuration.

_io_configuration is a critical method. Even though we have materializers, Spark might require additional packages and configuration to work with a specific filesystem. This method is used as an interface to provide this configuration.

_additional_configuration takes the submit_args, converts, and appends them to the overall configuration.