Deploying Spark Applications on Kubernetes: A Practical Guide to spark-submit and Spark Operator

Combining Kubernetes and Apache Spark has become a powerful tool for processing big data; it offers several advantages over traditional approaches. Packaging Spark applications and dependencies in containers simplifies deployment and ensures consistency across environments. Let's quickly compare this approach with the traditional approach before we look at an example. 

Why use Kubernetes? What about YARN?

When it comes to sharing resources, Kubernetes is much more efficient than YARN. Dynamic allocation allows Spark to add executors to and remove executors from a given deployment based on the given workload, and Kubernetes reallocates resources between applications as they finish, which reduces idle time. 

Kubernetes's efficiency also reduces costs, because with Kubernetes, provisioning and de-provisioning is based on the given workload. Kubernetes allows for a cloud-agnostic approach, enabling organizations to build once, but deploy anywhere. 

What about Spark Operator?

Spark Operator introduces custom resource definitions (CRDs) that allow for the declarative management of Spark applications through Kubernetes Manifest. It uses Mutating Admission Webhook, which handles customizations for Spark driver and executor pods based on annotations on the pods added by the controller. Spark operators allow for centralized administration of all Spark-related workloads, which reduces operational overhead. 

But YARN still has benefits, right?

YARN provides built-in support for data locality, which can significantly improve the performance of data-intensive workloads. There are two levels of locality: Node-level locality and Rack-level locality. But while YARN offers some advantages in data locality for Hadoop-centric workloads, using the Spark Operator on Kubernetes provides a more flexible, cost-effective, and easy-to-manage platform for running Spark workloads in modern cloud-native environments.

Spark-submit or Spark Operator?

Both spark-submit and Spark Operator can be used to run Spark applications on Kubernetes, but they differ in their respective approaches and complexities. I have an example here that deploys using both methods. For the following reasons, I would use the Spark operator:

• Handles the entire life cycle of the Spark application, including clean up
• Manages Spark applications more easily than spark-submit 
• Declarative approach to configurations