Deploying Spark Applications on Kubernetes: A Practical Guide to spark-submit and Spark Operator

Combining Kubernetes and Apache Spark has become a powerful tool for processing big data; it offers several advantages over traditional approaches. Packaging Spark applications and dependencies in containers simplifies deployment and ensures consistency across environments. Let's quickly compare this approach with the traditional approach before we look at an example. 

Why use Kubernetes? What about YARN?

When it comes to sharing resources, Kubernetes is much more efficient than YARN. Dynamic allocation allows Spark to add executors to and remove executors from a given deployment based on the given workload, and Kubernetes reallocates resources between applications as they finish, which reduces idle time. 

Kubernetes's efficiency also reduces costs, because with Kubernetes, provisioning and de-provisioning is based on the given workload. Kubernetes allows for a cloud-agnostic approach, enabling organizations to build once, but deploy anywhere. 

What about Spark Operator?

Spark Operator introduces custom resource definitions (CRDs) that allow for the declarative management of Spark applications through Kubernetes Manifest. It uses Mutating Admission Webhook, which handles customizations for Spark driver and executor pods based on annotations on the pods added by the controller. Spark operators allow for centralized administration of all Spark-related workloads, which reduces operational overhead. 

But YARN still has benefits, right?

YARN provides built-in support for data locality, which can significantly improve the performance of data-intensive workloads. There are two levels of locality: Node-level locality and Rack-level locality. But while YARN offers some advantages in data locality for Hadoop-centric workloads, using the Spark Operator on Kubernetes provides a more flexible, cost-effective, and easy-to-manage platform for running Spark workloads in modern cloud-native environments.

Spark-submit or Spark Operator?

Both spark-submit and Spark Operator can be used to run Spark applications on Kubernetes, but they differ in their respective approaches and complexities. I have an example here that deploys using both methods. For the following reasons, I would use the Spark operator:

• Handles the entire life cycle of the Spark application, including clean up
• Manages Spark applications more easily than spark-submit 
• Declarative approach to configurations

Rate limiting in Kubernetes using Istio

Rate limiting is one of my favorite sections. I have implemented rate limiting as part of API Gateway and had multiple versions of it tailored to the needs of the downstream applications that were leveraging them in various ways. It's also another use case that is specific to microservices that can be implemented in Kubernetes. It can also be an add-on to existing rate limit implementations that can be based on customer or subscription.

First, let's talk about the security measure, and how its absence or improper implementation can lead to several security problems:

Vulnerability to Attacks

Denial of Service (DoS) and Distributed Denial of Service (DDoS) Attacks

Attackers can overwhelm an API with a flood of requests, exhausting server resources and making the API unresponsive or unavailable to legitimate users. It causes Resource Exhaustion, depleting server resources like CPU, memory, and network bandwidth, potentially causing system crashes or slowdowns.

Brute Force Attacks

Lack of rate limiting makes it easier for attackers to conduct brute force attacks on authentication endpoints, attempting numerous login attempts in rapid succession. Back in 2014, this was one of the reasons that Apple's authentication was compromised as hackers were able to hack into celebrity accounts. Apple has hardened the authentication service since then, and it's not a concern anymore with 2-factor authentication being the standard.

Credential Stuffing

This almost always happens by bots scanning most known URLs for easy access with common and stolen credentials. Without rate limits in place, this only exacerbates the problem.

Increased Attack Surface

Unlimited API access provides more opportunities for attackers to discover and exploit other vulnerabilities in the system.

Business Impact

This has a direct business impact with services having poor response because of resource exhaustion. It may also lead to excessive financial costs and service disruptions.

Implementing proper rate limiting policies is crucial for maintaining API security, ensuring fair resource allocation, and protecting against various forms of attacks and abuse. It's an essential component of a comprehensive API security strategy.

So how can you implement it , If you are already using Kubernetes and service mesh Its just a matter of adding it as part of your configuration . I have provided an example here -> Github 

Without Rate limit I am able to make 20 Request per second

                                      With Rate limit I am only able to make 10 Request per second 
                        

Istio

Istio, a popular service mesh for Kubernetes, provides built-in support for both rate limiting and circuit breaking. It can address almost all the security concerns described above . It may be possible by other service mesh as well , but I am going to use Istio for demonstration .

Protection Against Brute Force

Rate limit with Circuit Breaker help protect your services from being overwhelmed.

Fine-grained Control

You can set different limits and thresholds for different services or endpoints.

Improved Resilience

Circuit breakers prevent cascading failures, while rate limiting controls the flow of requests.

You can also do Rate limit , Load Balancing and routing on your Ingress and will not require a Service Mesh .  But this will be limited to the services accessed through ingress 

Containerization Github List  ->   Github Containers

AKS: Exploring Capabilities and Building Your First Sample Application

Azure offers a fantastic opportunity for developers to explore container services in the cloud with its $200 free credit for new accounts. 

This credit allows you to experiment, learn, and build applications without incurring immediate costs. Let's dive into how you can make the most of this offer and get hands-on experience with Azure Kubernetes Service (AKS).

What can you do with the Free Credit 

When you sign up for an Azure free account, you receive $200 credit to use within 30 days. This credit gives you access to a wide range of Azure services, including AKS. Here's how you can leverage this opportunity:

Explore Azure Portal 

Familiarize yourself with the Azure portal interface, where you'll manage your resources and services.

Create Resource Groups 

Organize your projects by creating resource groups, which act as containers for related resources.

Deploy AKS Clusters

This was my main focus to deploy a simple app in AKS and understand the Kubernetes integrations .

Azure's control plane management offers significant advantages, particularly for smaller organizations without dedicated DevOps teams.Azure handles the control plane

taking care of patches and upgrades, allowing you to focus solely on application deployment. This approach streamlines your workflow and reduces operational overhead.

While using Docker Hub is convenient for testing Azure services, it doesn't seem to be supported . I was able to use ACR ( Azure Container Registry ) , this would be integrated solution for Azure . However most organizations will prefer private container registries for security and compliance reasons.

Horizontal Pod Autoscaling (HPA)

HPA in cloud environments offers more comprehensive scaling capabilities.HPA manages not just pod scaling but also coordinates with cluster autoscaling for efficient node management.

Cloud platforms excel in scenarios requiring rapid and frequent scaling, offering flexibility that's harder to achieve with on-premises solutions.

Admission Controllers and Add-ons

While many Admission Controller use cases can be implemented as add-ons in Azure Kubernetes Service (AKS), it's important to consider:

Native Kubernetes Flexibility: Some operations that are straightforward in native Kubernetes might require additional configuration or external services in managed Kubernetes offerings.

Trade-offs: The convenience of managed services often comes at the cost of some flexibility. Evaluate whether the simplification aligns with your specific requirements and use cases.

My test stops here but you can do more by Deploying your ML workloads , Implement CICD ,etc

Sample Application Deployment in AKS

To help you get started, I've created a sample Java application that can be deployed in AKS.

Github

The application and detailed deployment instructions are available in my GitHub repository. Here's a high-level overview of the steps:

• Clone the Repository. -  Download the sample application code and Kubernetes manifests.
• Build the Docker Image - Package your Java application into a container image.
• Push to Container Registry -  Upload your image to Azure Container Registry (ACR).
• Create AKS Cluster -  Use the Azure portal or CLI to set up your Kubernetes cluster.
• Deploy the Application -  Apply the Kubernetes manifests to deploy your app to AKS.

Access the Application -  Configure ingress or use port forwarding to access your running application.

Containerization Github List  ->   Github Containers

Integrating gRPC for Efficiency

Message brokers like Apache Kafka are excellent tools for building fault-tolerant architectures, particularly for implementing efficient back-end workflows. Kafka's design, which includes a queue system, can introduce a lag in service-to-service communication that cannot provide immediate responses. However, Kafka's ease of use and high reliability often make it preferable to Microservices.

When it comes to service-to-service communication within an application, many developers rely on REST APIs. This preference is likely due to REST's widespread familiarity and understanding, rather than optimal performance. For high-performance service calls within an application, gRPC is a more suitable option. It supports streaming payloads and offers significant performance benefits.

One challenge with using gRPC and Protocol Buffers is the need to define message structures and generate client and server stubs. Despite this overhead, the advantages of implementing gRPC are substantial, especially for performance-critical applications.

Microservices deployed in Kubernetes as pods, configuring gRPC involves changing properties to specify the service name instead of the host. This setup does not require a service mesh, and the initial configuration effort is rewarded with improved performance. For a demonstration of this setup in Kubernetes, you can refer to the following examples: a basic setup (Github main) and a setup with streaming support (Github Streaming Example).

Containerization Github List  ->   Github Containers

Ownership in the Kubernetes Ecosystem

 I was reviewing my presentation on Containers and Kubernetes with my colleague, and one important issue came up that is not widely talked about or discussed. It’s not a technical issue but an ownership issue. Don’t confuse it with the ownership of Kubernetes setup or security. It has to do with the ownership of features like Container Runtime and Service Mesh. This is not a technical article but something to think about as you roll out new features with Kubernetes.

Take Service Mesh, for example. It affects the way pods communicate with each other and offers additional capabilities with respect to metrics, monitoring, and security tooling. There are cases where this may not be the best for applications that look for high performance and don’t want to share the pod’s resources. Also, the application may be exporting the metrics as part of the framework.

Traffic Shaping and Canary Deployment also fall into a similar category. It’s usually the application teams that take ownership of these features. Developers, being fully aware of Kubernetes capabilities, may still choose to do this in microservice-based orchestration rather than relying on Kubernetes, as it may need coordination, approval, and guidance from DevOps.

Most Service Mesh implementations use sidecars. Sidecars consume resources and will affect the HPA. I talked about it here. The burden of successful HPA is not only with developers but also with DevOps, and most of these features will require developers to work closely with DevOps, something that is lacking in most organizations.

What do you think?

Containerization Github List  ->   Github Containers

Scaling Smarter: Understanding the Intricacies of HPA in Kubernetes

My 2 minutes on Kubernetes Horizontal Pod Autoscaler (HPA) and how it can be tricky to get it right 

Scaling Behavior

In my experience, scaling up generally works well because higher resource demands are quickly met. However, scaling down doesn't always occur as expected when resource usage decreases. This aspect often isn't included in performance testing criteria. Engineers and QA teams typically test for scale-up and performance requirements but may not thoroughly test elasticity.

Let's consider a simple java application but this can be applied to any app that has memory limits 

Are you using a Service Mesh ( This is fun ! ) 

When using services using sidecars, such as those injected by service meshes like Istio, there are several important considerations:

Resource calculation

HPA calculates resource usage across all containers in a pod, including sidecars. This means the total CPU and memory usage will include both your application container and the sidecar.

Adjust target utilization

Due to the additional resource consumption by sidecars, you may need to adjust your target CPU or memory utilization percentage. For example, if your original target was 80%, you might need to lower it to account for the sidecar's resource usage.

Metrics 

If you want to scale based on metrics from a specific container (e.g., your application container) rather than the entire pod, you can use container resource metrics. This allows you to ignore the sidecar's resource usage when making scaling decisions. So do you really need this then ?

When you need more fine-grained control, you can use custom metrics to scale your pods. This is especially useful if you want to scale based on application-specific metrics rather than just CPU or memory usage.

Handling High Memory Usage

Suppose you have configured the correct memory settings, but encounter high memory usage due to a specific rollout or an untested use case. The first response from an SRE is usually to increase the memory allocation. If you have an Xmx value specified, it also needs to be increased. I believe it's better not to set an Xmx value at all. The same principle applies to the Xms value when scaling down.

Its needles to say avoid hardcoding values for Xmx and Xms. Instead, drive these values from a configuration file, such as values.yaml. 

Testing it out

I created an application that consumes memory when API is called , jut for testing this out . Minikube environment with networking is not the perfect setup , but you can get this working in most of the high-end laptops 

Github

Containerization Github List  ->   Github Containers

eBPF: The Secret Weapon for Achieving Network Performance Breakthroughs

 I wrote an article about using eBPF based service-mesh in EKS ( eBPF with AWS ) The main benefit is in with high performance networking by shortening the network path . But How does it work ? How is it done otherwise ? Lets talk about it

How is it done traditionally ?

Service mesh uses Kube-Proxy that runs on every Node takes care of routing all the request from the Kubernetes nodes to the pod. It also takes care of service abstraction , and load balancing for all the pods that are part of the service .It uses Linux routing and updates the routing rules based on the pods deployed in the Node .When new service is created it updates the iptable rules to redirect traffic from the service Cluster IP to the backend pods. 

So this is good what's the issue ?

Its the network path and latency associated with it. Also the updates that needs to be done in the iptables whenever pods get deployed /removed . eBPF allows packet processing directly in the linux kernel eliminating the need for packet-to-packet NAT transformation with iptables. eBPF can also optimize the service network and facilitate direct pod-to-pod communication without intermediate NAT processing  

But what about TLS ?

Applications offload the network encryption requirements to the service mesh or the underlying network . Simply using transparent encryption ensures that the traffic is encrypted with in the cluster.Both IPSec and Wireguard are supported by Kubernetes networking by Cilium and Calico. There is also option to use TLS for initial certificate exchange and endpoint authentication , this way the application is used for identity rather than the node . 

Containerization Github List  ->   Github Containers

Traffic Shaping Strategies Using Kubernetes and Istio

Service mesh in Kubernetes has evolved there are multiple types of service mesh for different use cases. Either you need a hybrid network with VM's and containers or high performance options using eBPF . 

This article I want to provide a quick how to on how to do traffic shaping with Istio. It is way easier to do this rather than applying this on application. Once you try out it you will be using it most often for Development and QA . 

Well Can't I do this in my Microservice ?

Yes , its common e.g Applying the new version of the API only particular type of customer or only for a certain list of customers . Here are the reasons you should consider Kubernetes approach 

Unified traffic management and Fine-grained control

Kubernetes provides more granular control over network traffic at the infrastructure level, allowing you to shape traffic based on various criteria such as source, destination, and protocol. This level of control may not be easily achievable through microservice design patterns alone.

Infrastructure-level management

Traffic shaping in Kubernetes operates at the cluster level, affecting all services running within it. This allows for consistent traffic management across the entire application ecosystem, regardless of individual microservice implementations.

Dynamic Scaling

If you are using in Production Kubernetes can automatically adjust traffic shaping rules based on the current state of the cluster, such as the number of pods or resource utilization. This dynamic approach is harder to implement solely through microservice design patterns.

How about in Development and QA ?

Good thing about this is also the fact that you can try this locally or in your Dev and QA to stimulate and test various cases , even if you are not using containers in production . Create different containers for different version of the API's deploy them using Kubernetes  . I have provided the step by step instruction below 

Step by Step

Github

Containerization Github List  ->   Github Containers