Deploying Spark Applications on Kubernetes: A Practical Guide to spark-submit and Spark Operator

Combining Kubernetes and Apache Spark has become a powerful tool for processing big data; it offers several advantages over traditional approaches. Packaging Spark applications and dependencies in containers simplifies deployment and ensures consistency across environments. Let's quickly compare this approach with the traditional approach before we look at an example. 

Why use Kubernetes? What about YARN?

When it comes to sharing resources, Kubernetes is much more efficient than YARN. Dynamic allocation allows Spark to add executors to and remove executors from a given deployment based on the given workload, and Kubernetes reallocates resources between applications as they finish, which reduces idle time. 

Kubernetes's efficiency also reduces costs, because with Kubernetes, provisioning and de-provisioning is based on the given workload. Kubernetes allows for a cloud-agnostic approach, enabling organizations to build once, but deploy anywhere. 

What about Spark Operator?

Spark Operator introduces custom resource definitions (CRDs) that allow for the declarative management of Spark applications through Kubernetes Manifest. It uses Mutating Admission Webhook, which handles customizations for Spark driver and executor pods based on annotations on the pods added by the controller. Spark operators allow for centralized administration of all Spark-related workloads, which reduces operational overhead. 

But YARN still has benefits, right?

YARN provides built-in support for data locality, which can significantly improve the performance of data-intensive workloads. There are two levels of locality: Node-level locality and Rack-level locality. But while YARN offers some advantages in data locality for Hadoop-centric workloads, using the Spark Operator on Kubernetes provides a more flexible, cost-effective, and easy-to-manage platform for running Spark workloads in modern cloud-native environments.

Spark-submit or Spark Operator?

Both spark-submit and Spark Operator can be used to run Spark applications on Kubernetes, but they differ in their respective approaches and complexities. I have an example here that deploys using both methods. For the following reasons, I would use the Spark operator:

• Handles the entire life cycle of the Spark application, including clean up
• Manages Spark applications more easily than spark-submit 
• Declarative approach to configurations

Rate limiting in Kubernetes using Istio

Rate limiting is one of my favorite sections. I have implemented rate limiting as part of API Gateway and had multiple versions of it tailored to the needs of the downstream applications that were leveraging them in various ways. It's also another use case that is specific to microservices that can be implemented in Kubernetes. It can also be an add-on to existing rate limit implementations that can be based on customer or subscription.

First, let's talk about the security measure, and how its absence or improper implementation can lead to several security problems:

Vulnerability to Attacks

Denial of Service (DoS) and Distributed Denial of Service (DDoS) Attacks

Attackers can overwhelm an API with a flood of requests, exhausting server resources and making the API unresponsive or unavailable to legitimate users. It causes Resource Exhaustion, depleting server resources like CPU, memory, and network bandwidth, potentially causing system crashes or slowdowns.

Brute Force Attacks

Lack of rate limiting makes it easier for attackers to conduct brute force attacks on authentication endpoints, attempting numerous login attempts in rapid succession. Back in 2014, this was one of the reasons that Apple's authentication was compromised as hackers were able to hack into celebrity accounts. Apple has hardened the authentication service since then, and it's not a concern anymore with 2-factor authentication being the standard.

Credential Stuffing

This almost always happens by bots scanning most known URLs for easy access with common and stolen credentials. Without rate limits in place, this only exacerbates the problem.

Increased Attack Surface

Unlimited API access provides more opportunities for attackers to discover and exploit other vulnerabilities in the system.

Business Impact

This has a direct business impact with services having poor response because of resource exhaustion. It may also lead to excessive financial costs and service disruptions.

Implementing proper rate limiting policies is crucial for maintaining API security, ensuring fair resource allocation, and protecting against various forms of attacks and abuse. It's an essential component of a comprehensive API security strategy.

So how can you implement it , If you are already using Kubernetes and service mesh Its just a matter of adding it as part of your configuration . I have provided an example here -> Github 

Without Rate limit I am able to make 20 Request per second

                                      With Rate limit I am only able to make 10 Request per second 
                        

Istio

Istio, a popular service mesh for Kubernetes, provides built-in support for both rate limiting and circuit breaking. It can address almost all the security concerns described above . It may be possible by other service mesh as well , but I am going to use Istio for demonstration .

Protection Against Brute Force

Rate limit with Circuit Breaker help protect your services from being overwhelmed.

Fine-grained Control

You can set different limits and thresholds for different services or endpoints.

Improved Resilience

Circuit breakers prevent cascading failures, while rate limiting controls the flow of requests.

You can also do Rate limit , Load Balancing and routing on your Ingress and will not require a Service Mesh .  But this will be limited to the services accessed through ingress 

Containerization Github List  ->   Github Containers

AKS: Exploring Capabilities and Building Your First Sample Application

Azure offers a fantastic opportunity for developers to explore container services in the cloud with its $200 free credit for new accounts. 

This credit allows you to experiment, learn, and build applications without incurring immediate costs. Let's dive into how you can make the most of this offer and get hands-on experience with Azure Kubernetes Service (AKS).

What can you do with the Free Credit 

When you sign up for an Azure free account, you receive $200 credit to use within 30 days. This credit gives you access to a wide range of Azure services, including AKS. Here's how you can leverage this opportunity:

Explore Azure Portal 

Familiarize yourself with the Azure portal interface, where you'll manage your resources and services.

Create Resource Groups 

Organize your projects by creating resource groups, which act as containers for related resources.

Deploy AKS Clusters

This was my main focus to deploy a simple app in AKS and understand the Kubernetes integrations .

Azure's control plane management offers significant advantages, particularly for smaller organizations without dedicated DevOps teams.Azure handles the control plane

taking care of patches and upgrades, allowing you to focus solely on application deployment. This approach streamlines your workflow and reduces operational overhead.

While using Docker Hub is convenient for testing Azure services, it doesn't seem to be supported . I was able to use ACR ( Azure Container Registry ) , this would be integrated solution for Azure . However most organizations will prefer private container registries for security and compliance reasons.

Horizontal Pod Autoscaling (HPA)

HPA in cloud environments offers more comprehensive scaling capabilities.HPA manages not just pod scaling but also coordinates with cluster autoscaling for efficient node management.

Cloud platforms excel in scenarios requiring rapid and frequent scaling, offering flexibility that's harder to achieve with on-premises solutions.

Admission Controllers and Add-ons

While many Admission Controller use cases can be implemented as add-ons in Azure Kubernetes Service (AKS), it's important to consider:

Native Kubernetes Flexibility: Some operations that are straightforward in native Kubernetes might require additional configuration or external services in managed Kubernetes offerings.

Trade-offs: The convenience of managed services often comes at the cost of some flexibility. Evaluate whether the simplification aligns with your specific requirements and use cases.

My test stops here but you can do more by Deploying your ML workloads , Implement CICD ,etc

Sample Application Deployment in AKS

To help you get started, I've created a sample Java application that can be deployed in AKS.

Github

The application and detailed deployment instructions are available in my GitHub repository. Here's a high-level overview of the steps:

• Clone the Repository. -  Download the sample application code and Kubernetes manifests.
• Build the Docker Image - Package your Java application into a container image.
• Push to Container Registry -  Upload your image to Azure Container Registry (ACR).
• Create AKS Cluster -  Use the Azure portal or CLI to set up your Kubernetes cluster.
• Deploy the Application -  Apply the Kubernetes manifests to deploy your app to AKS.

Access the Application -  Configure ingress or use port forwarding to access your running application.

Containerization Github List  ->   Github Containers

Power of OSCAL in Streamlining Security Assessments for Containers

OSCAL (Open Security Controls Assessment Language) is a standardized framework developed by NIST (National Institute of Standards and Technology) in collaboration with FedRAMP to digitize and automate security authorization processes. It provides a common machine-readable language for expressing security controls, assessments, and related information.

OSCAL includes several interconnected models:

• System Security Plan (SSP): Documents how security controls are implemented in a system.
• Security Assessment Plan (SAP): Outlines the approach for assessing security controls.
• Security Assessment Report (SAR): Presents the results of security control assessments.
• Plan of Action and Milestones (POA&M): Tracks identified vulnerabilities and planned remediation activities

I have talked about SBOM here   . While OSCAL is not currently mandated, it offers significant value for companies doing business with the federal government:

• Alignment with FedRAMP : FedRAMP is actively involved in OSCAL development, indicating its potential future importance in federal compliance.
• Complementary to SBOM: Like Software Bill of Materials (SBOM) for container security, OSCAL provides standardized security assessment information.
• Automation Potential: OSCAL enables automation of security assessment processes, including vulnerability scans and SAR generation.

Vulnerability Scans for Containers can be done in ongoing basis and Security Assessment Report (SAR) can be automated easily . Here is a simple example of this implementation

Implementaion -> Github

Containerization Github List  -> Github Containers

Scaling Smarter: Understanding the Intricacies of HPA in Kubernetes

My 2 minutes on Kubernetes Horizontal Pod Autoscaler (HPA) and how it can be tricky to get it right 

Scaling Behavior

In my experience, scaling up generally works well because higher resource demands are quickly met. However, scaling down doesn't always occur as expected when resource usage decreases. This aspect often isn't included in performance testing criteria. Engineers and QA teams typically test for scale-up and performance requirements but may not thoroughly test elasticity.

Let's consider a simple java application but this can be applied to any app that has memory limits 

Are you using a Service Mesh ( This is fun ! ) 

When using services using sidecars, such as those injected by service meshes like Istio, there are several important considerations:

Resource calculation

HPA calculates resource usage across all containers in a pod, including sidecars. This means the total CPU and memory usage will include both your application container and the sidecar.

Adjust target utilization

Due to the additional resource consumption by sidecars, you may need to adjust your target CPU or memory utilization percentage. For example, if your original target was 80%, you might need to lower it to account for the sidecar's resource usage.

Metrics 

If you want to scale based on metrics from a specific container (e.g., your application container) rather than the entire pod, you can use container resource metrics. This allows you to ignore the sidecar's resource usage when making scaling decisions. So do you really need this then ?

When you need more fine-grained control, you can use custom metrics to scale your pods. This is especially useful if you want to scale based on application-specific metrics rather than just CPU or memory usage.

Handling High Memory Usage

Suppose you have configured the correct memory settings, but encounter high memory usage due to a specific rollout or an untested use case. The first response from an SRE is usually to increase the memory allocation. If you have an Xmx value specified, it also needs to be increased. I believe it's better not to set an Xmx value at all. The same principle applies to the Xms value when scaling down.

Its needles to say avoid hardcoding values for Xmx and Xms. Instead, drive these values from a configuration file, such as values.yaml. 

Testing it out

I created an application that consumes memory when API is called , jut for testing this out . Minikube environment with networking is not the perfect setup , but you can get this working in most of the high-end laptops 

Github

Containerization Github List  ->   Github Containers

Traffic Shaping Strategies Using Kubernetes and Istio

Service mesh in Kubernetes has evolved there are multiple types of service mesh for different use cases. Either you need a hybrid network with VM's and containers or high performance options using eBPF . 

This article I want to provide a quick how to on how to do traffic shaping with Istio. It is way easier to do this rather than applying this on application. Once you try out it you will be using it most often for Development and QA . 

Well Can't I do this in my Microservice ?

Yes , its common e.g Applying the new version of the API only particular type of customer or only for a certain list of customers . Here are the reasons you should consider Kubernetes approach 

Unified traffic management and Fine-grained control

Kubernetes provides more granular control over network traffic at the infrastructure level, allowing you to shape traffic based on various criteria such as source, destination, and protocol. This level of control may not be easily achievable through microservice design patterns alone.

Infrastructure-level management

Traffic shaping in Kubernetes operates at the cluster level, affecting all services running within it. This allows for consistent traffic management across the entire application ecosystem, regardless of individual microservice implementations.

Dynamic Scaling

If you are using in Production Kubernetes can automatically adjust traffic shaping rules based on the current state of the cluster, such as the number of pods or resource utilization. This dynamic approach is harder to implement solely through microservice design patterns.

How about in Development and QA ?

Good thing about this is also the fact that you can try this locally or in your Dev and QA to stimulate and test various cases , even if you are not using containers in production . Create different containers for different version of the API's deploy them using Kubernetes  . I have provided the step by step instruction below 

Step by Step

Github

Containerization Github List  ->   Github Containers

Confused about Multi architecture ?

Are you confused 😕 about Multi architecture in Docker  ?  Just think of Shoes 👞.... yeah shoes and how you use them .. 

• 
  You have a shoe closet with multiple shoe types (e.g., running shoes, hiking boots, dress shoes)
• Manual selection of the right shoe type for each activity
• Appropriate footwear for various terrains and occasions
• Central storage for all shoe types

Similarly Docker Multi-Architecture Images has 

• One image tag, multiple CPU architectures (e.g., x86, ARM, PowerPC)
• Automatic selection of the right architecture by the Docker runtime
• Same application runs across diverse hardware platforms
• Single management point for multiple architecture versions

Docker can still run anywhere right ?

Yes ,you can use the same shoe every where but not ideal right . Similarly traditional images are built for a specific architecture (e.g., x86-64 or ARM64). Multi-architecture images contain multiple versions of the same image, each built for a different architecture, bundled together under a single tag.

How does it do it ?

Multi-architecture images use manifest lists (also called fat manifests) to specify which image version should be used for each supported architecture. When a client pulls a multi-architecture image, Docker automatically selects the appropriate version based on the host system's architecture

Build is complicated ?

Yes kind of ,Multi-architecture images require separate builds for each target architecture, often using tools like Docker Buildx . Here is a example it not that difficult 

Github

Yeah , but like why .. What's the main advantage ?

Just like using the right shoe will give you the best performance and utilization . Multi-architecture images allow for more efficient use of resources by supporting diverse CPU architectures without maintaining separate images . This is especially realized in the cloud by using the correct architecture-specific image, you can achieve better resource utilization and potentially reduce costs, especially in cloud environments with diverse instance types 

Containerization Github List  ->   Github Containers