Scaling Smarter: Understanding the Intricacies of HPA in Kubernetes

My 2 minutes on Kubernetes Horizontal Pod Autoscaler (HPA) and how it can be tricky to get it right 

Scaling Behavior

In my experience, scaling up generally works well because higher resource demands are quickly met. However, scaling down doesn't always occur as expected when resource usage decreases. This aspect often isn't included in performance testing criteria. Engineers and QA teams typically test for scale-up and performance requirements but may not thoroughly test elasticity.

Let's consider a simple java application but this can be applied to any app that has memory limits 

Are you using a Service Mesh ( This is fun ! ) 

When using services using sidecars, such as those injected by service meshes like Istio, there are several important considerations:

Resource calculation

HPA calculates resource usage across all containers in a pod, including sidecars. This means the total CPU and memory usage will include both your application container and the sidecar.

Adjust target utilization

Due to the additional resource consumption by sidecars, you may need to adjust your target CPU or memory utilization percentage. For example, if your original target was 80%, you might need to lower it to account for the sidecar's resource usage.

Metrics 

If you want to scale based on metrics from a specific container (e.g., your application container) rather than the entire pod, you can use container resource metrics. This allows you to ignore the sidecar's resource usage when making scaling decisions. So do you really need this then ?

When you need more fine-grained control, you can use custom metrics to scale your pods. This is especially useful if you want to scale based on application-specific metrics rather than just CPU or memory usage.

Handling High Memory Usage

Suppose you have configured the correct memory settings, but encounter high memory usage due to a specific rollout or an untested use case. The first response from an SRE is usually to increase the memory allocation. If you have an Xmx value specified, it also needs to be increased. I believe it's better not to set an Xmx value at all. The same principle applies to the Xms value when scaling down.

Its needles to say avoid hardcoding values for Xmx and Xms. Instead, drive these values from a configuration file, such as values.yaml. 

Testing it out

I created an application that consumes memory when API is called , jut for testing this out . Minikube environment with networking is not the perfect setup , but you can get this working in most of the high-end laptops 

Github

Containerization Github List  ->   Github Containers

eBPF: The Secret Weapon for Achieving Network Performance Breakthroughs

 I wrote an article about using eBPF based service-mesh in EKS ( eBPF with AWS ) The main benefit is in with high performance networking by shortening the network path . But How does it work ? How is it done otherwise ? Lets talk about it

How is it done traditionally ?

Service mesh uses Kube-Proxy that runs on every Node takes care of routing all the request from the Kubernetes nodes to the pod. It also takes care of service abstraction , and load balancing for all the pods that are part of the service .It uses Linux routing and updates the routing rules based on the pods deployed in the Node .When new service is created it updates the iptable rules to redirect traffic from the service Cluster IP to the backend pods. 

So this is good what's the issue ?

Its the network path and latency associated with it. Also the updates that needs to be done in the iptables whenever pods get deployed /removed . eBPF allows packet processing directly in the linux kernel eliminating the need for packet-to-packet NAT transformation with iptables. eBPF can also optimize the service network and facilitate direct pod-to-pod communication without intermediate NAT processing  

But what about TLS ?

Applications offload the network encryption requirements to the service mesh or the underlying network . Simply using transparent encryption ensures that the traffic is encrypted with in the cluster.Both IPSec and Wireguard are supported by Kubernetes networking by Cilium and Calico. There is also option to use TLS for initial certificate exchange and endpoint authentication , this way the application is used for identity rather than the node . 

Containerization Github List  ->   Github Containers

eBPF with AWS

I have initially talked about eBPF and how it can be used for security tooling refer - ebpf. One of the main advantages is high performance networking you can see that Facebook adopted this in 2017. Performance being a main advantage there . Want to know how read here high-performance-networking

Cilium has worked on bringing the high performance networking to the Kubernetes service mesh and private setups have already started using it . But you are using Amazon EKS or other cloud provider you are also dealing  with compatibility requirements as well . High performance networking is  now part of AWS . If your application is using EKS there may be workloads that require high performance networking it's worth trying them out . 

Why Sidecars are less favored ?

Sidecars was a way to include additional features with security tooling and observability but soon the resources become a concern . Sidecars consume additional CPU, memory, and network bandwidth, which can be significant, especially in large-scale deployments. Each sidecar runs as a separate container, which means more resources are required per pod

Step by Step instructions 

https://github.com/aws-samples/cilium-service-mesh-on-eks

Supply Chain Risks on Spotlight

The CrowdStrike incident underscores the need for robust supply chain risk management strategies. Organizations must assess and mitigate risks associated with their technology providers and other critical suppliers

While some airlines in India demonstrated resilience by implementing manual processes like handwritten boarding passes, many others were unable to operate, revealing a stark contrast in contingency planning. This incident emphasizes several key lessons for organizations:

 Comprehensive asset management

 Organizations must maintain a detailed inventory of all assets within their systems. This includes knowing which devices are Windows-based, which are hosted on Azure, and how they are affected by such outages. A lack of comprehensive asset management can significantly prolong the recovery process.

Mandatory update testing

The importance of thoroughly testing updates before deployment cannot be overstated. Both software providers and clients should conduct extensive testing to prevent widespread disruptions.

Effective contingency planning

Having a "Plan B" is crucial. Organizations need to develop and regularly update backup plans to maintain operations during system failures.

Third-party risk assessment

Companies should regularly evaluate the security posture of their vendors and service providers to minimize potential issues in their supply chain.

By addressing these areas, organizations can better prepare for and mitigate the impacts of similar incidents in the future. The outage serves as a wake-up call for businesses to prioritize supply chain resilience, comprehensive asset management, and robust contingency planning in an increasingly interconnected digital landscape

Traffic Shaping Strategies Using Kubernetes and Istio

Service mesh in Kubernetes has evolved there are multiple types of service mesh for different use cases. Either you need a hybrid network with VM's and containers or high performance options using eBPF . 

This article I want to provide a quick how to on how to do traffic shaping with Istio. It is way easier to do this rather than applying this on application. Once you try out it you will be using it most often for Development and QA . 

Well Can't I do this in my Microservice ?

Yes , its common e.g Applying the new version of the API only particular type of customer or only for a certain list of customers . Here are the reasons you should consider Kubernetes approach 

Unified traffic management and Fine-grained control

Kubernetes provides more granular control over network traffic at the infrastructure level, allowing you to shape traffic based on various criteria such as source, destination, and protocol. This level of control may not be easily achievable through microservice design patterns alone.

Infrastructure-level management

Traffic shaping in Kubernetes operates at the cluster level, affecting all services running within it. This allows for consistent traffic management across the entire application ecosystem, regardless of individual microservice implementations.

Dynamic Scaling

If you are using in Production Kubernetes can automatically adjust traffic shaping rules based on the current state of the cluster, such as the number of pods or resource utilization. This dynamic approach is harder to implement solely through microservice design patterns.

How about in Development and QA ?

Good thing about this is also the fact that you can try this locally or in your Dev and QA to stimulate and test various cases , even if you are not using containers in production . Create different containers for different version of the API's deploy them using Kubernetes  . I have provided the step by step instruction below 

Step by Step

Github

Containerization Github List  ->   Github Containers

Confused about Multi architecture ?

Are you confused 😕 about Multi architecture in Docker  ?  Just think of Shoes 👞.... yeah shoes and how you use them .. 

• 
  You have a shoe closet with multiple shoe types (e.g., running shoes, hiking boots, dress shoes)
• Manual selection of the right shoe type for each activity
• Appropriate footwear for various terrains and occasions
• Central storage for all shoe types

Similarly Docker Multi-Architecture Images has 

• One image tag, multiple CPU architectures (e.g., x86, ARM, PowerPC)
• Automatic selection of the right architecture by the Docker runtime
• Same application runs across diverse hardware platforms
• Single management point for multiple architecture versions

Docker can still run anywhere right ?

Yes ,you can use the same shoe every where but not ideal right . Similarly traditional images are built for a specific architecture (e.g., x86-64 or ARM64). Multi-architecture images contain multiple versions of the same image, each built for a different architecture, bundled together under a single tag.

How does it do it ?

Multi-architecture images use manifest lists (also called fat manifests) to specify which image version should be used for each supported architecture. When a client pulls a multi-architecture image, Docker automatically selects the appropriate version based on the host system's architecture

Build is complicated ?

Yes kind of ,Multi-architecture images require separate builds for each target architecture, often using tools like Docker Buildx . Here is a example it not that difficult 

Github

Yeah , but like why .. What's the main advantage ?

Just like using the right shoe will give you the best performance and utilization . Multi-architecture images allow for more efficient use of resources by supporting diverse CPU architectures without maintaining separate images . This is especially realized in the cloud by using the correct architecture-specific image, you can achieve better resource utilization and potentially reduce costs, especially in cloud environments with diverse instance types 

Containerization Github List  ->   Github Containers

Difference between Multistage vs Docker-in-Docker

Developers often confuse two important Docker concepts: multi-stage builds and Docker-in-Docker (DinD). While both involve Docker and are used in build processes, they serve different purposes and are implemented in distinct ways. Let's clarify these concepts with descriptions and examples

Multistage

Multistage build is for the developers building the docker image to use multiple stages . The main reason is to minimize the image size and practice better security as it minimizes the attack surface . There are usually multiple FROM statements for each of the stages in the build . 

Here is an example of the nodejs application built using this approach . Go application will be a better example as only the binaries will be copied to the final image , I am using nodejs for simplicity 

Github Docker Multistage

Docker-in-Docker

Docker-in-Docker refers to running a Docker daemon inside a Docker container . Its mainly used by CI/CD pipelines to build container images . This doesn't necessarily improve security , in-fact as the daemon needs to run as root it has to be run with the privileged flag ( may be the only place where this is okay ). 

It can be also used for Integration and Sanity testing as the main Docker container acts as a platform for all the other containers . You can use tools like Docker Compose to bring up multiple containers that represent different services in your application stack and run the tests . After tests are complete, you can easily tear down the entire environment, ensuring a clean slate for the next test run and improving reproducibility

here is an nodejs example demonstrating this

Github Docker-in-Docker

Containerization Github List  ->   Github Containers