eBPF: The Secret Weapon for Achieving Network Performance Breakthroughs

 I wrote an article about using eBPF based service-mesh in EKS ( eBPF with AWS ) The main benefit is in with high performance networking by shortening the network path . But How does it work ? How is it done otherwise ? Lets talk about it

How is it done traditionally ?

Service mesh uses Kube-Proxy that runs on every Node takes care of routing all the request from the Kubernetes nodes to the pod. It also takes care of service abstraction , and load balancing for all the pods that are part of the service .It uses Linux routing and updates the routing rules based on the pods deployed in the Node .When new service is created it updates the iptable rules to redirect traffic from the service Cluster IP to the backend pods. 

So this is good what's the issue ?

Its the network path and latency associated with it. Also the updates that needs to be done in the iptables whenever pods get deployed /removed . eBPF allows packet processing directly in the linux kernel eliminating the need for packet-to-packet NAT transformation with iptables. eBPF can also optimize the service network and facilitate direct pod-to-pod communication without intermediate NAT processing  

But what about TLS ?

Applications offload the network encryption requirements to the service mesh or the underlying network . Simply using transparent encryption ensures that the traffic is encrypted with in the cluster.Both IPSec and Wireguard are supported by Kubernetes networking by Cilium and Calico. There is also option to use TLS for initial certificate exchange and endpoint authentication , this way the application is used for identity rather than the node . 

Containerization Github List  ->   Github Containers

Layered Security Strategies: Implementing Defense in Depth for Containerized Applications

Containers are great they let you "Build Once Run Anywhere" and lets you create the Isolated environments that lets gives freedom to developers to choose their programming languages.  No more worrying about different versions of libraries for different products and its great for deploying micro services. 

Containerization is also that involves Developers , DevOps , Security and your Platform team. I am outlining some of the high level items with respect to securing them . 

Reduce Attack Surface

Choose your base images carefully and install only the required libraries and packages , better option is go to with Distroless images that has only has the application and the runtime dependencies .

Smaller image inherently improves security by reducing the attack surface and increasing security . It can also increase performance by increasing the download speed and reduces the cost .  Main disadvantage is it will require a multistage build e.g , Go application will have a first stage build that will have a go compiler and all the dependencies and second stage that will package the executable with the required runtime packages

Image Scan

Scan your images for Vulnerabilities . You can either do them as part 

of your 

• Build process
• CICD pipeline 
• Registry scan ( if you have your own Registry ) 

Scanning for Vulnerability is a good security practice and will also be part of your compliance depending on your industry. 

SBOM

Provides visibility to all the software components and libraries that are part of your container image . It will help identify Vulnerabilities and Patch them quickly . Note : Vulnerabilities can be reported later than your original scan and SBOM can be valuable in this aspect .It's can be a good Asset Management tool . Its a mandate that all organizations that sell software to U.S. Federal Government are required to provide a Software Bill of Material (SBOM) as part of Supply Chain Security ( as of June 2023) 

No Root 

Running containers as root increases possibility of  Vulnerability exploitation potentially allowing attackers to escape the container and compromise the host. It also violates the Principle of Least Privilege.  Docker uses dedicated appuser and appgroups . Kubernetes always make sure the following are set as part of your Security Context

runAsNonRoot: true 

allowPrivilegeEscalation: false

Privilege Escalation is one of the major ways that Vulnerabilities in the container are exploited . Here is a good example of a Vulnerability reported in 2022  https://nvd.nist.gov/vuln/detail/CVE-2022-0492

Keep your Secrets Secret

Don't put your passwords or sensitive information in Environmental Variables , Source Code or Docker file . Use Docker secrets , Hashi Corp Vault or your Cloud Providers key management services . Always ensure that secrets are encrypted at rest and in transit . Use role based access control and grant access on need to know basis . 

Networking

Avoid exposing services directly to the internet when possible .Implement mutual TLS encryption between services , almost all service mesh options with Kubernetes support mTLS.  TLS Termination  - Check your Ingress controllers for proper TLS termination for incoming traffic

Remember there is nothing like a perfect or ideal setup and a no-risk environment .with respect to Securing your containers you can "Do more with less" by just following these practices . 

eBPF: Redefining the Boundaries of Security, Networking, and Observability

eBPF is extended packet filter that is used to run custom code in the Kernel. Its used for monitoring and security tooling . But as I looked more into this the capabilities are much more than that.Lets see why .. 

Everything goes through the Kernel ! 

All the apps , irrespective of if its a docker container , host program or AWS lambda needs to run in the Kernel . Anything to do with the Network , Files , Processes and Memory are handled by the Kernel . So it provides a single point to add a global capability rather than in the application or a specific to implementation e.g k8s service mesh 

How does it run ? 

Runs in user space , eBPF has helper functions to integrate with kernel . kprobe ( Kernel Probe) allows to attach the custom functionality to System calls

How does program Interface ? 

Helper functions facilitated by kernel probe helps to read data back . Data structures such as BPF maps and Hash table maps are also used to store data that can be also used for integrating with other BPF programs 

How is it powerful ? 

No need to restart , redeploy the apps . You get the benefits immediately 

Where else its used ? 

Katran uses it for high performance load balancer . Used by FB since 2017 ( https://engineering.fb.com/2018/05/22/open-source/open-sourcing-katran-a-scalable-network-load-balancer/ ) on a side note , the service mesh model used by Containers may not be performant as it needs to access network layer often . 

K8s using side cars may not cover all the applications in organization . eBPF can solve the problem since everything needs to go through the 

Where else it can be used ? 

Used for Monitoring , Security tooling and High performance networking . It can be much more effective in reducing the number of calls when compared to sidecars used in container security . 

Root User?

Need to have sudo access to setup and run the eBPF modules.New permissions introduced in kernel 5.3 for specific permissions with respect to running eBPF 

• CAP_BPF 
• CAP_PERFMON 
• CAP_NET_ADMIN 

 Perfmon and bpd are required for tracking NetAdmin and Bpf privileges are required for networking 

What about Kernel upgrades?

Kernel upgrades are usually backward compatible , and don't change that often . Also the latest versions add more capabilities to eBPF . 

SecComp 

Secure computing is a module that is used to run the kernel in a strict and restrictive mode . It a utility powered by eBPF that limits the set of instructions that can be performed by kernel . This allows the user to run untrusted code 

Systracking 

Falco is setup in all kernels and has configs that are used to enable and configure the eBPF programs. User will be able to leverage the capabilities without using the eBPF code . LSM hooks LSM hooks can also be used to track the system calls. As the linux upgrades are backward compatible And not much has been altered over the years .Its relatively safe to use the hooks for security tooling 

More reading eBPF 

A Beginner's Guide to eBPF Programming with Go 

https://www.youtube.com/watch?v=uBqRv8bDroc&t=885s

eBPF  and Kubernetes: Little Helper Minions for Scaling Microservices 

https://www.youtube.com/watch?v=99jUcLt3rSk 

XDP  -https://en.wikipedia.org/wiki/Express_Data_Path