In a previous blog, we discussed how a malicious actor can exploit CVE-2020-8554 to bypass Kubernetes' RBAC isolation and conduct a man-in-the-middle attack. We demonstrated the network connections between the victim and the attacker. In this blog, we show how NeuVector protects your applications against this Kubernetes vulnerability.
We also briefly discussed the attack vector using ExternalName service in the last blog post. This attack scenario is not well discussed in the community. The following diagram shows how an attacker can initiate the Man-in-the-middle attacks if an ExternalName service is used in the Kubernetes cluster. The ExternalName service is designed for the internal pod to reach the DNS name based server outside of the cluster. The attacker can create the external IP service with the same address as the external server, the client's traffic will be redirected to the pods under the attacker's control.
Because NeuVector enforcers are able to see all of the pods' network connections and packets, we can draw a network map of the attack scenario. However, this gives us a view of two separated connections, one connection from the victim as the client to the external network, the other connection from the node to the attacker's container. In order to present a real end-to-end network view, we also need to understand how the attacker's external IP service is defined.
By integrating with the Kubernetes control plane, we can correlate the exposed external IPs with the real-time network connections. In this case, we discovered that a pod (victim) is making a connection to an exposed external IP. We can draw a network map like this,
In this diagram, instead of presenting the connection to the external network, we draw a line between the victim and the real destination, the exposed service created by the attacker. Additionally, we raise an alert on this connection. (Note that the client now is in the "discover" mode.)
To understand how we can draw this conclusion, we need to see what use case that external IP service is designed for in Kubernetes.
If there are external IPs that route to one or more cluster nodes, Kubernetes Services can be exposed on those externalIPs. Traffic that ingresses into the cluster with the external IP (as destination IP), on the Service port, will be routed to one of the Service endpoints.
External IP service is used to allow a Kubernetes service to be accessed from outside of the cluster. If an exposed external IP is accessed by an internal pod, it indicates something abnormal, whoever is making the connection is under the attack, thus we raise an alert even in the "Discover" mode. If the client is in the "Protect" mode, we will block the connection.
# oc exec client -n client -- curl example.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0
curl: (7) Failed to connect to example.com port 80: Operation timed out
command terminated with exit code 7
In the network graph, you can see the connection and retries are blocked.
Two Protection Mechanisms
It's now a good time to talk about two mechanisms that we use to protect the container environment at run-time.
The first approach is called the whitelist model. Either defined explicitly by the developers using CRD (covered in this blog post) or learned in the "Discover" mode, the application's baseline can be established. The baseline covers the application's network, file system and process activities. For any activity that deviates from the baseline, we will raise an alert or block the action even before it takes place.
The second approach is, if we know an activity is part of malicious processes, we will proactively report and stop it. This is how we detect and block crypto mining programs, modification of sensitive files, Apache Struts RCE (remote code execution) attacks, C&C tunnels and this man-in-the-middle attack, to name a few examples.
Summary
In contrast to the passive audit-based approach recommended by other vendors, NeuVector takes more proactive actions to identify and prevent CVE-2020-8554 in Kubernetes from getting exploited by the attackers. NeuVector enforcers keep watching containers' network activities and take it as the independent source of truth, then we augment that with the knowledge of the Kubernetes cluster. This combination gives us a unique capability to mitigate this man-in-the-middle attack and protect your Kubernete application in production.
