What Is Complete Run-Time Container Security?

By Gary Duan

Securing containers in production is no easy task. And getting it wrong or deploying weak run-time container security will ultimately leave you vulnerable to an attack. It would only be a matter of time before you will be scrambling to recover and possibly facing bad publicity from a data breach.

Run-time container security is primarily a prerequisite for an effective security posture that involves spanning both legacy and containerized deployments. It is the ability to capture all activities in the container/application environment.

The activities that comprise run-time container security include analyzing container and host activity, monitoring open ports, and network inspection of the protocols and payloads in connections. Monitoring these activities is critical because they are used to communicate with other applications, file systems, processes, and require certain privileges.

So why is run-time container security so critical? Fundamentally, containers spend most of their life-cycles at run-time, especially when they are running in the production environment, serving requests received from the Internet or other internal microservices. Containers, just like other application infrastructures, are under constant scans, attempted attacks and data exfiltration attempts. While preventative security measures such as vulnerability scanning and system hardening are best practices, they are insufficient to provide true run-time protection.

Considering such a dynamic container environment, NeuVector is equipped to gather all the run-time activities to detect suspicious activity from pre-defined container activity patterns. The solution also determines deviations from normal process and network baselines, which are created by behavioral learning techniques.

It is only after you have deep insight into all of this type of information can you then efficiently determine, for example if a bad actor is probing into the network or if real command-and-control (C&C) server channels have been established to do further damage.

Traditional Security Approaches Don’t Work for Containers

Traditional perimeter-based network security companies, such as next generation firewalls (NGFW) and web application firewalls (WAFs), whether virtual or physical deployments, are typically positioned at the edge of the network and don’t run alongside the workload.

There are other security companies that just provide endpoint protection but don’t examine network traffic. Most of these legacy solution don’t even have visibility into the modern cloud-based endpoint, which is the container itself. So why have we had two or more different types of security companies providing security solutions? It’s because in the old infrastructure, these were separate technologies managed by different teams with different workflows.

In a container environment, it’s all one continuous pipeline from dev to production, and the host, network, and endpoint are all part of that pipeline. A complete run-time container security solution should have deep visibility and protection for all of this.

How Should Run-Time Container Security Operate?

Actually, it made sense in the past to have different silos of security but such an approach has become obsolete in today’s digital world. Now with cloud-native environments, for the first time, we have the opportunity to converge both the network and workload security to form a single viewpoint.

If you provide security solely at the endpoints, ultimately the bad actors have already infiltrated the network. On the other hand, if you simply opt for the perimeter-based protection, there is the high potential for undetected lateral movements.

However, by using the network as the single source of truth, the bad actor can be stopped before it reaches the workloads, and before data is stolen over the network. The NeuVector technology is positioned with a unique viewpoint that enables the run-time container security solution to protect both the endpoint and network at one time. Host security is also included as a bonus. In essence, this creates the most effective security posture for container run-time security.

Running in sync with the workloads is important as this provides real visibility into how the workloads are deployed. Container endpoint and network protection must scale automatically with workload scaling. A significant advantage is that NeuVector correlates everything on the network, workload and host to detect the ‘kill chain’ of an attack. This way, we can correlate all the events and activities from previously different security domains to track a kill chain in action.

In the past, with traditional security deployments, endpoint security was the main application workload protection mechanism. However, today we are in the era of containers, and this surfaces the need for the ability to analyze different elements such as container file system activity and processes. Legacy endpoint security doesn’t go deep enough in a container environment.

In addition, traditional endpoint security solutions are also challenged due to the deployment and scalability that the containerized environments present. Not to mention that they fail to understand the critical metadata of the container.

Today, customers demand a ‘complete’ security solution. They don’t want different security solutions that only cover one aspect of the hosts, containers, network and other infrastructure. They want a one-stop solution that is all-encompassing for containers.

Attack Surface and Micro-Perimeters

The way bad actors attempt to penetrate systems and applications is the same for containers, but the systems under attack are different. Essentially, a kill chain still exists for container attacks, just with different targets.

There are multiple steps and techniques that bad actors can use to try to compromise the system. But now, we have distributed systems, multi-cluster deployments, along with orchestration layers and a variety of tools. As a result, there are more attack vectors available for the bad actors to compromise. NeuVector follows a unique approach to stop each attack at different stages of the kill chain. It’s never enough just to focus on one type of attack vector; rather it’s necessary to protect all the possible vectors, or as many as possible.

Today we require many smaller micro-perimeters around each microservice. The increase in perimeters has been the result of the evolution in new technologies. If you examine this with a birds-eye view, you will notice that today there is actually a larger attack surface than before. And this broad attack surface is made up of different elements which represent many smaller attack surfaces. This is compounded by the fact that there is a new attack surface that didn’t exist before – the container orchestration and related services layer.

Vulnerability Scanning Is Not Enough

Although we have vulnerability scanners they can only find known vulnerabilities and in many cases even the known vulnerability may not have a patch. In fact, most of the vulnerabilities are never patched. As a result, users often have to run their applications with vulnerabilities. And they are unable to detect zero-day attacks since scanning and hardening has been accepted as the norm for ‘good enough’ security.

As a result, bad actors will find a way in and exploit the weakness that you just can’t catch with scanning. This is especially true for applications built using popular languages, such as java, nodejs, ruby or python, where there is a potential to find hundreds of vulnerabilities that are not patched. This is compounded by the fact that even if you do find vulnerability, there is a chance that you might have already been running in production with that vulnerability for months. There is also an abundance of vulnerabilities that have not been discovered yet.

Keep in mind that even if you do have a patch, you are only patching the known vulnerabilities. This is a major issue in case of vulnerability scanning. By itself, it’s simply not good enough. There is also a window period which is the time period that it takes to patch. It’s not a simple task to upgrade everything and in the meantime, bad actors can penetrate into the network.

Even with vulnerability scanning, you still need to continuously monitor and check for anomalous behavior. Vulnerabilities scanning is merely a good practice to reduce the obvious and not to make it that easy for bad actors. This is why you need a run-time container security solution.

Why Is the Network Part of Run-Time the Most Important?

Deep network visibility and protection is the most critical part of run-time container security because it acts as the first layer of defense before bad actors can actually reach the workload. This is the reason why we had traditional firewalls in the first place because administrators wanted to quarantine or block the attack before it reached the workload. The network is also the last line of defense to protect against data breaches. And in between the first and last lines of defense is preventing the network use for expanding an attack in an east-west direction. Proper network controls will limit the ‘blast radius’ of an attack.

Logically, when you have the ability to examine the network at the packet level, there is nothing that can be hidden as this is the only real source of truth to identify anomalous behavior. When you examine packets at layer 7 and in the payloads, you can really see how the application works. This is far superior to layer 3 and layer 4 protections, which can also be important. But these layers are more concerned about crudely allowing or denying connectivity, for instance, can A talk to B without considering application protocols or embedded attacks.

What you really want to know is how the applications are communicating. Are they using valid protocols? Do the payloads contain attempts to hack into the application? When you look deeper into the network you can, for example, set a policy so that one application can read from the database but the other application has the additional privilege to delete entries.

The network layer is the only location where you can enforce these policies. Analyzing layer 7 traffic is an absolute necessity to detect and fully understand the protocols. Only then can you provide advanced technologies, such as, deep packet inspection (DPI). When you apply DPI to identify attacks, detect sensitive data, or verify application access, you further reduce the attack surface. It’s the only place that enables security to help enforce business policy.

In summary, inspecting network traffic reveals how an application communicates with other applications. And it’s the only place that can stop attacks before they reach the application. It’s also the last chance to prevent data breaches by exploited applications which send data out over the network.

Monitor and Protect Orchestration Systems Like Kubernetes and Istio

Recently, there have been vulnerabilities on orchestration systems which presents an additional attack surface for bad actors. Orchestration tools and all related system services also need to be protected during run-time. The Kubernetes API server or docker run-time could have been exploited by hackers. Kubernetes also has a global key-value database to store the deployment of the application’s intent state. Unfortunately this is all stored in a centralized database that is not password protected.

NeuVector doesn’t just protect the application workloads at run-time but any tools and orchestration systems that are in use are also monitored. New technologies like the Istio service mesh are complex to deploy, and misconfigurations can create attack vectors in addition to vulnerabilities which have yet to be discovered. That’s why we just released service mesh integration for Istio, Linkerd2 and AWS App Mesh. So not only can we monitor the network connections of application workloads, we can also inspect the service mesh infrastructure and the communications between the control and data plane, even if the communication is encrypted!

Open Source Security?

Open source security technologies can provide certain limited security capabilities. For example Bro, which is similar to Wireshark, can capture the network traffic and visualize network data, which has its own benefits. Other open-source components may include vulnerability scanning functionality or layer 3/4 network policy but again they don’t have the full coverage required for complete run-time container security.

In reality, open source solutions do not provide end-to-end protection. They focus on just one piece of the technology and require customization, integration, and maintenance since they are not supported by a commercial vendor.

There are security features in Kubernetes, Calico, and Istio technologies, but these are not security products exclusively designed with all what security teams need. Logging, forensics, packet capture, and management of rules in one place are essential for proper security management. The orchestration systems and open source technologies do not provide this.

Security Based on Network Guessing?

Don’t be fooled into thinking that pretty diagrams showing you network connections between container objects are a true container firewall with network deep packet inspection. It’s easy to inspect the deployment manifests of container services and the open ports or syscalls during run-time and create static diagrams showing ‘presumed’ network connections. But these are just guesses, rather than real-time indications of true network traffic that is being filtered and inspected.

There are also container security solutions that require DevOps to inject code into their applications, or run sidecar containers to secure it at the network layer. Don’t forget that if you have a thousand containers you’ll have a thousand sidecars, each taking resources and needing management. A run-time container security solution must be non-intrusive to the environment protected, scale up/down automatically, and be managed just like any other container service deployed.

The NeuVector Solution for No-Compromise Run-Time Security

NeuVector provides a complete view and protection of all of the threat vectors and things that can be used in an attack, including the network, containers, hosts, and orchestration systems. NeuVector is the only vendor that offers true run-time security protection. Our solution offers unique DPI in the form of a container firewall, thereby analyzing east-west, ingress and egress traffic. If you are considering a run-time container security solution without this level of network protection, you a missing a critical aspect of container security.

As discussed above, run-time container security is an important piece of the security puzzle; I might say the most important. However, what is often needed is a full lifecycle security platform that starts with the developer. A full lifecycle security platform would start with DevOps code scanning, vulnerability scanning during build process, and registry scanning.

Then the critical link between the CI/CD pipeline and run-time is admission control and CIS benchmarks, which can act as gates to production deployments. Here you can define policies for what is going to be allowed to be deployed in your environments. And finally, complete run-time container security with deep network protection.

NeuVector provides a full life-cycle container security solution with complete run-time protection. This provides real security in depth where security can be integrated into all these stages from build to ship to run.