How to Implement Container Security Automation Into the Kubernetes Pipeline
Enforcing security and compliance requirements in modern cloud-native pipelines can be a challenge without at least some level of container security automation. The increased attack surface of container infrastructures makes security even more important, but security and DevOps teams can’t afford to slow the pipeline with manual processes.
The CI/CD Pipeline Attack Surface
Before we look at container security automation, let’s understand why security is so critical to container and Kubernetes pipelines. From the beginning of the CI/CD pipeline into production, there are attack points where security can be comprised. The diagram below provides a summary of the most important critical security issues which keep DevOps and security teams awake at night.
- Critical Vulnerabilities. Vulnerabilities can be introduced in the build phase, registry, and even into the production environment.
- Security Breaks Pipeline. Adding manual configuration steps for security reasons or introducing process checkpoints for security reviews can slow or even stop the continuous integration and release of new or updated applications.
- Misconfiguration. Cloud-native tools including development, registries, and Kubernetes are both new and complex, leading to misconfigurations which compromise security.
- New Attack Surfaces. The critical infrastructure such as the container orchestrator Kubernetes, run-time CRI-O or containerd, and supporting infrastructure such as registries are all attack surfaces.
- Inadequate Protection in Production. Good security hygiene includes rigorous vulnerability scanning, compliance auditing, and hardening of production infrastructures. But no amount of scanning and preparation will eliminate the risk of zero-day, insider, and phishing attacks.
Properly addressing these security concerns should be the primary focus for integrating security into the pipeline.
10 Steps to Container Security Automation – A Summary
In this article we’ll take a high level look at ten integration points which can serve as a guide for 10 steps for container security automation. While there are many more than ten, these are highlighted as first steps because they are simple and impactful for improving security. Here’s a summary of these, followed by a pipeline reference diagram. The detailed guide with full descriptions of each of the steps can be downloaded in pdf.
- Build-Phase Scanning. Container images should be scanned for vulnerabilities and compliance violations during the build phase so the build step can be stopped (failed) in order to force correction or remediation. Integration is made easy through plug-ins and extensions for popular tools such as Jenkins, CircleCI, Azure DevOps, Gitlab, Bamboo etc.
- Registry Scanning. After images pass build-phase scanning, they are staged into registries and should also be scanned there. New vulnerabilities can be discovered or introduced after images are pushed to registries. Registry scan results can also be linked to Admission Controls (see #7 below) to prevent unauthorized or vulnerable images from being deployed.
- Production Scanning, CIS Benchmarks & Compliance Auditing. Scanning and auditing should extend into the staging and production environments with run-time vulnerability scans and running of CIS benchmarks for Kubernetes and Docker, as well as any custom compliance checks.
- Risk Reporting and Vulnerability Management. Although addressing high risk environments and interpreting vulnerability management reports typically involve some manual review, alerting and correlation can be done automatically to speed and ease assessment and remediation.
- Security Policy as Code. Defining and deploying security rules for new applications can be automated as code in the same way that Kubernetes supports deployment manifests in standard yaml files. In this way, new or updated workloads are automatically protected as they are deployed to production with security policy as code.
- Application Behavioral Learning. Behavioral learning is an important technique for automatically characterizing application behavior such as network connections, protocols used, processes required and file access activity allowed. We will see later how this can work together with Security Policy as Code (#5) to automate run-time security from dev to production.
- Admission Controls. Admission controls are an important bridge between the CI/CD pipeline and the production environment. Once rules are established, vulnerable or unauthorized deployments can be automatically prevented.
- Container Network Firewall. A layer 7 (application layer) container firewall will automatically enforce network segmentation by blocking network attacks and unauthorized connections. The creation and maintenance of these whitelist network rules can be automated as code (#5) utilizing behavioral learning (#6)
- Container Workload & Host (Endpoint) Security. Containers and hosts should be continuously monitored for suspicious and unauthorized behavior such as processes and file activity. These activities can be automatically blocked. The creation and maintenance of these whitelist rules can be automated as code (#5) utilizing behavioral learning (#6).
- Alerting, Response, and Forensics. Finally, automated alerting, response, and forensic capture can be initiated for suspicious activity. These should include quarantine of containers, packet captures, and custom notifications to case management systems.
Who Should Implement Container Security Automation?
The 10 steps summarized above provide a great start to security automation. But who should be responsible for each of these steps?
Let’s take a simplified view of two important roles: DevOps (which includes DevSecOps), and Security. These considerations can help determine who is responsible for each step:
- Security teams are ultimately chartered with protecting critical assets, especially in a production environment. However, the ‘shift-let’ movement to build security into the pipeline earlier means that DevOps needs to understand security concepts and own some aspects of security automation.
- While security teams are often trained on concepts such as network segmentations, firewalls, endpoint security, IDS/IPS and incident response, they are often not as familiar with modern cloud orchestration tools such as containers and Kubernetes. They will need to learn which of these traditional security concepts should be applied and enforced in cloud-native deployments, and how they should be implemented.
- While DevOps teams are rapidly becoming experts on automated CI/CD pipelines and how to manage orchestration tools such as Kubernetes, they often lack the understanding and experience of security technologies and best practices in production environments required to fend off the constant attacks on the infrastructure and applications.
- New cloud-native tools such as registries, orchestrators (Kubernetes), service meshes and other open source tools introduce new attack surfaces that are not known to either security or DevOps teams.
- New concepts such as ‘infrastructure as code’ and ‘security policy as code’ will require collaboration between DevOps and security teams in order to enable automation of pipelines in a secure environment.
In general, the earlier steps (1-4) are typically the responsibility of DevOps and Compliance teams, with the later steps (8-10) being the responsibility of Operations and Security teams. The middle steps (5-7) are the bridge between the CI/CD pipeline and the production environment with Security Policy as Code and Admission Controls being managed by the DevOps teams with oversight from the Security team.
How Should the Container Security Automation Be Done?
There are tools and techniques available for automating all the 10 steps and more, from plug-ins to simple scripting. Many container security automations are already built into tools like NeuVector to simplify security enforcement.
- Plug-Ins and Extensions. The most frequently used security automations have plug-in or extensions, such as a Jenkins plug-in for build-phase scanning.
- Built-In Integrations. Modern cloud-native tools such as Kubernetes offer resources to enable security integrations such as admission controls, CRDs or RBAC enforcement. These can be leveraged by security tools like NeuVector to provide a rich set of security features automatically integrated with the orchestrator.
- REST APIs. For any integration that needs to be highly customized or is not supported by a plug-in, a fully featured REST API provides a way to script security automations. For example, all scanning, policy rules, user management and every configurable function in NeuVector can be controlled through the REST API.
- Alerting and notifications can be automated through traditional methods such as SYSLOG events or modern triggers such as webhooks. These can not only trigger alerts and notifications but can open cases and trigger responses such as quarantines and packet captures.
In most cases, some simple configuration for customizing the security policy is required initially, and even this can be automated through REST APIs. Once configured, scanning, monitoring, enforcement, updating, and alerting can all be automated.
What is Virtual Patching?
Virtual Patching enables security teams to ‘virtually patch’ a vulnerability in a running container or host, without needing to update or replace the running asset with a patched version. It essentially protects the running asset (container, host etc.) against an attempted exploit of the vulnerability.
The best way to do this is to automatically characterize and whitelist all application container behavior such as network connections, processes, and file activity, then lock it down. The workload or host are then ‘virtually patched,’ and any attempted exploit will create an unauthorized network connection, process, or file access which can be blocked.
Virtual patching protects against vulnerability exploits, embedded malware, zero-day attacks, and insider & phishing attacks.
[Additional Reading: See this article for more information about End-to-End Vulnerability Management and Virtual Patching]
Take the First Steps Now to Container Security Automation
The road to a fully automated pipeline with integrated security controls all the way into production starts with a few easy first steps. Which of the ten steps in this document you start with depends on where you are in the process of automating your pipeline. What is most important is to get started with what can be done easily and what is most critical at this time, then create a roadmap to fully container security automation.
