‘K’ is for ‘Komplex’: Key Kubernetes Considerations for Security Teams

The use of Kubernetes introduces complexity to the modern attack surface and requires a different approach to security than traditional IT infrastructure. Security teams need a base understanding of Kubernetes architecture, configurations and deployment processes to effectively manage risk. Here’s what you need to know.

According to the Cloud Native Computing Foundation’s “Annual Survey 2021: The year Kubernetes crossed the chasm,” Kubernetes adoption has surged, with 96% of polled organizations saying they’re either using or evaluating Kubernetes – a record high since the first CNCF survey in 2016. “Container adoption and Kubernetes has truly gone mainstream – usage has risen across organizations globally, particularly in large businesses,” the report reads.

Kubernetes is an open-source platform that orchestrates the automation of large container deployments. While it enables developers to quickly build, deploy and scale cloud-native applications, it also introduces a number of new challenges for security teams.

During KubeCon 2022 North America Ayse Kaya, Head of Strategic Insights and Analytics at Slim.AI, started the keynote “What We Learned Dissecting the World’s Most Popular Containers” with the question: “What can be more complex than biological systems, except maybe Kubernetes?” 

She then went on to share some shocking data from Slim.AI's “Public Container Report 2022,” which was released in October 2022 and is based on an analysis of the top 165 public containers on Docker Hub; on a survey of 300 developers, DevOps practitioners and engineering executives; and on scans of hundreds of thousands of unique container images:

• 60% of the top public containers had more vulnerabilities than they did a year prior
• Container vulnerabilities were detected at a rate that’s four times faster than the remediation rate
• Only about 1 in 4 of the developers polled had the skills to harden their containers for production use

The point is that things are getting worse and not better as Kubernetes adoption increases. At Tenable, we believe a large part of this is due to Kubernetes being managed by the developer side of an organization, where security controls and automation have not caught up with IT best practices and are missing critical elements.

In the blog Five Core Principles for Hybrid Cloud Security, author Tom Croll highlights the fact that security and cloud teams think about security differently. “When developers think about cloud security, they think about technical controls…cool features that can enable their cloud-native applications running on containers or Kubernetes. When security teams think about controls, they want to know about risk, both qualitative and quantitative. For these reasons alone, it is not good practice to allow cloud teams to design security controls.”

For these reasons, it is incumbent upon security teams to take leadership across all of their environments, including Kubernetes.

Kubernetes: infinite attack surface and complexity

“Complexity is the worst enemy of security, and our systems are getting more complex all the time.”

— Bruce Schneier, Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World

As Bruce Schneier noted in his book Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World, “Complexity is the worst enemy of security, and our systems are getting more complex all the time.” Kubernetes introduces a new and complex addition to the modern attack surface; it enlists an infinite army of multi-cloud environments, on-premises data centers, internet of things (IoT) devices, personal computers, edge devices and more. Kubernetes requires a different approach to security than traditional IT infrastructure. Controls need to be applied consistently across all environments, and the importance of proper configuration can not be understated.

We continue to see misconfigurations being cited across customers of all sizes as the primary reason for security incidents. Such misconfigurations result from a lack of policies and controls being implemented consistently within an organization. We see this confirmed in VMware’s the State of Kubernetes Security 2022 report, which ranks the inability of an organization to apply policies consistently and control access to clusters as the top two security concerns. These are two key areas that have been traditionally governed by SecOps teams.

Source: VMware, The State of Kubernetes Security 2022

Moving past the challenges of Kubernetes security requires security and cloud teams to work together. Security teams must overcome three of the most common challenges surrounding Kubernetes security, including:

• The ephemeral nature of Kubernetes workloads
• Insecure default configurations
• Consistent application of policies

The ephemeral nature of Kubernetes workloads

One of the key challenges for security teams in securing Kubernetes and containers is dealing with short-lived workloads. These ephemeral workloads can be spun up and down in a matter of seconds, making it difficult for traditional approaches to security, such as runtime scanning, to keep up.

For example, modern applications are made up of dozens or hundreds of individual container images and microservices. The images used to deploy containers in Kubernetes pods or other container clusters are updated separately, have different base operating systems and software packages, and present a much greater security challenge than long-lived virtual machines and monolithic applications.

To address this challenge, security teams need to adopt new security measures that are specifically designed for Kubernetes. These measures include using Kubernetes-native tools, such as Kubernetes Network Policies, which provide granular control over network traffic within the cluster. Security teams can also leverage Kubernetes Admission Controllers, which can be used to enforce policies and validate configuration changes before they are applied to the cluster.

Insecure default configurations

Kubernetes comes with a number of default configurations that may not be secure out of the box. This can lead to new Kubernetes clusters being deployed in insecure ways. Even if you have a team of Kubernetes experts, it may not always be obvious what the “right” settings should be. These might include setting memory limits or the need to get a cluster working by giving it root access. Examples of insecure default settings include:

• Privileged containers: By default, Kubernetes containers have access to the host operating system, which can be a security risk if a malicious actor gains access to a container.
• Unsecured APIs: The Kubernetes API server is the primary control plane for managing Kubernetes clusters. By default, the API server is unsecured and can be accessed by anyone with network access to the Kubernetes cluster.
• Role-based access control (RBAC) settings: By default, Kubernetes clusters allow any authenticated user to access and modify any Kubernetes resource. This can be a significant security risk, as it enables lateral movement and privilege escalation.

To address the above challenges, security teams should leverage industry benchmarks such as the Center for Internet Security (CIS) Kubernetes benchmark. Such benchmarks provide a set of best practices for securing Kubernetes clusters. They cover areas such as authentication and authorization, network policies and pod security policies.

Consistent application of policies

The final challenge security teams face is the lack of defined security policies and processes. Without clear policies and processes, security teams may struggle to enforce security measures consistently across teams and environments. This can lead to security gaps and inconsistencies in security posture. To address this challenge, security teams should adopt policy-as-code, which provides security with a mechanism for enforcing policies consistently across teams and environments. Policy-as-code can be applied at several different stages in the development process, and we encourage users to apply it everywhere they can. For example:

• Developers can scan for policy violations before code is committed, reducing the likelihood of problems and delays later on/li>
• Code can be scanned in build pipelines to identify violations, schedule fixes and document compliance with the policy. The scan history is also a great resource for identifying training needs and streamlining later security assessments
• Code can be scanned in release pipelines to verify and document compliance before deployment
• Each policy can be embedded into the pipeline artifacts, to be used after deployment for automated compliance checks, targeted monitoring and so forth/li>

By adopting a single policy framework, security teams can ensure that security measures are consistently applied and that there are no gaps in security posture.

In conclusion, the adoption of Kubernetes has grown rapidly over the years, with more organizations using it to deploy cloud-native applications. However, this growth also comes with new challenges for security teams due to the complexities Kubernetes brings to the attack surface. To overcome these challenges, security and cloud teams must collaborate and adopt new security measures designed for Kubernetes. However, it's important to recognize that Kubernetes security is one piece of the larger puzzle of securing a cloud-native infrastructure. Security teams must also be aware of the risks posed by misconfigurations in the cloud infrastructure itself, as well as vulnerable applications that run on Kubernetes. By consistently applying security policies across both infrastructure and applications, security teams can better identify and prioritize vulnerabilities based on the potential business impact. Ultimately, a holistic and proactive approach to security is key to protecting your organization from cyber threats.

For more on Kubernetes and other cloud topics, we encourage you to join us for the bi-weekly Cloud Coffee Break.

(Editor's note: This blog was updated on June 12, 2023 to clarify some of the third-party stats cited in it.)