Page Comparison

Kubernetes (K8s) Security is a combination of multiple protection mechanisms on multiple levels.

Container image security

The main goal of K8s is to run a containerised application. Therefore, building a secure container image is important.

What is a container image?

Container, such as Docker, is a way to package and run application in an isolated environment. A container image (e.g. Docker Image) is the template for creating a container. A Dockerfile is declarative method for building a Docker (container) image. A Dockerfile is a script of all the commands, in order, required to build an image.

Code Block
FROM ubuntu:22.04 COPY . /app RUN make /app CMD python /app/app.py

The above Dockerfile shows that a container image is made up of layers:

FROM develops a layer from the ubuntu Docker image.
COPY creates a layer for making an application directory in the container and adding files from the current directory to this Docker app directory.
RUN layer builds your application.
CMD forms a layer for specifying what command to run within a container.

These different layers can have bugs and security vulnerabilities which can be targeted by hackers. The invaders can break out of container and get access to host machine (read data/files, kubelet configuration).

Container Image Vulnerabilities and Best practices:

Using code from unknown/untrusted sources which can include tools of OS or packages/libraries.
Image size can have an impact on its vulnerabilities. The smaller the image, the lesser the attack surface. Small and lightweight base images (e.g. Alpine} mean there are fewer packages installed, and only what is required for the app is installed, meaning you have strict control over and knowledge of what is installed and running inside the container.
Create a service user and run the container as a non-root.
Eliminate unnecessary dependencies from building app to deploying app(libraries/packages).
Continuous scanning and auditing of the image are beneficial for vulnerability detection, timely remediation, and compliance.

Container Image Scanners Recommendation:

Anchore is a container security platform that protects you at every step. It helps organisations ensure the security and compliance of containerised applications throughout the development and deployment lifecycle. It is designed to analyse and monitor container images for security vulnerabilities, policy violations, and compliance issues. Automates container vulnerability scanning. Anchore has an enterprise and open source version (syft and grype).

Key features:

Continuous Visibility: Generate and store Software Bill of Materials (SBOMs) at each step in the development process for a complete inventory of software components and dependencies.
Software Bill of Materials (SBOM Monitoring): Monitor the SBOM repository for continuous visibility into new vulnerabilities that arise — even post-deployment.
Continuous Security: Uncover vulnerabilities, secrets, and malware across the development process with fewer false positives.
Drift Detection: Track SBOM drift to detect suspicious activity, new malware, or compromised software in your builds.
Policy Enforcement: Ensure compliance with National Institute of Standards and Technology (NIST), Centre for Internet Security (CIS), Federal Risk and Authorization Management Program (FedRAMP), and other standards with prebuilt policy packs.
Remediation: Catch and fix security issues faster and easier with remediation recommendations.
Reporting: Give security teams insights across applications and teams with flexible dashboards and reports.
Trust and Assurance: Create and share application-level SBOMs to meet requirements of customers and US agencies.

Use Anchore When:

Comprehensive container security throughout the development lifecycle is a priority.
Fine-grained control over security policies and compliance is essential.

Conclusion:

Offers comprehensive security with features like continuous visibility, policy enforcement, and dynamic secrets.

Container Image scanner at RAL:

Harbor is an open-source container image registry that offers advanced features.

Key features:

access control: Define fine-grained access controls and Role-Based Access Control (RBAC) policies to regulate user permissions.
vulnerability scanning: Harbor includes a built-in vulnerability scanner to identify security issues within container images.
- Scan results: Harbor provides detailed scan results, including information about vulnerabilities, severity levels, and remediation suggestions.
- Automated scanning: Schedule automated scans to regularly check for new vulnerabilities in container images.
- Policy enforcement: Define security policies to automatically reject images with critical vulnerabilities, ensuring a proactive security approach.
security and compliance policies: Enforce security and compliance policies for images, ensuring adherence to organizational standards.
an immutable image repository: Ensure immutability by preventing the overwriting of existing images, maintaining version history.
audit logging: Harbor logs all user activities, providing an audit trail for security and compliance purposes.

By combining image management features with security functionalities, Harbor simplifies the process of maintaining a secure and compliant container image repository, aligning with best practices for container security.

Use Harbor When:

An open-source container registry with integrated scanning capabilities is needed.
Simplified container image management is a priority.

Conclusion:

Harbor: Primarily a container registry with built-in scanning, offering features like access control, vulnerability scanning, and policy-based replication.

Network Security:

Network security is challenging due to the complexity of K8s networking. By default, Kubernetes does not provide network isolation, allowing all communication between pods and other resources like other pods, namespaces, and IP address blocks (excluding traffic to and from the Pod's assigned node). Kubernetes objects known as NetworkPolicy deal with this problem.

NetworkPolicy:

A Kubernetes resource that allows you to define how pods are allowed to communicate with each other and other network endpoints.

Prerequisites:

Network policies are implemented by the Container Networking Interface (CNI) networking plugin.
CNI Plugin recommendation: Calico
- Calico is an open-source networking and security solution designed to facilitate seamless communication and secure networking within Kubernetes clusters.
- Efficient Networking: Calico uses Border Gateway Protocol (BGP) routing for efficient networking.
- Network Policy Enforcement: Allows the definition/creation and enforcement of network policies for controlling traffic between pods, enhancing security and isolation.
- Flexibility in IP Address Management: allowing for easy integration with existing network infrastructures without requiring an overlay.
- It is designed for high performance, offering low-latency networking suitable for demanding applications and workloads.
- It has a steep learning curve, may introduce resource overhead, and may require complex configurations for users with limited networking expertise.
- Calico also has limited built-in monitoring and may depend on BGP, which may be seen as a limitation in environments where BGP is not the preferred routing protocol.
- For more information contact: Jacob Ward, Tom Birkett.

Writing Network Policies

Define Network Policies using YAML file.
podSelector: mandatory and the pods that this policy will apply to (the policy target) which should be allowed as ingress sources or egress destinations.
policyType: is optional, although it is generally recommended to specify the kinds of policies that are included in this policy, ingress and/or egress.
- Pod isolation methods ingress and egress which can be manipulated by creating a NetworkPolicy.
- Ingress: permits incoming traffic to reach the target pods. It exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.
- Egress: permitted traffic leaving the target pods. By default, a pod is non-isolated for egress; all outbound connections are allowed.
- ipBlock: To specify which IP Classes Inter-Domain Routing (CIDR) ranges to allow as egress destination or ingress source. Pod IPs are unpredictable and fleeting, hence these should be cluster-external IPs.

Example of a NetworkPolicy:

Code Block

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

Best Practice:

Ensure all pods are covered by a NetworkPolicy.
Use precise Ingress/Egress target selectors.
Create a default deny policy before creating allow policies to ensure your cluster is fully covered by network policies, shielding new Pods from unintentional network exposure even if a dedicated policy is not set up.
Code Block
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny spec: podSelector: {} policyTypes: - Ingress - Egress
Use RBAC: Role-based access control (RBAC) is a crucial aspect of K8s network security, allowing administrators to regulate access based on roles. Users are created indirectly by importing a list of users to the cluster or generating client certificates for specific users. RBAC allows administrators to manage who has access to the Kubernetes API, specifying which operations a user, group, or service account can perform on which resources. Admins can choose from different authentication strategies, and the API server uses these methods to handle all requests. RBAC restricts threats' access to resources and reduces network-borne risks.
Avoid Default Ports: When deploying Kubernetes, services like API server, kubelet, kube-scheduler, kube-controller-manager, and etcd are often configured to listen on default ports 6443, 10250, 10259, 10257, and 2379-2380. Choosing custom ports for these Kubernetes services will reduce predictability.
Assign someone to review and update your policies on a regular basis.
To ensure that your policies are operating as intended, test them. Create a sample application, deploy K8s cluster, define and apply NetworkPolicy.
- Using ‘Kubectl exec’ to access pods and perform network connectivity tests.
- Illuminatio is an open-source tools used for testing Kubernetes Network Policy. With Illuminatio, you can check if your network policies are correctly configured and are successfully imposing the required network segmentation. Here's how Kubernetes Network Policies are used with Illuminatio:
  - Test Case Generation: Using your current Network Policies and pod settings, Illuminatio creates test cases. To determine whether traffic between distinct pods is permitted or prohibited, it generates a variety of scenarios.
  - Test Execution: Following the creation of test cases, Illuminatio sets up test pods and tries to connect them in accordance with the scenarios that have been specified. After that, it confirms if the traffic behaviour is consistent with the desired result that your network policies have defined.
  - Reporting on Test Results: Illuminatio offers thorough reports on the outcomes, emphasising any differences between the behaviour that was expected and what was observed. This aids in finding any gaps or incorrect configurations in your network policies.
- Utilise audit logs: Enabling and reviewing audit logs increases the likelihood of discovering potential network compromise activity. Regular reviews may take place weekly or monthly. Event-triggered reviews will require audit logs to be review more frequently like daily to evaluate the impact following such incidents.
- Encrypt communication: A specialised infrastructure layer called a Kubernetes service mesh is used to monitor, manage, and regulate communication between microservices running on a Kubernetes cluster. Service mesh runs on top of the network that a CNI plugin creates.
  - Some security features are:
    - Communication authorisation
    - Encryption
    - Authentication
  - Example:
    - Isito: an open-source service mesh platform that connects, secures, controls, and observes services within a Kubernetes cluster. It provides a set of tools for managing microservices and their interactions, offering features such as traffic management, security, and observability.
Network Policies Use cases:
- Managing database access: Network Policies enforce database access restrictions in Kubernetes, preventing other apps from communicating with the database server, ensuring it is only accessible by the app it's part of.
- Pod isolation: Block all ingress traffic to sensitive Pods in your cluster using a Network Policy to enhance workload security.
- Communication between specific namespace/app: Kubernetes namespaces facilitate communication between specific apps or namespaces, allowing for stronger multi-tenancy by network-isolating resources associated with different apps, teams, and environments.

Logging and Monitoring:

Monitoring security events, auditing logs for real-time detection and response to security incidents. Zeek, a passive, open-source network security monitoring (NSM) technology, is used by SCD/RAL to evaluate network traffic and offer insightful data about a network's security and performance. While Zeek is traditionally used in a variety of network security scenarios, including intrusion detection and traffic analysis, its integration with Kubernetes can enhance the monitoring and security capabilities of Kubernetes clusters.

Some Benefits of Zeek integration with Kubernetes:

Network Traffic Analysis: Zeek deployment in a Kubernetes cluster enables monitoring of traffic between pods, nodes, and outside entities. It offers insight into how microservices communicate with one another.
Security monitoring: Zeek aids in identifying security issues, detecting malware, and assessing potential threats to Kubernetes clusters by analysing traffic within a Kubernetes environment.
Incident response: Zeek in Kubernetes aids in investigating and responding to security incidents by providing a comprehensive view of network traffic, aiding in identifying the source and impact of security events.
Policy enforcement: Zeek could play a crucial role in enforcing network security policies within a Kubernetes cluster, ensuring compliance with predefined rules and identifying policy violations.
Visibility into Pod-to-Pod communication: Zeek can analyse intra-cluster communication, offering valuable insights into service dependencies, potential vulnerabilities, and abnormal behaviour.
For more information contact: Liam Atherton, James Acris and David Crooks

Secret Management

Secret management is crucial for securely storing, accessing, and managing sensitive information like passwords, tokens, and API keys to prevent misuse or unauthorised access. Kubernetes Secrets, a simple feature of K8s, are Base64-encoded representations of sensitive data managed within the K8s API and access control via RBAC. Best practices include secure and encrypting Secrets using tools like HashiCorp Vault, an open-source tool for centralized secret management. Vault reduces risk of breaches and data exposure through identity-based security automation and encryption-as-a-service. It is a powerful tool for organizations with complex and dynamic secret management requirements.

Key Features of Vault:

Dynamic Secrets: Vault can dynamically generate short-lived (created secrets can be configured to expire for any duration you specify), per-application or per-request credentials. Therefore, you can monitor who, when, and what a user is doing while also managing the spread/number of users and passwords around your organisation.
Encryption as a Service: Vault can be used to encrypt and decrypt data (the app can use Vault to encrypt data without having to worry about it themselves), offering encryption as a service.
Integration with Kubernetes: Vault has a Kubernetes authentication backend, allowing Kubernetes clusters to authenticate against it.
Fine-Grained Access Controls: Vault provides fine-grained access controls and audit logs.

Challenges:

Complexity: Setting up and configuring Vault can be more complex compared to using Kubernetes Secrets.
Operational Overhead: Requires additional infrastructure and operational overhead.

Choosing Between Them:

Use Kubernetes Secrets When:

Your secret management needs are straightforward.
You prefer native Kubernetes tooling.
Your use case doesn't require advanced features.

Use HashiCorp Vault When:

You need advanced secret management features.
Fine-grained access control and audit logs are critical.
Dynamic secret generation and encryption as a service are essential.
Require more resilience and control over sensitive data, allowing it to persist independently of the Kubernetes cluster or namespace. This means that even if you destroy the cluster or namespace, the secrets stored in Vault remain intact and accessible. Vault in a different cluster.

Kubernetes security holes:

Kubernetes, despite being a powerful and widely used container orchestration platform, is not immune to security vulnerabilities. Here are some common security holes that can affect Kubernetes clusters:

Denial of service (DoS):

A malicious attacker's actions can leave legitimate users or clients unable to access the system or service, creating a vulnerability. For instance, if someone were to make multiple simultaneous queries to your Kubernetes API server, in an attempt to overload the Kubernetes API server with a high volume of requests, the API server might stop responding to other legitimate requests. In addition, DoS attacks can target other components of the Kubernetes cluster, such as the networking layer or worker nodes, causing service disruption or resource exhaustion.

Privilege Escalation:

Security vulnerabilities in the Kubernetes API server itself, such as buffer overflows or injection attacks, can be exploited by attackers to execute arbitrary code or gain unauthorized access to the cluster. Any user can connect to a backend server through the Kubernetes application programming interface (API) server by crafting a specific network request. Once established, an attacker could submit any kind of request to the backend directly across the network connection. Even worse, these requests are verified using the Transport Layer Security (TLS) credentials of the Kubernetes API server. Also, vulnerabilities in Kubernetes API endpoints or components may allow attackers to escalate their privileges within the cluster. To reduce these vulnerabilities, administrators must often patch and upgrade Kubernetes components.

Privilege escalation can also occur within Kubernetes pods or containers. Container escape is a prevalent vulnerability in Kubernetes. An attacker that manages to access a privileged container or creates a new, privileged container will possess all the host's capabilities, allowing them to access host resources and potentially compromise other containers that are hosted on the same host. If taken advantage of, allows a hacker to access the host with higher rights. The privilege escalation flaw makes it possible for any user to gain full administrator privileges and gain unauthorized access to resources or compromising sensitive data. Strong isolation protocols, such PodSecurityPolicies, must be put into place to reduce the possibility of container escape attacks.

Versions Compared

Old Version 1

New Version Current

Key