Container Security Protection – Orchestration Security
Orchestration Security
The maturity of the container technology pushes the development and implementation of microservices. More and more enterprises choose to adopt a mircoservice architecture to build their applications. Container orchestration tools are responsible for managing container clusters that carry various services. Arguably, it is container orchestration tools that support core services in a variety of projects adopting a mircoservice architecture. This document takes the most popular orchestration tool in the community, Kubernetes, as an example to describe security protection measures that container orchestration tools should take.
Kubernetes Security Protection
Both the Kubernetes community and third-party security management authorities have improved and enhanced the security of components and resources of Kubernetes. Main improvements are as follows:
1 Computing Resouce Security
The Kubernetes community provides a Pod Security Policy[i], which is a cluster-level resource that controls security sensitive aspects of the pod specification. The PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields. They allow an administrator to control the following:
Table 4.1 PodSecurityPolicy fields
| Control Aspect | Field Name |
| Running of privileged containers | privileged |
| Usage of host namespace | hostPID, hostIPC |
| Usage of host networking and ports | hostNetwork, hostPorts |
| Usage of volume types | Volumes |
| Usage of the host files ystem | allowedHostPaths |
| Whitelist of Flexvolume drivers | allowedFlexVolumes |
| Allocating an FSGroup that owns the pod’s volumes | fsGroup |
| Requiring the use of a read only root file system | readOnlyRootFilesystem |
| The user and group IDs of the container | runAsUser, supplementalGroups |
| Restricting escalation to root privileges | allowPrivilegeEscalation, defaultAllowPrivilegeEscalation |
| Linux capabilities | defaultAddCapabilities, requiredDropCapabilities, allowedCapabilities |
| The SELinux context of the container | SELinux |
| The AppArmor profile used by containers | annotations |
| The seccomp profile used by containers | annotations |
| The sysctl profile used by containers | annotations |
2 Cluster Security
Controlling access to the Kubernetes API
The API server uses the Transport Layer Security (TLS) protocol for all API traffic, so as to ensure the inter-service access security. It also provides such access control mechanisms as access control, access authorization, and admission control.
(2) Controlling access to Kubelet
Kubelets expose HTTPS endpoints which grant powerful control over the node and containers. By default, Kubelets allow unauthenticated access to this API. Production clusters should enable Kubelet authentication and authorization.
(3) Controlling the capabilities of a workload or user at runtime
- Limiting resource usage on a cluster
Resource quota limits the number or capacity of resources granted to a namespace. This is most often used to limit the amount of CPU, memory, or persistent disk a namespace can allocate, but can also control how many pods, services, or volumes exist in each namespace.
- Preventing containers running with root privileges
A pod definition contains a security context that allows it to request access to running as a specific Linux user on a node (like root), access to run privileged or access the host network, and other controls that would otherwise allow it to run unfettered on a hosting node. The pod security policy of Kubernetes can limit which users or service accounts can provide dangerous security context settings, for example, volume mounts.
- Restricting network access
The network policies for a namespace allow application authors to restrict which pods in other namespaces may access pods and ports within their namespaces. Quota and limit ranges can also be used to control whether users may request node ports or load balanced services. Additional protections may be available that control network rules on a per plug-in or per environment basis, such as per-node firewalls and physically separating cluster nodes.
- Controlling access to the pod
By default, there are no restrictions on which nodes pods may access. However, Kubernetes provides users with various policies for controlling the location of nodes where the pod runs, including NodeSelector, affinity, and anti-affinity policies.
(4) Protecting cluster components
- Restricting access to etcd
For APIs, write access to the etcd backend for the API is equivalent to gaining root on the entire cluster. Administrators should always use strong credentials from the API servers to their etcd server, such as mutual authentication via TLS client certificates. A recommended practice is to isolate the etcd servers behind a firewall that only the API servers may access.
- Enabling audit logging
The audit logger is a feature that records actions taken by the API for later analysis in the event of a compromise.
- Restricting access to alpha or beta features
Alpha and beta Kubernetes features are in active development and may have limitations or bugs that result in security vulnerabilities. Therefore, users are advised to disable features they do not use.
- Receiving alerts for security updates and reporting vulnerabilities
Those who join the kubernetes-announce[ii] group can receive emails about security announcements. See the security reporting page[iii] for more on how to report vulnerabilities.
- Encrypting etcd data
In general, the etcd database contains any information accessible via the Kubernetes API and may grant an attacker significant visibility into the state of clusters. Therefore, users are advised to always encrypt their backups using a well reviewed backup and encryption solution, and consider using full disk encryption where possible.
Kubernetes 1.7 contains encryption at rest, an alpha feature that will encrypt Secret resources in etcd, preventing parties that gain access to users’ etcd backups from viewing the content of those secrets. While this feature is currently experimental, it may offer an additional level of defense at critical times.
- Reviewing third-party integrations before enabling them
Many third-party integrations to Kubernetes may alter the security profile of clusters. Therefore, when enabling an integration, users should always review the permissions that an extension requests before granting it access. The pod security policy of Kubernetes helps improve the security.
- Rotating infrastructure credentials frequently
The shorter the lifetime of a secret or credential is, the harder it is for an attacker to make use of that credential. Setting short lifetimes on certificates and automating their rotation is a good way for security protection. Therefore, set short lifetimes on certificates for authentication.
3 Summary
Attention should be paid to the following risks during the use of the orchestration tool.
(1) Whether the authentication and authorization mechanisms are standards-compliant
An independent privilege control policy can be provided for each access interface of the orchestration tool. For example, for the Kubernetes’ access interface, API Server, since the communication between Kubernets components is all API-driven, restricting users to access the cluster and controlling operations allowed for users are the first line of defense. Much the same thing is found in Docker CLI.
(2) Whether the orchestration tool fully validates user inputs
A regular expression rule can be set for filtering user inputs, so as to block illegitimate content.
(3) Whether the orchestration tool thoroughly processes anomalies at runtime
The process of anomaly processing at runtime is essential for ensuring stable running of the orchestration tool. Currently, logs of components within the Kubernetes cluster are available for query in the case of an anomaly at runtime. If an application runs improperly in the cluster, troubleshooting should be conducted first from the perspective of the cluster and then the application itself.
(4) Whether the running environment is secure
The security of the external running environment is a prerequisite to that of the orchestration tool. The tools for protecting the security of the host system can be used to enhance the security of the external running environment. In addition, the container host should be hardened to mitigate host security configuration errors. The container engine should be updated to the latest version, so as to avoid damage caused to the host by vulnerabilities existing in old versions. Last but not the least, importance should also be attached to some privilege and audit issues.
(To be continued)
[i] Kubernetes Pod Security Policies, https://kubernetes.io/docs/concepts/policy/pod-security-policy
[ii] Kubernetes-announce, https://groups.google.com/forum/#!forum/kubernetes-announce
[iii] https://kubernetes.io/docs/reference/issues-security/security/
