At the RSA Conference 2023, Yossi Weizman, Senior Security Researcher at Microsoft Defender for Cloud, shared with us the lateral movement of the Kubernetes (aka K8s) cluster and its impact on the cloud environment. Based on Yossi’s speech and NSFOCUS researchers’ understanding, this paper describes the use of lateral movement from the perspective of attack, and puts forward suggestions on monitoring and defending such activities from the perspective of defenders.
K8s is a portable and extensible open-source platform for managing containerized workloads and services. According to the report of stackshare, K8s has been used by more than 3,000 companies, including some well-known leading organizations. There is no doubt that K8s is one of the most popular container management platforms. In general, lateral movement refers to the process in which an attacker will continue to access or control other intranet machines using the captured host as a springboard after obtaining the control authority of an intranet machine.
In this article, three types of lateral movement— within the K8s cluster, between the cluster and the cloud, and cross-cloud — are introduced.
Note: Unless otherwise specified, all pictures are from the presentation of this speech.
K8s solves many common problems caused by container proliferation by classifying containers into pods. Pod provides a layer of abstraction for container grouping to help schedule workloads and provide necessary services such as network and storage for these containers. As the pod is the basic business production endpoint, it directly handles the interaction with users. Attackers can control the pod through various means (Web vulnerabilities, business logic errors, etc.). The background of this article assumes that in a K8s environment where the pod is controlled by an attacker, then how can an attacker expand its influence range and steal more valuable information through the types of lateral movements within the cluster, between the cluster and the cloud, and across the cloud.
Figure 1 K8s Architecture Diagram
Different Types of Lateral Movements
- Lateral movement within the cluster
When the attacker takes over the pod, the lateral movement in the cluster in K8s is characterized by RABC configuration and utilization. Role based access control (RBAC) is a method to regulate and control access to computers or network resources based on the role of users in the organization. The RBAC authentication mechanism uses the rbac.authorization.k8s.io API group to drive authentication decisions, allowing dynamic policy configuration through the K8s API. RBAC API declares four K8s objects: Role, ClusterRole, RoleBinding, and ClusterRoleBinding. Role is always used to set access permissions in a namespace; When creating a Role, you must specify the namespace to which the Role belongs. In contrast, ClusterRole is a permission control resource of a cluster scope. Role Binding is to assign the permissions defined in the role to one or a group of users. It contains a list of several principals (users, groups or service accounts) and references to the roles obtained by these principals. RoleBinding executes authorization in the specified namespace, while ClusterRoleBinding executes authorization in the cluster scope.
Therefore, attackers often use the trapped pod to access the key, take over the node, and even control the cluster, but the above operations become more and more difficult with the iterative update of K8s. However, improper permission configuration will allow lateral movement within the cluster. As shown in Figure 2, this pod has acquired the ability to update its own permissions, and the consequences of the fall are self-evident. In Figure 3, other similar sensitive permission assignments can also cause extremely serious consequences.
Figure 2 Inappropriate pod permission configuration
Figure 3 Some Sensitive Permissions
- Lateral movement from cluster to cloud
Why can clusters move to the cloud? Because cloud services or cloud resources are used in the cluster, such as calling image text resources in cloud storage, uploading data to cloud storage, or using cloud services to manage K8s, etc. This involves how the cloud authenticates K8s and how K8s interacts with the cloud, which can be classified into the following three categories:
- Store cloud credential AK/SK, long-term token or other equivalent identity credential information in the node.
- Cloud Instance Metadata Services (IMDS) : provides information about currently running virtual machine instances, which can be used to manage and configure virtual machines.
- Single sign on using the OIDC protocol: OpenID Connect (OIDC) extends the OAuth 2.0 authorization protocol, making it also available as an authentication protocol. OIDC can be used to enable single sign on (SSO) between applications that support OAuth through a security token called “ID token”.
In the first case above, at the RSA Conference 2023, Yossi introduced the use of Service Principal Names (SPN), that is, malicious pods create new backdoor pods and attach SPN information in the configuration of the backdoor pods; In the second case, considering that the node (kubelet) runs on the virtual machine environment, and the request sent through the pod is generally equivalent to the request sent by the node, this method can be used to obtain virtual machine information or request token to control the virtual machine; In the third case, the virtual machine resources on the cloud are controlled through SA and other sharing authentication mechanisms, and metadata information is obtained as shown in Figure 4, and the utilization example is shown in Figure 5.
Figure 4 Example of obtaining metadata information of virtual machine
Figure 5 Metadata Information Utilization
- Lateral Movement across the Cloud
Cross-cloud lateral movement involves some deployment architecture scenarios of K8s: K8s clusters in a multi-cloud environment, identity authentication in a multi-cloud environment, and supply chain attacks in a multi-cloud environment. Among them, K8s cluster in a multi-cloud environment involves cross-cloud service provider construction and operation cluster, and the more dangerous behavior is to expose cluster API to the public network; In a multi-cloud environment, identity authentication involves the authentication mechanisms of various cloud service providers, and whether there will be potential disclosure of authentication credentials or other sensitive information in their combination; Supply chain attack in a multi-cloud environment refers to the use of cross-cloud malicious components, malicious images, etc. to achieve lateral movement of cross-cloud service providers, as shown in Figure 6.
Figure 6 Example of cross-cloud horizontal movement
(1) On the K8s control plane, you can: use Kubeaudit to audit sensitive and suspicious behaviors (deploy images of non-credit sources; pods with abnormal behaviors, such as attaching sensitive storage; probe behaviors, such as viewing your own permission configuration; sensitive API calls, such as “get secret”).
(2) In the cloud service control plane, you can: monitor the use behavior of the K8s cluster’s cloud identity credentials (abnormal credential use, such as continuous use of credentials; suspicious cloud service calls, such as calling storage interfaces with key files or sensitive information).
- The overall idea of cloud and K8s: both cluster level and cloud level risks are considered;
- Credential is one of the keys to K8s security: use audit tools to monitor all authentication behaviors;
- Minimum authorization principle: authorization according to different business requirements of pod, node/master node;
- K8s ThreatMatrix: Timely follow up on the latest K8s risk matrix and its professional suggestions.