Technical Report on Container Security (III)-1

Security Risks and Challenges — Vulnerability and Security Risk Analysis

Vulnerability and Security Risk Analysis

As a specific implementation of the container technology, Docker is getting more and more popular in recent years. To some extent, Docker has become a typical representation of the container technology. Docker is based on the common client/server (C/S) architecture design as shown in Figure 3.1. In this architecture, the Docker daemon (server) runs on the Docker host and the Docker client sends requests to the daemon. This chapter uses Docker as an example to describe containers’ vulnerabilities and security risks.

Figure 3.1 Docker’s C/S architecture

• Software Risks

Docker surely contains security risks in its software design and code implementation. This section analyzes security risks in Docker from two aspects: software design and code vulnerabilities.

• Software Design

Docker, though implementing excellent operating system-level isolation in design, contains security risks in multiple aspects such as its default networking mode, sharing of the operating system kernel with the host, sharing of host resources, use of the Linux capabilities mechanism, and insufficient isolation. The following describes attacks that could result from security risks in Docker.

1. Local area network (LAN) attack

As for the network implementation, Docker supports multiple networking modes. In default bridge mode, containers created on the same host will all connect to the docker0 bridge. This means that those containers constitute an LAN. In this case, LAN-targeting attacks like ARP spoofing, sniffing, and broadcast storm will pose security threats to the containers. Therefore, if more than one container is deployed on the same host, you need to perform proper network configuration and set access control rules to isolate networks with borders.

2. Denial-of-service (DoS) attack

Container instances share resources of the host, including the underlying CPU, memory, and disk resources that are scheduled and allocated by the host’s operating system in a centralized way. If resources used by containers are not properly restricted and managed, a disparity may exist in the use of resources among containers, causing exhaustion of resources of hosts and the cluster in worst-case scenarios and finally leading to a denial of service.

3. Vulnerable system call

The major difference between Docker and virtual machines (VMs) is that Docker shares the operating system kernel with the host. If the host kernel contains a horizontal unauthorized access vulnerability or privilege escalation vulnerability, attackers compromising containers can still be able to exploit such vulnerability to escape to the host for malicious operations, though containers run in non-privileged mode.

4. Root privilege sharing in privileged mode

Now that the containers share the same operating system kernel with the host, when containers run in privileged mode, root users of the containers have root access to the host and can nearly perform operations without limit.

5. Non-isolated file system

Docker has isolated the file system, but fails to isolate important system files such as /sys, /proc/sys, and /proc/bus.

Docker has protection solutions in place to deal with the preceding attacks. Therefore, such attacks can be prevented as long as proper protection configuration is performed. For details, see the related official documentation[i].

• Software Vulnerabilities

According to statistics of China National Vulnerability Database of Information Security (CNNVD[ii]), by July 31, 2018, a total of 38 vulnerabilities had been discovered in Docker which is integrated in open-source projects or products from vendors such as Cisco, Boot2Docker, and Jenkins.

Figure 3.2 shows the number and percentage of Docker vulnerabilities discovered in recent years. We can see that each year witnessed a certain quantity of Docker vulnerabilities announced and there is no sign of decrease in vulnerabilities, indicating that this software still contains security risks after years of development.

Figure 3.2 Percentage of Docker vulnerabilities in recent years[1]

Of 33 vulnerabilities assigned a risk level, more than 90% are medium-risk or above and 36% high-risk and critical. Four critical vulnerabilities have a CVSS score of 10, including two respectively assigned CVE-2014-9357 and CVE-2016-9223. The CVE-2014-9357 vulnerability exists in Docker 1.3.2 and allows a remote attacker to execute arbitrary code with root privileges via a crafted image or “build” in a Dockerfile in an LZMA (.xz) archive. The CVE-2016-9223 vulnerability exists in the Docker Engine configuration of Cisco CloudCenter Orchestrator due to a misconfiguration. A remote attacker could exploit it to deploy Docker containers with high privileges on the affected system. Figure 3.3 shows the distribution of all Docker vulnerabilities in terms of risk level.

Figure 3.3 Distribution of Docker vulnerabilities[2] by risk level

Figure 3.4 shows the distribution of security risks that could be incurred by all vulnerabilities, including privilege and access control, rear links, resource management errors, and input validation. For example, the vulnerability (CVE-2016-8867) in Docker Engine 1.12.2 could pose privilege and access control threats. This vulnerability exists because Docker Engine enables ambient capabilities with misconfigured capability policies. An attacker, via a malicious image, could bypass user permissions to access files within the container file system or mounted volumes.

Figure 3.4 Impacts of Docker vulnerabilities[3]

• API Security

According to the architecture, Docker listens on a UNIX socket by default, for example, unix:///var/run/docker.sock. For cluster management, Docker also provides a REST API for remote management, allowing remote access through TCP.

For example, when Docker Swarm is used, the Docker node will open TCP port 2375 which is bound to the IP address 0.0.0.0. Running as a daemon in the background, Docker, by default, exposes the non-encrypted listening port 2375. Besides, a user, when executing the startup parameter, dockerd -H=0.0.0.0:2375 -H unix:///var/run/docker.sock, can enable Docker to listen on port 2375 on all local addresses. In this way, the Docker daemon can execute Docker command requests it receives, for example, executing the docker –H tcp://$HOST:2375 ps command to retrieve all Docker instances on the $HOST host. Now that the user can execute the docker ps command, he or she can also run commands such as docker run and docker rm.

However, enabling the Docker Remote API without any encryption and access control is very risky. In particular, once spotting the default port 2375 exposed on the Internet, an attacker can gain access to container data without authentication, resulting in sensitive information disclosure. Also, the attacker can delete data on containers or directly access sensitive information on the host by taking advantage of characteristics of containers. Doing so, the attacker may gain the server’s root privileges to manipulate sensitive files and finally take full control of the server.

In May to July, 2018, we used NTI[iii] to search for all port 2375 on the network and found that 337 IP addresses bound to this port were exposed on the Internet. Figure 3.5 shows the global distribution of hosts with port 2375 exposed. This reflects the popularity of Docker, but also reveals that this program is not used in a proper way and relevant operations need to be regulated to prevent security risks.

Figure 3.5 Global distribution of hosts with port 2375 exposed

Figure 3.5 shows geographical statistics of 337 exposed IP addresses. Globally, most of the exposed Docker services were found in three countries: China ranked first with 197 (52%) exposed IP addresses, followed by USA (65 or 17%) and Germany (19 or 7%).

Figure 3.6 shows the distribution of Docker hosts exposed in China. Zhejing took the largest proportion (29%) with 57 exposed IP addresses and Beijing and Guangdong came in second and third respectively with 43 (22%) and 35 (18%) exposed addresses.

Figure 3.6 Distribution of port 2375 exposed in China

Figure 3.7 shows domain name statistics of Docker’s 337 IP addresses, including IP addresses of certain known public cloud vendors.

Figure 3.7 Distribution of domain names of exposed Docker IP addresses

It should be noted that port 2375 exposure on the Internet is a kind of misconfiguration rather than a Docker vulnerability. For this point, an official Docker document gives a warning that binding Docker to a TCP port accessible from the Internet allows a malicious user to obtain root privileges of the Docker host. The warning is as follows:

Warning: Changing the default docker daemon binding to a TCP port or Unix docker user group will increase your security risks by allowing non-root users to gain root access on the host. Make sure you control access to docker. If you are binding to a TCP port, anyone with access to that port has full Docker access; so it is not advisable on an open network.

Apart from port 2375 for remote access to the Docker daemon, we also analyzed Kubernetes exposure in July 2018. Scanning the entire Internet for port 6443 (default SSL port of the Kubernetes API server) by using NTI, we found that a total of 12,803 Kubernetes services were exposed on the Internet. Figure 3.8 shows the global exposure of Kubernetes services.

Figure 3.8 Global exposure of the Kubernetes service

Figure 3.9 Global distribution of managed service providers with Kubernetes exposed

Like port 2375 for remote access to the Docker daemon, the Kubernetes service was mainly exposed in the USA, China, and Germany. The USA saw the most Kubernetes services (4886 or 38%) exposed and China (2582 or 20%) and Germany (1423 or 11%) respectively came in second and third. Most of those exposed Kubernetes services are deployed on public clouds such as Amazon Web Services (AWS) and Aliyun.

Figure 3.10 Distribution of exposed Kubernetes services in China

Figure 3.11 Distribution of managed service providers that have Kubernetes exposed in China

As shown in Figure 3.10, Kubernetes hosts in China were mostly exposed in Beijing, Zhejiang, and Guangdong. Beijing topped the ranking list with 695 (27%) exposed hosts, followed by Zhejiang (635 or 25%) and Guangdong (274 or 11%).

From the above, we can see that container orchestration systems have developed rapidly at home and abroad, and due to container deployment characteristics, the exposure distribution of containers deployed on public clouds, data centers, and orchestration system services is in line with the development trend of container services and cloud services. As containers and container orchestration systems are deployed in large quantity, numerous services are exposed on the Internet, broadening the attack surface. To make matters worse, a large portion of such services lack basic security configurations and hardening measures and can therefore easily be exploited by attackers to compromise the availability and integrity of systems or by third parties to launch a variety of attacks.

• Insecure Images

Developers often download images from the official Docker Hub repository, of which some are from official organizations developing certain software in images and a lot are contributed by third-party organizations and even individuals. Throughout the application lifecycle, developers, testing personnel, and O&M personnel will download and run images as required. Therefore, it is really important to check images before the container runs.

Apart from Docker Hub, there are a great number of third-party image repositories such as China’s NetEase Cloud[iv] (registration is required), USTC Mirror[v] (public and open), DaoCloud[vi] (registration is required), and Aliyun[vii]. These third-party image repositories make it convenient to obtain images, but also bring potential security risks. For example, downloading software included in images is insecure because possibilities exist that the downloaded images contain malicious backdoors or are tampered with in transit.(1) Some images contain vulnerable software.

A relevant research report[viii] shows that more than 30% of official images in Docker Hub contain high-risk vulnerabilities and almost 70% of the total have high or medium-risk vulnerabilities. We selected 10 images with highest downloads and evaluations from Docker Hub and scanned their latest versions with Clair[ix]. The following table presents our analysis results.

Table 3.1 Vulnerability statistics of some images in Docker Hub

As shown in the preceding table, a great majority of images contain high-risk vulnerabilities and some even have dozens. Vulnerable images mainly belong to application software, while mainstream operating systems are relatively secure because of proper vulnerability management. For example, no CVE vulnerabilities are found in CentOS and Alpine.

(2) Some malicious images are uploaded by attackers.

If hackers install malware such as trojans and backdoors in their created images before uploading such images to public repositories like Docker Hub, users may have an insecure container environment from the day they use such images, let alone days afterward.

For example, Docker does not handle images in a secure way as it supports three compression algorithms, Gzip, Bzip2, and XZ. Gzip and Bzip2 are relatively secure, owing to the use of the Go standard library. XZ is built upon the open-source XZ Utils written in C, instead of the native Go, and therefore has a possibility of being maliciously written by hackers. Once the malicious writing occurs, XZ will be potentially vulnerable to arbitrary code execution. In fact, as long as there is a vulnerability in XZ, a user is likely to crash the system when executing the docker pull command to pull images.

For specific files, you can typically use antivirus (AV) software to check whether they are secure. However, AV software currently lacks sufficient support for image scanning. If users would like to check the security of images to be downloaded, all they can do is determine if backdoors exist (for example, run images and install AV software to scan them) and make sure that their requests are directed to official images.

(3) Some images are tampered with via man-in-the-middle attacks.

If users download images from the image repository in a non-encrypted way, these images in transit may be tampered with through man-in-the-middle attacks. To address this issue, Docker 1.8 provides the content trust mechanism.

• Container Isolation Losing Effect

As containers share the operating system kernel with the host, security risks may be incurred once container isolation loses effect.

1. Once the attacker successfully compromises the operating system kernel used by containers, he or she is able to access the file system on the host or break into other containers, rendering container isolation void.
2. The host’s file system can be mounted to the directory of more than one container so that they can access this file system. An attacker, once gaining access to one container and learning the file system directory, can get into file systems of other containers so as to finally cause information disclosure, content defacement, or other security issues.

(To be continued)

[1] Data as of July 31, 2018

[2] Data as of July 31, 2018

[3] Data as of July 31, 2018

[i] Docker Documentation, https://docs.docker.com/

[ii] CNNVD, http://www.cnnvd.org.cn/index.html

[iii] NTI, https://nti.nsfocus.com/

[iv] NetEase Cloud, https://www.163yun.com/product/repo

[v] USTC Mirror, http://mirrors.ustc.edu.cn/

[vi] DaoCloud, https://www.daocloud.io/account/signup

[vii] Aliyun, https://opsx.alibaba.com/mirror

[viii] Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, https://www.banyanops.com/blog/analyzing-docker-hub/

[ix] CoreOS Clair, https://github.com/coreos/clair