Deep Analysis of Memcached Large DRDoS Attacks – China Telecom DamDDoS & NSFOCUS Jointly Released

Recently, many domestic and foreign security companies and agencies issued warnings about the Memcached Distributed Reflection Denial of Service attack, which aroused the concern of all parties. According to our monitoring, the peak traffic for this attack has now reached 1.35T. On Feb. 27, Memcached’s reflection DDoS attacks ranged from hundreds of megabytes to a maximum of 500G. A few days later, the volume of attack spiked rapidly and doubled its own record. The attacks took place more tens of hundreds of times a day, showing explosive growth, which may lead to significant consequences.

Table of Contents

• 1. Event Review
• 2. Attack Analysis
  • 1 What is Memcached?
  • 2 Memcached distribution
  • 3 How does Memcached form a DRDoS attack?
  • 4 Memcached attack characteristics
• 3. Memcached Attack Defense Reinforcement Recommendations
  • 1 Memcached system self-examination recommendations
  • 2 Memcached attack traffic cleaning
  • 3 Memcached system protection and reinforcement

1.Event Review

According to China Telecom DamDDoS, attack monitoring data show that in as short as 5 days – from Monday to Friday (February 26 to March 2 at 06:00), there have been 79 cases using the Memcached protocol amplification attacks around the world. The total daily attack traffic reached 419TBytes.

Memcached Reflection Amplification DDoS Attacks – Day

Memcached Reflection Amplification DDoS Attacks – Total Traffic per Day

Among them, there are 68 attacks targeting Memcached in China, with attacks being frequent seen in Jiangsu and Zhejiang provinces. The maximum single attack against China peaks 505Gbps. The longest attack took place on March 1, lasting 1.2 hours, with a total attack traffic of 103.8TBytes.

Memcached Reflection Amplification DDoS Attacks in Various Provinces in China

In terms of impact, all Internet businesses may become targets of Memcached DRDoS attacks. Broadband service providers suffer from large traffic attacks, resulting in the outbound bandwidth fully occupied and the normal business not accessible, on the other hand, enterprise internal Memcached systems may be used by criminals and become an accomplice. We urge customers in all regions and industries to exercise vigilance and beware of Memcached Reflection attacks that directly impact the server or use the attacks as a cover-up to carry out other types of attacks along to further jeopardize information security.

2.Attack Analysis

2.1 What is Memcached?

Memcached is a high-performance caching system for open source distributed memory object and is mainly used to improve the scalability of web applications. It can effectively solve many problems of big data caches and is widely used worldwide. Memcached stores small pieces of data based on the key-value of the memory and uses the data to complete database calls, API calls, or page renderings. Attackers make use of the key-value function to create a large flow Memcached reflection attacks. This will be described later in details.

2.2 Memcached distribution

According to the latest statistics, a total of 3,790 Memcached servers are being utilized worldwide to participate in these Memcached reflection amplification attacks. These sources of reflection are distributed across 96 countries around the world. Among them, the United States accounted for 1/4 of the world total.

Distributed Memcached servers in China ranked second, accounting for 12.7%. The share of provinces in China is as follows, Guangdong, Beijing and Zhejiang are TOP3.

The statistics of the NSFOCUS Network Threat Intelligence (NTI) show that there are 104,506 Memcached servers worldwide at risk of being utilized. The distribution is as follows:

Source: NSFOCUS Threat Intelligence Center

Geographically, Memcached servers are the most available in the United States, followed by China.

Source: NSFOCUS Threat Intelligence Center

These active Memcached reflectors provide a powerful tool for building super volume DRDoS attacks. If no counter measures are taken in time, the number of attacks based on Memcached is expected to continue to increase, with serious consequences.

2.3 How does Memcached form a DRDoS attack?

The construction of Memcached reflection attacks is divided into the following three steps:

1.Collect reflector IP

Find the open Memcached system through NTI / Shodan and other search engines and further obtain the system IP;

2. Configure the reflector

Make use of the open Memcached system as a reflector and modify the key-value configuration to achieve large storage capacity for the purpose of constructing a reflection amplification attack;

3. Launch a refletion attack

The attacker fakes his IP address into the target address and sends a request to the Memcached reflector to read the information Memcached stored in the key-value. Memcached responds to the forged source IP upon receipt of the request, creating a reflection.

Memcached Reflection Attack Diagram

When a large number of Memcacheds are utilized simultaneously and reply with the same forged source IP, a heavy traffic DRDoS attack against this forgery source IP(victim) is easily formed.

2.4 Memcached attack characteristics

DRDoS (Distributed Reflective Denial-of-Service) is one type of DDoS attacks. NSFOCUS has featured in the DDoS Trend Report published in 2015, 2016 and 2017 (report download link http://www.nsfocus.com.cn/research/report_3.html ). The report clearly points out the popularity and level of harm of reflection attacks through actual use cases and data statistics. The outbreak of Memcached reflection attacks further shows that the heat of DRDoS attacks will continue.

Prior to this, the major types of DRDoS security vendors detected were mainly SSDP reflection, DNS reflection, NTP reflection. The following table (from US-Cert) lists the magnifications for various types of reflection attacks with more details:

Memcached reflection attacks are much more damaging than just other reflection attacks in terms of magnification, and data provided by US-Cert show that it achieves an astonishing 51,000x magnification.

How does Memcached achieve this magnification while compared with other reflection attacks? One of the important reasons is the key-value function of Memcached. As mentioned earlier, the role of key-value is to determine the size of storage capacity, under normal circumstances the value of key-value is usually not more than a few kilobytes. When Memcached is exploited by an attacker as a reflector, the value of key-value can be modified to more than 1 million bytes.

We replicated the whole process of the attack in our laboratory.

The first step, use the command to modify the key-value parameter on Memcached to raise the magnification.

send = “set t 0 900 1048501” + “\ r \ n” + ‘a’ * 1048501 + “\ r \ n”

socket.sendall (send)

As tested, the maximum value of key-value is 1048501.

The second step through the get command to read Memcached storage information, and get back to the target IP.

get = “\ x00 \ x00 \ x00 \ x00 \ x00 \ x01 \ x00 \ x00get t \ r \ n”

socket.sendto (get, (host, 11211))

The formed attack message is as follows:

The minimum size of request packets that trigger a Memcached reflection attack is 15 bytes, including 8 bytes (RFC defined) + 3 bytes (get) +1 (space) + 1 byte minimum (key name) +2 byte (\ r \ n), however the request data returns 105 million bytes, in theory, can be enlarged by as close to as 70,000 times. The powerfulness of the attack’s amplification capability forms a sharp contrast with other types of DRDoS attacks.

3.Memcached DDoS Attack Defense Reinforcement Recommendations

3.1 Memcached system self-examination recommendations

The formation of the attack provides us with a good sample of early warning, security products can detect the key-value configuration of the Memcached system before it is exploited as an attack source to intercept. The detection process is as follows:

(1) Check the TCP or UDP packets whose destination port is 11211 (ensure that it is a Memcached server).

(2) Check whether the message comes with a “set” command (set the command format, see Appendix), if so (3), or end the test;

(3) Check whether the value of the bytes field (the 1048501 marked in the figure below) after the set command exceeds the threshold value. If yes, you can suspect that the packet is abnormal.

(4) After detecting this kind of unusual control message, we can take the following two actions:

1. Alert and monitor. In order to prevent mistakes, it is recommended to monitor the flow changes behind the Memcached server to further determine whether the server is used as a reflector.
2. Discard directly. If there is sufficient evidence that the record was maliciously added, dropping directly ensures that the server is not being used as a reflector.

Self-examination example: Suppose the following is the data obtained from a sa public network Memcached server.

Most of the time, the value ​​is within 64K, but the last one reaches nearly 1M, with a significant difference from the others, basically we judge that there is a problem with the last record and that the server may have been exploited.

3.2 Memcached attack traffic cleaning

When facing such a large-scale threat of DDoS attacks, all network security nodes should enforce their defense and start with attack protection and outbound traffic cleaning to fully guarantee the security of infrastructure and business. In response to this attack, we provide the following protection advice:

• Operators

Operators and IDC are upstream of the network and possess strong bandwidth resources. They are the most immediate victims of attacks and the first level of protection. Operators can flexibly control routing policies and protection policies for fast filtering.

1) Configure UDP access control rules on the border to block or limit the packets with source port 11211;

2) Configure BGP FlowSpec policy on the border to block or limit UDP traffic from source port 11211;

3) Use the DDoS protection device to block or throttle UDP traffic of source port 11211.

• Enterprise Users

Enterprise users are usually close to the service terminals, familiar with the characteristics of their business traffic, making the policy configuration clearer and more flexible.

1) Large enterprise customers can adopt the same strategy with operators to configure UDP access control rules or BGP FlowSpec policies at the egress boundary device to block or rate UDP traffic with source port 11211;

2) An enterprise customer who does not have the export route configuration permission can configure a protection policy on devices such as anti-DDoS devices to block or rate UDP traffic whose source port is 11211;

3) Considering that the export bandwidth of an enterprise may be fully occupied, enterprise customers can perform a cleaning policy upstream through an operator or cloud cleaning service. The policy content also blocks or throttles UDP traffic with a source port of 11211.

3.3 Memcached system protection and reinforcement

For the users who are using Memcached system, we make the following suggestions in order to avoid being exploited by attackers, making Memcached the source of attacks, attacking traffic externally and affecting own system performance.

1) Configure a URPF policy on the border network device to filter outbound false source IP packets.

2) In-depth inspection before the Memcached system, directly filtering the over-large data packet in the third field of the set key 0 900 64000 in the message feature, so as to intercept the Memcached system before being modified to become the source of the attack;

3) Check Memcached service security to see if the Memcached service listens on the UDP port. Find Memcached process to see if there is a -l parameter, if not, set it as default – 0.0.0.0. If Memcached service does not need to listen to UDP, disable UDP. Details refer to Memcached official documents:

https://github.com/memcached/memcached/wiki/ConfiguringServer#udp

Appendix

The basic syntax of the “set” command is as follows:

set key flags exptime bytes [noreply]

value

The parameters are as follows:

key: Key in the key-value structure used to find the cached value.

flags: An integer parameter that includes a key-value pair that clients use to store additional information about key-value pairs.

exptime: The length of time the key-value pair is stored in the cache (in seconds, 0 means forever)

bytes: The number of bytes stored in the cache

noreply (optional): This parameter tells the server no need to return data

value: stored value (always in the second line) (which can be directly understood as the key-value structure value)