Yamux Memory Exhaustion Vulnerability via Active::pending_frames property

Summary

Attack scenario
The Rust implementation of the Yamux stream multiplexer uses a vector for pending frames. This vector is not bounded in length. Every time the Yamux protocol requires sending of a new frame, this frame gets appended to this vector. This can be remotely triggered in a number of ways, for example by:

1. Opening a new libp2p Identify stream. This causes the node to send its Identify message. Of course, every other protocol that causes the sending of data also works. The larger the response, the more data is enqueued.
2. Sending a Yamux Ping frame. This causes a Pong frame to be enqueued.

Under normal circumstances, this queue of pending frames would be drained once they’re sent out over the network. However, the attacker can use TCP’s receive window mechanism to prevent the victim from sending out any data: By not reading from the TCP connection, the receive window will never be increased, and the victim won’t be able to send out any new data (this is how TCP implements backpressure). Once this happens, Yamux’s queue of pending frames will start growing indefinitely. The queue will only be drained once the underlying TCP connection is closed.

Components
https://github.com/libp2p/rust-yamux/blob/yamux-v0.13.1/yamux/src/connection.rs#L289

Details

This attack is inspired by the HTTP/2 Rapid Reset Attack (CVE 2023-44487), HTTP/2 Ping Flood (CVE-2019-9512), and the QUIC Path Validation attack (see my blog post: https://seemann.io/posts/2023-12-18-exploiting-quics-path-validation/).

Impact

An attacker can cause a remote node to run out of memory, which will result in the corresponding process getting terminated by the operating system.

Depending on the application protocols running on top of rust-libp2p, higher amplification factors are possible. For example, image a protocol that sends out 10 MB of data as a result of an incoming request. By issuing that request and sending a Yamux stream window update (together ~100-200 bytes), the victim would now enqueue the entire 10 MB into its frame buffer. Any block transfer / sync protocols might be good candidates.

In addition to consuming huge amounts of memory, this attack also drives up the victim’s CPU load, such that the allocation of memory at some point becomes CPU-limited.

This was originally submitted by @marteen-seemann to the Ethereum Foundation bug bounty program.