In environments such as cloud servers, cross-border operations, or scenarios involving high-concurrency access, IT operations teams often encounter a puzzling issue: despite bandwidth not being fully saturated and CPU load appearing moderate, stress tests or peak business periods still result in network packet loss, latency jitter, and an increase in TCP retransmissions. Frequently, the root cause is not a bandwidth bottleneck, but rather an imbalance in the distribution of network interface card (NIC) receive interrupts. A single CPU core becomes overwhelmed by incoming packets, leading to queue overflows and subsequent packet loss. RSS (Receive Side Scaling) and RPS (Receive Packet Steering) are core mechanisms within the Linux kernel designed for load balancing on the network receive side. Proper tuning allows network traffic to be distributed across multiple CPU cores, alleviating interrupt overload on individual cores and effectively reducing packet loss caused by queue overflows.
Differences Between RSS and RPS
RSS (Receive Side Scaling) is a hardware-level feature of the NIC, where the NIC chip itself performs hash-based traffic distribution. Upon receiving a packet, the NIC calculates a hash based on a four-tuple (source IP, destination IP, source port, and destination port) to assign the flow to a specific hardware receive queue. Each queue is bound to a specific CPU core, which then handles the associated hardware interrupts. RSS relies on NIC hardware support, and the number of queues is limited by the hardware; in cloud environments, the number of RSS queues for a virtual NIC is typically fixed at startup. RPS (Receive Packet Steering) is a software-level mechanism implemented within the kernel. When a NIC has few hardware RSS queues—or lacks RSS support entirely—RPS distributes packets arriving at a single receive queue to other CPU cores for processing within a software interrupt (softirq) context. In short: RSS is a hardware-based distribution mechanism and should be the preferred choice; RPS serves as a software-based supplement to compensate for a lack of hardware queues. While they are often used together, their configuration logic remains entirely independent.
The primary cause of packet loss is closely linked to receive queues and the CPU's capacity to handle interrupts. NICs utilize a hardware receive ring buffer (rx ring buffer) to store incoming packets, which then triggers an interrupt to notify the CPU to retrieve them. If all network traffic is directed to a single CPU core, that core must simultaneously handle hardware interrupts, software interrupts, and application workloads. When the processing speed fails to keep pace with the rate of incoming packets, the hardware ring buffer quickly fills up, causing newly arriving packets to be dropped—a phenomenon known as "rx ring drop." For many cloud VPS users, an increase in `rx_dropped` or `rx_fifo_errors` (as shown in `ethtool` statistics) is a classic indicator of packet loss due to queue overflow. It is important to note that RSS (Receive Side Scaling) and RPS (Receive Packet Steering) only address packet loss caused by single-core overload on the receiving side. They are ineffective if packet loss stems from saturated egress bandwidth, firewall rate-limiting, poor network link quality, or undersized TCP buffers. Therefore, it is essential to identify the specific type of packet loss before attempting any tuning.
The first step is preliminary troubleshooting to determine whether the packet loss is caused by a receive queue overflow. Begin by using `ethtool` to examine the network interface's hardware statistics, focusing on metrics such as `rx_dropped`, `rx_fifo_errors`, and `overlimits`; a continuous rise in these values indicates an overload on the receiving side. Use `cat /proc/interrupts` to check the distribution of network interrupts; if the vast majority of interrupts are concentrated on a single CPU core while others handle almost none, it indicates a lack of traffic distribution. Next, check the current number of RSS queues using the command `ethtool -l eth0` (which displays both the maximum supported RX queues and the current configuration), and use `ethtool -S eth0` to analyze the packet distribution across each RX queue. If the number of hardware RSS queues is less than the number of available CPU cores, the hardware-based distribution has reached its limit, necessitating the use of RPS for software-based traffic distribution.
The second step involves tuning the RSS hardware queues. RSS takes precedence over RPS; whenever possible, hardware-based distribution should be used. You can adjust the number of RX queues using the command `ethtool -L eth0 rx N`, where `N` must not exceed the maximum number of queues supported by the network interface. Note that virtual network interfaces in cloud environments often have fixed limits, and in some virtualization setups, changes to the number of RSS queues may require shutting down the instance or restarting the network interface to take effect. It is advisable not to blindly set the number of queues to the maximum; instead, align the number of RX queues with the number of CPU cores allocated to the workload—for instance, configuring 4 RX queues for a 4-core instance. Once these adjustments are complete, configure interrupt affinity—either via `/etc/irqbalance.conf` or by manual assignment—to bind interrupts from different RX queues to distinct CPU cores. This prevents multiple hardware queues from contending for the same core. A critical caveat: since RSS relies on a 4-tuple hash, all packets from a single TCP flow will consistently map to the same queue and the same CPU. Consequently, high-bandwidth single-stream connections (such as large file transfers) cannot be distributed across multiple cores via RSS; RPS is similarly ineffective for splitting individual flows. In such cases, you must instead adjust the ring buffer size.
The third step involves configuring RPS (Receive Packet Steering) software-based traffic distribution. The primary configuration file for RPS is located at `/sys/class/net/eth0/queues/rx-*/rps_cpus`. This file contains a bitmask that specifies which CPU cores are permitted to process softirqs for packets arriving on that particular queue. For instance, on a 4-core system, a mask of `0xf` allows packets to be distributed across CPU0 through CPU3. It is generally recommended to apply the same CPU mask to all RX queues, ensuring coverage across all designated processing cores. A related parameter, `rps_flow_cnt`, defines the number of entries in the RPS flow table; a value approximately one-quarter of the RX ring buffer size is recommended. Setting this too low can lead to hash collisions and ineffective traffic distribution, while setting it too high consumes unnecessary kernel memory. An example command to modify this is: `echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus`. Note that RPS introduces a slight kernel CPU overhead; therefore, it should not be enabled indiscriminately on low-load or small-scale instances, where the additional softirq overhead may outweigh any potential benefits.
Additional tuning parameters work in tandem with RSS/RPS. The hardware RX ring buffer size can be increased—for example, using `ethtool -G eth0 rx 4096`—to provide the CPU with more buffering time, thereby helping to mitigate packet loss during transient traffic bursts. Additionally, tune the kernel network parameters: `net.core.netdev_max_backlog` (the kernel's receive backlog queue) should be increased moderately, especially in scenarios utilizing RPS. Other parameters, such as `net.core.somaxconn` and the TCP memory buffers (`tcp_rmem`, `tcp_wmem`), should be adjusted based on specific requirements. Regarding the `irqbalance` service: if you have manually configured RSS interrupt affinity, you should disable `irqbalance` to prevent it from overriding your manual settings. Conversely, if you prefer not to manage affinity manually, keep `irqbalance` enabled to allow for automatic load balancing. Note that any RPS configurations made via `sysfs` will be lost upon restarting the network interface or the server; therefore, these settings must be persisted by adding them to `rc.local` or a `systemd` script.
Tuning must always be validated; never deploy changes directly to production without verification. Validation should be conducted in three stages: First, examine the interrupt distribution (using `watch cat /proc/interrupts`) to ensure that network interface interrupts are evenly spread across multiple CPUs. Second, monitor packet loss metrics—specifically, observe `rx_dropped` via `ethtool` to confirm that the count is no longer increasing. Third, perform stress testing—using tools like `iperf` or `tcpreplay`, or by subjecting the system to actual production traffic—to evaluate retransmission rates, latency, and application-level error counts. It is also essential to maintain a control group and record metrics such as software interrupt (`si`) usage, CPU utilization, and packet loss counts both before and after tuning. A common pitfall is continuing to tweak traffic distribution parameters when packet loss fails to improve after RSS/RPS tuning; in such cases, you should instead investigate other potential bottlenecks, such as egress constraints, firewall QoS settings, rate limiting imposed by the cloud service provider, or packet loss occurring on the network link itself.