Server bandwidth being saturated by a single IP address is a common and highly damaging problem in operations and maintenance. When a server's outbound bandwidth is instantly exhausted, all normal business traffic is affected, manifesting as website inaccessibility, soaring latency, API request timeouts, dropped SSH connections, or even complete server downtime. Attackers often use a large number of requests to consume bandwidth resources, causing the server to enter a "bandwidth exhaustion" state, resulting in service unavailability. To quickly identify which IP is saturating bandwidth and effectively block it, a multi-layered approach combining system tools, traffic monitoring, firewall policies, and log analysis is needed to rapidly mitigate damage and restore business stability.
Bandwidth saturation by a single IP address typically falls into two categories: one is offensive traffic, such as malicious scanning, CC attacks, abnormal request flooding, and port probing; the other is abnormal business requests, such as API spamming, batch API calls, excessive repeated crawling by certain web crawlers, or internal service errors leading to infinite retries. Whether it's an attack or a business anomaly, the core characteristic is that a specific IP or a range of IPs generates a large amount of traffic within a short period, completely saturating the outbound bandwidth. This results in extremely high server latency or even server outages. The key to identifying the source IP is to quickly analyze real-time traffic within the window of time when the server is still responsive.
When you suspect that bandwidth is being saturated by a particular IP, the most direct way is to use the iftop tool to check real-time bandwidth usage. iftop displays the traffic volume for each connection, as well as the source and destination IPs of the traffic. Use the command:
iftop -n
The `-n` parameter disables domain name resolution, allowing you to quickly locate IP addresses. In the iftop list, if an IP's send or receive bandwidth is significantly abnormal, such as consistently consuming 50%, 80%, or even 100% of the bandwidth, it's likely the source. The iftop interface allows sorting by RX, TX, and TOTAL, which helps identify the IP consuming the most bandwidth among a large number of connections.
If the server is temporarily slow and cannot be immediately displayed by iftop, you can use nethogs to check process-level traffic to determine if a program malfunction is causing the high traffic.
nethogs -d 2If the traffic is found to be originating from an internal process rather than an external IP address—for example, due to an API loop or a program logic error causing continuous data transmission—then the process should be dealt with immediately, rather than blocking the external IP address.
To further examine more detailed connection information, the `netstat` command can be used to analyze the current number of connections.
netstat -tunp | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head
This command displays the IPs with the most current connections. If an IP shows a large number of connections, especially exceeding hundreds or even thousands, it is likely initiating high-frequency access or attacks. Judging IP anomalies based on connection count is the most common method in CC attack troubleshooting. If an IP or network segment is found to have an extremely high number of connections, blocking should be initiated immediately.
Once a malicious IP address is identified, it needs to be quickly blocked using a firewall. For example, using iptables:
iptables -I INPUT -s 123.45.67.89 -j DROP
If the attack originates from a specific network segment, the entire IP segment can be blocked.
iptables -I INPUT -s 123.45.0.0/16 -j DROP
A server using nftables can perform the following:
nft add rule inet filter input ip saddr 123.45.67.89 drop
If firewalld is used, then it can be accessed via:
firewall-cmd --add-rich-rule='rule family="ipv4" source address="123.45.67.89" reject' --permanent
firewall-cmd --reload
After blocking an IP, monitor iftop, sar, or bandwidth again to confirm whether traffic has returned to normal. If bandwidth decreases and the server becomes smoother, it indicates that the source of the anomaly has been successfully blocked.
In some cases, malicious IPs will continuously change, such as in large-scale botnet attacks, proxy IP attacks, or spoofed IP attacks. In these situations, blocking a single IP is no longer effective, and a different strategy is needed, for example:
Limit on the number of connections:
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 50 -j DROP
Limit the request rate per IP:
iptables -A INPUT -p tcp --dport 80 -m limit --limit 30/s --limit-burst 50 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j DROP
These rules effectively prevent a single IP address from creating a large number of connections instantly, reducing the probability of bandwidth overload due to attacks.
If the attacker uses UDP flooding or ICMP flooding, it should be blocked through protocol restrictions:
iptables -A INPUT -p udp -j DROPHowever, caution is advised, as some services rely on UDP, such as VoIP and game servers.
High-frequency bandwidth attacks often require third-party scrubbing services for complete resolution, such as DDoS protected servers, CDN origin protection, and cloud firewalls. Especially when bandwidth attacks reach Gbps or even Tbps levels, a single machine cannot handle the load, necessitating diverting traffic to a professional anti-DDoS network where a scrubbing center filters out malicious traffic before it reaches the origin.
Besides real-time blocking, it's also necessary to analyze attack behavior records and examine log files, such as Nginx logs.
tail -f /var/log/nginx/access.logor Apache：
tail -f /var/log/httpd/access_logLogs can help determine if an IP is involved in a web crawler attack, repeatedly accessing a specific interface, or launching a high-concurrency attack on a specific URL. These records help analyze attack patterns, leading to more precise defense strategies.
To prevent similar problems from recurring, servers need an automated monitoring system. Automation can intercept traffic in the early stages of an attack, preventing bandwidth from being completely saturated. For situations where business interfaces are being flooded or program malfunctions cause bandwidth saturation, the logic needs to be optimized at the source, such as limiting web crawler access, adding anti-crawl mechanisms, setting API rate limits, caching duplicate requests, and adding CAPTCHAs.
In summary, quickly identifying IPs saturating bandwidth requires a combination of real-time traffic tools (iftop, nethogs), connection analysis tools (netstat, ss), log recording, and intelligent monitoring. After successful location, timely blocking using iptables, nftables, cloud firewalls, etc. In the long term, a combination of business anti-crawl measures, CDN protection, and high-defense resources should be used to prevent similar incidents from recurring. Bandwidth is an extremely precious resource for servers. Once it is fully occupied by attackers, business operations will be paralyzed instantly. Therefore, rapid investigation, immediate blocking, and long-term protection are all indispensable.
