Skip to content

Network Tuning

Optimizing TCP/IP Parameters for High-Throughput Networks

High-performance networks require fine-tuning of TCP/IP stack parameters to reduce latency, increase throughput, and adapt to workload characteristics. This section covers critical sysctl parameters and configuration strategies for optimizing network performance.

TCP Parameter Tuning

Key TCP parameters control congestion control, retransmission behavior, and buffer sizes. Use sysctl to adjust these dynamically:

# Enable TCP window scaling (required for high-throughput)
sudo sysctl -w net.ipv4.tcp_window_scaling=1

# Increase TCP receive buffer size (in bytes)
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

# Optimize keepalive settings
sudo sysctl -w net.ipv4.tcp_keepalive_time=600
sudo sysctl -w net.ipv3.tcp_keepalive_intvl=30
sudo sysctl -w net.ipv4.tcp_keepalive_probes=5

# Reduce TIME_WAIT state duration (use with caution)
sudo sysctl -w net.ipv4.tcp_tw_reuse=1
sudo sysctl -w net.ipv4.tcp_tw_recycle=1  # Deprecated in Linux 4.12+; use tcp_tw_reuse instead

Notes:
- tcp_rmem/tcp_wmem define minimum, default, and maximum buffer sizes. Larger values improve throughput but consume more memory.
- tcp_keepalive_* settings help maintain idle connections but may increase latency for short-lived connections.
- Avoid over-allocating buffers; monitor memory usage with free -h or vmstat.

Congestion Control Algorithms

Linux supports multiple congestion control algorithms (CCIDs) that impact network fairness and throughput. Choose the optimal algorithm based on your use case:

# List available congestion control algorithms
sysctl -n net.ipv4.tcp_congestion_control

# Switch to Cubic (default in most Linux distributions)
sudo sysctl -w net.ipv4.tcp_congestion_control=cubic

# Switch to BBR (requires kernel 4.9+ and CONFIG_NET_CLS_ACT)
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

Algorithm Overview:
- Cubic: Balanced performance for high-speed networks; widely supported.
- BBR (Bottleneck Bandwidth and Round-trip propagation time): Optimizes for high throughput and low latency in modern networks.
- DCTCP (Data Center TCP): Designed for low-latency, high-bandwidth data center environments.

To persist changes, add the tcp_congestion_control parameter to /etc/sysctl.conf.

Packet Handling and Kernel Configuration

Optimize packet processing by adjusting kernel-level buffer limits and enabling offload features:

# Increase maximum receive and send buffer sizes
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216

# Enable large receive offload (LRO) and generic segmentation offload (GSO)
sudo sysctl -w net.ipv4.tcp_low_latency=1
sudo sysctl -w net.ipv4.tcp_mtu_pro Probe=1

Kernel Module Parameters:
- LRO (Large Receive Offload): Combines incoming packets into larger buffers for reduced CPU overhead.
- GSO (Generic Segmentation Offload): Offloads segmentation to the NIC, improving throughput for high-bandwidth applications.

Verify NIC capabilities with ethtool:

ethtool <interface>

Troubleshooting and Validation

After tuning, validate changes using performance tools:

# Test throughput with iperf3
iperf3 -c <server_ip> -P 100  # 100 parallel streams

# Monitor active TCP connections
ss -antp

# Check for packet retransmissions
tcpdump -i <interface> port 80 -nn

Logs and Diagnostics:
- Check /var/log/messages or dmesg for kernel warnings related to network performance.
- Use netstat -s to analyze TCP statistics (e.g., retransmissions, collisions).

Key takeaways

  • Tune TCP buffer sizes (tcp_rmem, tcp_wmem) to match application demands.
  • Select congestion control algorithms (e.g., BBR) based on network latency and throughput goals.
  • Enable NIC offloads (LRO/GSO) to reduce CPU overhead for high-throughput workloads.
  • Validate changes with tools like iperf3 and monitor TCP statistics for anomalies.
  • Always test in controlled environments before applying changes to production systems.