100daysofhomelab

Bufferbloat (Day 20)

Vince

• 2 min read
Bufferbloat (Day 20)
Photo by Arthur Hinton / Unsplash

While browsing and editing large RAW photos over SMB, I noticed some high kind of latency and got asking if this could be reduced.

Some research later i found something called bufferbloat.

What's Bufferbloat?

I found this analogy of bufferbloat from Waveform.com that's worth sharing here:

Think of your internet connection like a sink with a narrow drain (your bandwidth limit). When someone downloads a large file, it's like dumping a bucket of water into the sink. Now if you try to do something time-sensitive - like gaming or a video call - those packets are like drops of oil trying to get through a sink full of water. They have to wait for all that "water" to drain first, causing lag and delays. That's bufferbloat.

Check out Waveform's bufferbloat test tool, and a more detailed ELI5 explanation.

In OPNsense, we can address this using traffic shaping - setting up pipes and queues with FlowQueue-CoDel, it ensures that packets from small flows are sent in a timely fashion, while large flows share the bottleneck’s capacity.

Preliminaries

I initially found guides for pfSense, but OPNsense has its own really nice guide on how to address bufferbloat here

Before messing around with creating pipes, queues, and rules, it's advisable to ran some tests to establish a baseline.

Before Optimization

My initial bufferbloat grade was a B, with some concerning latency spikes:

  • Download speed: 712.9 Mbps
  • Upload speed: 654.1 Mbps
  • Latency under load: +28ms download, +48ms upload

as seen in the screenshot below:

After Optimization

After setting up and tuning the traffic shaping rules:

  • Download speed: 588.8 Mbps
  • Upload speed: 516.9 Mbps
  • Latency under load: +12ms download, +14ms upload
  • Bufferbloat grade improved to A

as seen in the screenshot below:

So traded some raw speed for consistency.

Internal Network Optimization

After seeing improvements on the WAN side, I got more specific with my internal network.

I set it up for:

  • SMB traffic (port 445) where I needed lower latency for raw files
  • Specific devices on certain VLANs that needed more controlled latency

I really just needed it for low latency especially when editing raw files attached on over smb and it did help, not a drastic difference but something noticeable, and so I disabled the WAN side optimizations.