Debloating

Bufferbloat

Bufferbloat is a widespread problem present throughout the Internet, "end-to-end." Debloating is a "work in progress" industry wide and will take years. Ultimately, all buffering/queuing in operating systems needs to be carefully managed and be automatically adaptive to the data transfer rates. All network routers (and operating systems!) should be running with better Packet Scheduling"> and AQM including home routers (SQM). Unfortunately, existing algorithms such as RED are difficult to tune, and unlikely to work correctly. Worse, those algorithms can make performance much worse if not tuned properly. Consequently, many network operators never used these techniques at all.

CeroWrt is the test platform for improved SQM algorithms. To achieve ultimate latencies under load across the high bandwidth variation of 802.11 and broadband, new SQM algorithms need testing in addition to more complex changes in internal buffering; these will take time and therefore debloating will be a work in progress for an extended period.

In the upstream direction, the bottleneck link may be adjacent to your home devices (e.g. your laptop on wireless), and in your operating system, outside of our control; you may see problems therefore copying from your home device upstream to the Internet and/or your home file server. Unfortunately, TCP acks can be stalled behind packets queued in a particular direction, so bufferbloat in one direction can result in bad performance (poor latency) in the other direction. If you run Linux, you can help with debloating by working with those working on the work going on at bufferbloat.net. On other operating systems, you should contact your operating system vendor and complain. Be gentle (but insistent), however: before 2011, bufferbloat was not understood to be a general problem, and it will take time to overcome.

Note that bufferbloat only occurs in the device just before the bottleneck in a path. A common strategy when fixes for bufferbloat are not available for the devices either side of a bottleneck, therefore, is to try to arrange to move the bottleneck from a device which is badly bloated to one which is not: e.g. you might arrange to ensure that your wireless bandwidth is always bigger than your broadband bandwidth (and using bandwidth shaping and QoS to avoid the consequences of bufferbloat in that hop as best you can). We have found, however, that many QoS systems are badly broken as they do not also do SQM.

Home Routers

Current operating systems has been tuned over the last decade for maximum bandwidth over very long, high bandwidth LFN ("Large Fat Network") paths without consideration of bufferbloat (gigabit speed + over 100ms or longer paths).

The quest for "land speed" record bragging rights has destroyed latency, the speed we really need. The bandwidth-delay product (BDP) of these LFN paths is very high indeed, and buffers have been adjusted upwards accordingly. When these same operating systems are used in home routers (e.g. Linux and Apple's system), LFN tuning becomes a serious headache as the circumstances differ tremendously.

Home routers never operate in LFN's, rather, they operate in two network environments, sometimes simultaneously:

  1. Relatively low bandwidth (100Mbps or much less), sometimes long paths (10-100ms), (your broadband connection) and
  2. Medium bandwidth (never much more than 100Mbps wireless, and never more than 1Gbps wired), but very, very short paths (in your house, to other devices such as file storage, less than 1ms away).

We can now manage the buffering in the router better now, since there are never high bandwidth, high delay paths through these devices. If we've done this correctly, you'll never see lower bandwidth, but will see much better latency. Please let us know if you see any performance problems.

Debloating in CeroWrt

In mid-2012, Kathy Nichols and Van Jacobson devised the CoDel ("controlled delay") algorithm for minimizing the delay of packets in queues waiting to be sent. Their initial work has been validated theoretically as well as by implementations in CeroWrt. The algorithm, and the combination with "Flow queueing" - fq_codel - as implemented by Eric Dumazet has also been incorporated into recent Linux kernels, where it is can be used as the default queue discipline.

fq_codel has nearly eliminated bufferbloat for all wired interfaces.

Wireless interfaces have a more complicated queueing structure, with packet aggregation that can confuse the CoDel algorithms. Although CeroWrt dramatically improves latency for wireless interfaces, more research is necessary for further improvements on wireless links.

Cerowrt also contains an implementation of the AQM PIE, which is the upcoming standard in DOCSIS 3.1