Home Networking - Part 4 - Smart Queues Management (SQM)

  • Thursday, Oct 1, 2020

So, you’ve read through both parts of my guide to UK broadband , as well as the previous three parts of my guide to home networking  (and if you haven’t, it’s well worth it before continuing to read here, as a lot of the terminology I mention below is explained in more detail there).

But, let’s face it, as we saw in Part 1  of my guide to UK broadband, however good your home network setup is, our broadband infrastructure is simply not very good. This has become even more of an issue since the pandemic - residential UK broadband has never seen such high demand. Is there anything else that can help?

Well, the answer is yes and no. No doubt, the UK’s broadband networks are struggling to keep up with demand, and reliability and speeds can plummet as a result. This is a problem they have to deal with (and the consumer can do very little about). But there is one thing you can do to at least try to improve what you do have - and that is Smart Queues Management (SQM).



What is “BufferBloat”?


Nobody likes a stuttering internet connection

When you signup to an internet provider, you choose a package which gives you a certain speed for downloads and a certain speed for uploads. When one person is using that connection on their own, then they generally always get this speed (with fluctuations due to issues with the network infrastructure and/or provider) all to themselves. But this type of usage is very uncommon now - most households have a number of people all sharing this one connection. So what happens when everybody uses the connection at once and you run out of bandwidth? The answer is BufferBloat.


A DSL Reports speed test showing BufferBloat rating

Here’s a great analogy from a blog article called “Bufferbloat and the Ski Shop” :

Imagine a ski shop with one employee. That employee handles everything: small purchases, renting skis, installing new bindings, making repairs, etc. He also handles customers in first-come, first-served order, and accepts all the jobs, even if there’s already a big backlog. Imagine, too, that he never stops working with a customer until their purchase is complete. He never goes out of order, never pauses a job in the middle, not even to sell a Chapstick.

That’s dumb, you say. No store would do that. Their customers – if they had any left – would get really terrible service, and would never know when they’re going to be served. And you would be right.

The Ski-Shop Router

Unfortunately, a lot of network routers (both home and commercial) work just like that fictitious ski shop. And they give terrible service. These routers have a single first-in, first-out queue.. All traffic is queued, regardless of whether it’s big or small, whether there has been a lot of traffic from a particular source, or whether things are getting backed up.

Since the router has no global knowledge of what’s happening, it cannot inform a big sender to slow down, or throttle a particular stream of traffic by discarding some data (in the ski shop analogy, sending away a customer who has another long repair job). The dumb router simply accepts the data, buffers it up, and expects to send it sooner or later. To make matters worse, in networking (but not in ski shops), if delays get long enough, computers can resend the data, thinking that the original transmission must have been lost. These retransmissions further increase bloat and delay, because there are now two copies of the same data buffered up, waiting to be sent…

This is the genesis of the name “bufferbloat” – the router’s memory gets bloated with buffers… When the router doggedly determines to send that data, it blocks newer sessions from even starting, and the entire network gets slow.

Source: Bufferbloat and the Ski Shop - Random Neurons Firing - 1st May 2015

BufferBloat occurs when the connection speed you have (either for download or upload) is saturated with traffic. For example, if four people in a house on separate devices are all trying to watch Netflix in 4K at once, each stream uses 25Mbps. Bearing in mind that the average Openreach provider’s maximum download speed is 60-70Mbps, you will be trying to use 100Mbps of download speed. So what actually happens?

Well, the router’s buffers will fill up and then it will get confused as to who should get priority, as you are trying to use more bandwidth than is actually available. Hence the term “BufferBloat”. This means the streams will start to break up here and there.

This scenario becomes even more important with mixed usage. Back to our four person house - but this time, they are all trying to make a Zoom call at once (which means using their upload to send video and audio of themselves as well as their download to receive the other callers’ video and audio). Bear in mind that upload speeds on almost all UK providers are slower than download speeds (about 20Mbps max on average). Again, the router’s buffers will fill up, and it won’t know who to give priority to, leading to a sub-par performance for all of the house.

The usage scenarios above were less common historically, as video calling and 4K weren’t really a consideration. For years, most people’s routers were able to cope with the demands placed upon them - but now, to return to the analogy, suddenly everybody wants to go skiing.

Two great websites to test your connection for BufferBloat are DSL Reports  and Waveform . You’ll want to carry out this test using a wired only connection directly into your router (to eliminate any issues caused by your WiFi network) at a quiet time when nobody else is using your broadband connection. (NB - for DSLReports, select the appropriate connection type at the top - “DSL” for Openreach ADSL or FTTC, “Cable” for Virgin Media and “Gigabit / Fiber” for FTTH - and the test will start). Once it is finished, it will tell you your download speed, your upload speed, and give a BufferBloat rating as a grade (A+ being the highest and best), as well as some other information.

The most important grade to pay attention to is the BufferBloat rating. If it is anything below B, then you will experience BufferBloat when your connection is saturated (either download or upload, or both). This is where you may benefit from Smart Queues Management (SQM). (If the BufferBloat rating is A or higher, it may well be that your provider is already applying something like SQM at the exchange - for example, some Virgin Media areas use the latest cable internet standards than can actually apply SQM on their network - but this is quite rare).

NB - The DSL Reports and Waveform tests are two of the only ones that provide a BufferBloat rating, and are also generally the most accurate. Most of the others ( for example) are abused by ISPs to give false speeds .



What is Smart Queues Management (SQM)?



Some very entrepreneuring folks looked at the issue of BufferBloat and worked out that they could create an algorithm that can run on routers and other networking equipment to try to mitigate its effects. Here are some great websites with lots of detail about it:

Bufferbloat: Why it is Harming Your Broadband and How to Easily Fix It | Increase Broadband Speed

Introduction -

What Can I Do About Bufferbloat? -

It’s well worth reading all of the above to get a clearer understanding of SQM, but basically, it very cleverly monitors your connection and ensures that no one single user gets too much of the connection, and it also helps to prioritise latency-sensitive applications like video calls and online gaming, that get much more affected by BufferBloat, leading to stuttery calls and jumpy gaming.

NB - Some routers have a feature called “Quality of Service” (QoS) - this is a much less advanced version of SQM. Rather than an algorithm, QoS needs complex configuration for each application, and generally needs monitoring a lot more. It also only works for uploads, and is generally far inferior when compared to SQM - there is more info in “Bufferbloat and the Ski Shop” .



Okay, so how do I get SQM for my connection?


SQM requires specialist equipment

Because SQM is a relatively new concept, it isn’t available in very many consumer products at all. There is a good summary of what devices are able to utilise it here:

Methods to Slay Lag -

Really, you are going to need to either purchase a router from a specialist company like StopLagging or IQRouter, purchase a “prosumer” or enterprise level router from a company like Ubiquiti or Mikrotik, or buy (or flash) a consumer level router that runs the OpenWRT custom open source firmware.

Choosing, installing and configuring and SQM enabled device is nowhere near as simple as a standard commercial router, so I’d suggest consulting Hither Green IT  if you are unsure.

But once it is configured, SQM can be immensely helpful when a speed limited connection (that would be pretty much everyone’s in the UK!) is being used by a number of people heavily it once - it helps to ensure that Dad can stream Netflix in the living room while Mum makes a Zoom call in the back room, their son can play Call of Duty in his bedroom and their daughter can stream Amazon Prime Video in hers - and everyone gets a stutter free experience (provided everything is setup properly and the broadband provider’s connection is suffering no issues).





As you can see, despite the UK having a generally very poor outlook for broadband, there is a way to eek out a little more from your current connection. Countries with much wider full fibre optic coverage generally have download and upload speeds of 1,000Mbps so BufferBloat isn’t so much of a problem - it takes a lot of simultaneous usage to saturate that much bandwidth.

Hopefully, as the UK’s networks are slowly upgraded, faster speeds will become available, and the providers will start to use SQM at the exchange level, meaning end users don’t need to configure and apply it themselves.

As always, for expert advice and installation, contact Hither Green IT .