Upgrading Kernel with BBR in Centos 7

This is a simple guide to harness the best performance out of your KVMs

It doesnt always improve things; hope we can discuss below based on some observations.

Depending on many factors, the VPS may or may not perform at its best on a fresh install.

Let’s take the most stable, high performance Gold KVM with NVMe on 10Gbit network from Hostdoc and see how it performs fresh out of the oven:

Centos 7.6 with stock Kernel 3.10
yum -y update
wget -qO- bench.sh | bash
-------------------------------------------------------------------       
CPU model            : Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz              
Number of cores      : 3                                                     
CPU frequency        : 3392.026 MHz
Total size of Disk   : 29.0 GB (2.1 GB Used)
Total amount of Mem  : 1837 MB (61 MB Used)                                  
Total amount of Swap : 1022 MB (0 MB Used)
System uptime        : 0 days, 0 hour 11 min
Load average         : 0.26, 0.17, 0.14                                      
OS                   : CentOS 7.6.1810                                       
Arch                 : x86_64 (64 Bit)                                       
Kernel               : 3.10.0-957.1.3.el7.x86_64                             
----------------------------------------------------------------------       
I/O speed(1st run)   : 877 MB/s                                              
I/O speed(2nd run)   : 775 MB/s                                              
I/O speed(3rd run)   : 963 MB/s                                              
Average I/O speed    : 871.7 MB/s                                            
----------------------------------------------------------------------       
Node Name                       IPv4 address            Download Speed       
CacheFly                        205.234.175.175         158MB/s              
Linode, Tokyo, JP               106.187.96.148          15.2MB/s             
Linode, Singapore, SG           139.162.23.4            10.2MB/s             
Linode, London, UK              176.58.107.39           19.5MB/s             
Linode, Frankfurt, DE           139.162.130.8           17.4MB/s             
Linode, Fremont, CA             50.116.14.9             13.3MB/s             
Softlayer, Dallas, TX           173.192.68.18           111MB/s              
Softlayer, Seattle, WA          67.228.112.250          44.9MB/s             
Softlayer, Frankfurt, DE        159.122.69.4            10.1MB/s             
Softlayer, Singapore, SG        119.81.28.170           8.40MB/s             
Softlayer, HongKong, CN         119.81.130.170          9.77MB/s
----------------------------------------------------------------------
Node Name                       IPv6 address            Download Speed
Linode, Atlanta, GA             2600:3c02::4b           54.0MB/s
Linode, Dallas, TX              2600:3c00::4b           205MB/s
Linode, Newark, NJ              2600:3c03::4b           52.0MB/s
Linode, Singapore, SG           2400:8901::4b           11.7MB/s
Linode, Tokyo, JP               2400:8900::4b           112KB/s
Softlayer, San Jose, CA         2607:f0d0:2601:2a::4    24.0MB/s
Softlayer, Washington, WA       2607:f0d0:3001:78::2    23.4MB/s
Softlayer, Paris, FR            2a03:8180:1301:8::4     16.1MB/s
Softlayer, Singapore, SG        2401:c900:1101:8::2     2.66MB/s
Softlayer, Tokyo, JP            2401:c900:1001:16::4    11.4MB/s
----------------------------------------------------------------------
[root@host ~]#

Great performance right? Should be good enough for many, but if you really want to squeeze the best performance out of every precious dollar spent then…

First we verify current kernel version:

uname -r

You should see something like this:

3.10.0-957.1.3.el7.x86_64

Next install the ELRepo Repo:

sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
sudo rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

Install the latest kernel:

sudo yum --enablerepo=elrepo-kernel install kernel-ml -y

Check list of installed kernels, entry starts from 0,1,2,3 etc:

sudo egrep ^menuentry /etc/grub2.cfg | cut -f 2 -d \'

It should look similar to this:

CentOS Linux (5.0.0-2.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-957.5.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-957.1.3.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-96820b9851c24560b5f942f2496b9aeb) 7 (Core)

First on the list is number 0, so we set it to boot as default:

sudo grub2-set-default 0
reboot

Once back online we check that the kernel is latest:

uname -r
5.0.0-2.el7.elrepo.x86_64

OK, upgraded to latest available kernel. Now let’s do the BBR for more consistent network performance:

echo 'net.core.default_qdisc=fq' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_congestion_control=bbr' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Verify that BBR is enabled:

sudo sysctl net.ipv4.tcp_available_congestion_control

You shoud see:

net.ipv4.tcp_available_congestion_control = reno cubic bbr

Verify next:

sudo sysctl -n net.ipv4.tcp_congestion_control

Output should be:

bbr

Check that BBR module is loaded:

lsmod | grep bbr

Output should be something like:

tcp_bbr           20480    33

OK great, now let’s run the obligatory bench:

----------------------------------------------------------------------       
CPU model            : Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
Number of cores      : 3
CPU frequency        : 3392.026 MHz
Total size of Disk   : 29.0 GB (2.8 GB Used)
Total amount of Mem  : 1992 MB (62 MB Used)
Total amount of Swap : 1022 MB (0 MB Used)
System uptime        : 4 days, 16 hour 29 min
Load average         : 0.42, 0.11, 0.04
OS                   : CentOS 7.6.1810
Arch                 : x86_64 (64 Bit)
Kernel               : 5.0.0-2.el7.elrepo.x86_64
----------------------------------------------------------------------
I/O speed(1st run)   : 1.1 GB/s
I/O speed(2nd run)   : 1.1 GB/s
I/O speed(3rd run)   : 1.3 GB/s
Average I/O speed    : 1194.7 MB/s
----------------------------------------------------------------------
Node Name                       IPv4 address            Download Speed
CacheFly                        205.234.175.175         197MB/s
Linode, Tokyo, JP               106.187.96.148          16.7MB/s
Linode, Singapore, SG           139.162.23.4            11.1MB/s
Linode, London, UK              176.58.107.39           19.9MB/s
Linode, Frankfurt, DE           139.162.130.8           17.3MB/s
Linode, Fremont, CA             50.116.14.9             60.4MB/s
Softlayer, Dallas, TX           173.192.68.18           110MB/s
Softlayer, Seattle, WA          67.228.112.250          44.3MB/s
Softlayer, Frankfurt, DE        159.122.69.4            8.39MB/s
Softlayer, Singapore, SG        119.81.28.170           8.24MB/s
Softlayer, HongKong, CN         119.81.130.170          9.70MB/s
----------------------------------------------------------------------
Node Name                       IPv6 address            Download Speed
Linode, Atlanta, GA             2600:3c02::4b           114MB/s
Linode, Dallas, TX              2600:3c00::4b           171MB/s
Linode, Newark, NJ              2600:3c03::4b           5.17MB/s
Linode, Singapore, SG           2400:8901::4b           11.2MB/s
Linode, Tokyo, JP               2400:8900::4b           16.6MB/s
Softlayer, San Jose, CA         2607:f0d0:2601:2a::4    19.8MB/s
Softlayer, Washington, WA       2607:f0d0:3001:78::2    54.3MB/s
Softlayer, Paris, FR            2a03:8180:1301:8::4     15.4MB/s
Softlayer, Singapore, SG        2401:c900:1101:8::2     8.68MB/s
Softlayer, Tokyo, JP            2401:c900:1001:16::4    9.46MB/s
----------------------------------------------------------------------
[root@host ~]#
more (available) RAM
close to 40% increase in average I/O performance
more consistent network throughput

It is also possible that CPU performance is improved:

More beautiful benches showing powerful 10gbit + BBR:

Hope this guide is as useful as it is entertaining for you

Observations:

Tried many KVMs with different processors, and while all benefit from BBR on network througput, so far only Xeon E5-2XXX / Silver / Gold maintain or improve I/O and CPU performance with Kernel upgrade.

Most, if not all E3s, along with i7-6700K/7700K have worse performance.

On most E3s the CPU performance is noticeably affected. On i7-6700K / 7700K CPU performance is about the same.

However the I/O for SSD / NVMe is fine, but HDD speeds are greatly affected (down by 60%)

Woah, that’s awesome! I will have a play around with the same thing in Debian/Ubuntu when I get a spare chance, that definitely makes an improvement from the looks of it

Thanks for contributing

David wrote:

Woah, that’s awesome! I will have a play around with the same thing in Debian/Ubuntu when I get a spare chance, that definitely makes an improvement from the looks of it

Thanks for contributing

welcome

got a feeling that on Ubuntu it should work well out of the box, iirc stock kernel is 4.15.
interested to know if it does make a difference though

Excellent write up cybertech.
And a rare how to barely found elsewhere if even. Kudos.