OpenVPN performance

Talkabout · January 2, 2020, 6:30pm

Hi guys,

I recently bought a RockPi 4 to replace my Raspberry Pi as VPN gateway. The reason is that the performance numbers of OpenSSL/OpenVPN impressed me. Today I received my device and set it up, first with Ubuntu Server and then with Armbian Buster. In both cases I can reproduce good performance indicators when using the usual testing methods:

root@rockpi:~# openssl speed -evp aes-256-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 60081162 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 31535221 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 10636732 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 2871501 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 373949 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 16384 size blocks: 186738 aes-256-cbc's in 3.00s
OpenSSL 1.1.1  11 Sep 2018
built on: Tue Nov 12 16:58:35 2019 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-J6qvxk/openssl-1.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc     320432.86k   672751.38k   907667.80k   980139.01k  1021130.07k  1019838.46k

and

root@rockpi:~# openvpn --genkey --secret /tmp/secret
0 --tun-mtu 20000 --cipher aes-256-cbc--test-crypto --secret /tmp/secret --verb 0
Thu Jan  2 18:25:45 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

real    0m1.973s
user    0m1.960s
sys     0m0.008s

both these tests showed much lower number on a RPI (1/10th openssl, 12s openvpn) which is giving me approx. 7MB/s download speed via VPN. Unfortunately when checking the VPN throughput of RockPI4 I am getting approx. 2MB/s and I do not understand why. Based on the tests the RockPI must be 10 times faster than the RPI, but in fact it is (much!) slower. Is there a specific configuration necessary to make the RockPI perform better in OpenVPN environment? I have tried several things but nothing worked. I cannot believe that this is the max the RockPI can deliver! Help is much appreciated!

Thanks!

Bye

Dante4 · January 3, 2020, 12:02am

First, I’m not an expert in this, so think about it as general recommendation
Second, if it’s intended to run on CPU - don’t forget to use taskset to use hi-freq cores
Third, be sure to install all rockchips’ packages which Radxa provides in their repos (to be precise look for GPU’s driver section there https://wiki.radxa.com/Rockpi4/Sgminer).
Fourth, when it comes to software optimization - remeber, that RasPi have a whole other level of being accepted in “big project”'s society, just think, there is special OpenVPN package called PiVPN. RockPi on otherside only starts to be accepted. On raw scales RockPi have highground, but in terms of software development (which highly depends on users) - nothing beats RasPi (as for now at least).

Talkabout · January 3, 2020, 9:58am

Hi @Dante4,

thanks for your suggestions. I have tried to set CPU affinity to every CPU core available, but it does not change anything. The installation guide you showed is not working with Ubuntu Server, there those GPU drivers are not available.

In general the thing is that when using openssl/openvpn with the commands shown in my first thread the performance is nice, so the question is why this changes suddenly when it comes to throughput via VPN connection. I assume there are some settings to be configured to optimize the traffic, but at the moment I have no idea where those might be…

By the way, the reason why the crypto speed is so high in the test commands is that RockPI 4 is capable of crypto hardware support, so there shouldn’t be a need to change a particular CPU core…

Bye

Dante4 · January 3, 2020, 10:48am

Erm, please be sure, that you added radxa repository in this case, because they are available. And for hardware support of anything you first need to have correct driver for if. Please do this

    echo “deb http://apt.radxa.com/bionic-testing/ bionic main” | sudo tee /etc/apt/sources.list.d/apt-radxa-com.list
    wget -O - apt.radxa.com/bionic-testing/public.key | sudo apt-key add -
    sudo apt-get update && sudo apt-get -y upgrade

And then repeat installation step

Talkabout · January 3, 2020, 11:48am

I had the stable radxa repo in apt source lists. Now I changed it to testing and installed the GPU drivers. After reboot tried VPN again, same speeds, no change…

igorp · January 3, 2020, 12:17pm

Scripts has nothing to do with speed, optimisations or hw crypto optimisation which is what @Talkabout is interested in. Those are just install scripts. Ofc they are useful and handy, but they optimise nothing but install process.

A stereotype turned into religion to earn millions and have full control over millions of computers running Linux in a closed sourced virtual machine owned by Microsoft. Linux is here since 90", Unix before that …

Once you have a good communication with the hardware (kernel), you are done. Everything that works on RPi , works here or your desktop PC. Ofc there are grey areas, but in principle that’s it.

Works here too. If by any chance not on Radxa Debian, then on Armbian.

So you have to tell OVPN or Wireguard to use that somehow … Its common Rockchip problem. I would search widely.

Talkabout · January 3, 2020, 12:43pm

So you have to tell OVPN or Wireguard to use that somehow … Its common Rockchip problem. I would search widely.

and this is exactly what I do not understand. When running the tests (first post) in both cases (openssl and openvpn) the hardware crypto is used, because otherwise the numbers would have been much smaller. You can check out this page for reference:

github.com/ThomasKaiser/sbc-bench

Results.md

master

# Results

Below some results collected. Please keep in mind that these are **NOT** hardware performance numbers but depend on software/settings (see the differences kernel version makes for RockPro64 for example). The purpose of `sbc-bench` is to generate insights and not colorful graphs representing numbers without meaning. It's perfectly fine for the same hardware appearing multiple times with different numbers since those differ for a reason (software/settings).

Especially *openssl* numbers should be taken with a huge grain of salt since the benchmark numbers depend on kernel features and performance with other use cases (e.g. disk/filesystem encryption) [might look  differently](https://forum.armbian.com/topic/7763-benchmarking-cpus/?do=findComment&comment=59235).

So do **not** rely on collected numbers unless you carefully read through all the explanations and insights below and be prepared to conduct your own benchmarks if you really want to choose appropriate hardware for **your** use case.

## Some numbers

*The below table is also available sorted: [7-zip multi-threaded](Sorted-Results.md#7-zip-mips-multi-threaded), [7-zip single-threaded](Sorted-Results.md#7-zip-mips-single-threaded), [aes-256-cbc](Sorted-Results.md#openssl-speed--elapsed--evp-aes-256-cbc), [memcpy](Sorted-Results.md#memcpy), [memset](Sorted-Results.md#memset) and [clockspeed](Sorted-Results.md#clockspeed).*

*ROCK 5B, ROCK 5C Lite, Orange Pi 5, RK3568-ROC-PC and Khadas VIM4 numbers are preliminary since software support situation for RK358x and A311D2 is still in a very early stage. Same applies to all RISC-V numbers and [Apple M1 Pro](https://github.com/ThomasKaiser/sbc-bench/issues/47#issue-1300572558). Please also note that with RK35xx devices so far [measured clockspeeds differ from what's defined in device-tree due to PVTM](https://github.com/ThomasKaiser/Knowledge/blob/master/articles/Quick_Preview_of_ROCK_5B.md#pvtm) and also that all these RK35xx based devices perform more or less identical [or only differ by PVTM (silicon lottery) and settings](https://github.com/ThomasKaiser/sbc-bench/issues/75#issuecomment-1672817171).*

| Device / details | Clockspeed | Kernel | Distro | 7-zip multi | 7-zip single | AES | memcpy | memset | kH/s |
| ----- | :--------: | :----: | :----: | ----: | ------: | ------: | -----: | -----: | ---: |
| [AAEON BOXER-8654AI (Orin NX)](results/87lq.txt) | 1730 MHz | 5.15 | Jammy arm64 | 14820 | 2379 | 973830 | 6920 | 20470 | - |
| [Akaso M8S (S812)](results/3R3N.txt) | 1200 MHz | 5.10 | Buster armhf | 3050 | 885 | 32120 | 1160 | 3330 | - |
| [Amazon a1.xlarge](results/2iFY.txt) | 2300 MHz | 4.15 | Bionic arm64 | 8610 | 2406 | 1297960 | 4280 | 14220 | - |
| [AMD Seattle (Opteron A1100)](results/4Kqn.txt) | 2000 MHz | 4.15 | Bionic arm64 | 14080 | 2006 | 720710 | 3020 | 9530 | - |

This file has been truncated. show original

Here the RockPI 4 has one of the highest results, which was the reason I decided to try it. I also have a Rock64 lying around where I also tried to use it as VPN Gateway. The throughput was good, also the VPN speed was what I was expecting, but unfortunately I was not able to make it run stable with Armbian. I faced crashes all the time which is not good for a gateway to the internet But the speed, also with Rockchip hardware, was good enough. So now the question is what is the difference between those two (Rock64 and RockPi 4)?

Thanks!

Bye

Talkabout · January 3, 2020, 2:00pm

One more thing that is more than strange: if the RPI is not using hardware crypto and is able to deliver a bandwidth of approx. 7MB/s via VPN, how can it be that the RockPI 4 with a much faster processor is giving approx 2MB/s? It seems to be the case that the bandwidth is limited somewhere else and not by OpenVPN…

Dante4 · January 3, 2020, 3:50pm

Well, then my work is done, since we got someone who understands how it works.

But i would disagree with just one thing. Existing of PiVPN also means that there could be patches in OpenVPN that adds optimization for Raspberry Pi.

igorp · January 3, 2020, 4:43pm

This is science. I don’t mean to be harsh, but I really don’t like all this BS that is spread around toy for masses.

Find patches and find its explanation. Until then, they don’t exists.

Dante4 · January 3, 2020, 10:37pm

No, you have the point there, while i’m just speculate. I will test PiVPN on my RockPi in the following days.

While I’m at it @Talkabout may you try Debian and say are results same or not?

Talkabout · January 4, 2020, 11:05am

Hi @Dante4,

I have tried all the available distributions (armbian ubuntu/debian, ubuntu bionic server), in all cases results are the same. I am not able to get the speed above 3MB/s.

Thanks for your effort!

Bye

igorp · January 4, 2020, 11:41am

Also tried Armbian with kernel 5.4.y ? And searched forum for more info … someone should know something about this.

Talkabout · January 4, 2020, 11:56am

Yes, I tried Armbian with kernel 5.4 and 4.4, there is no change in behavior. Searching the net was not successful, I have not found anybody with exact this problem on RockPI 4. There are some posts about general performance tweaks like using a specific and isolated CPU core. I have tried all of them, no success. I have also tried several tweaks in the OpenVPN client configuration file (sndbuf/rcvbuf…) but still no change. That is why I think the limiting factor is not OpenVPN, especially because the crypto tests were showing very promising numbers. Even without hardware crypto RockPI should give higher numbers than a Raspberry PI, but it is not. That is why I was hoping to find a solution in this forum…

Bye

igorp · January 4, 2020, 1:33pm

There are many RK3399 devices out there. They are (in theory) identical in this.

Module represented for hw crypto acceleration is build as module. Perhaps you need to load module manually?

github.com/armbian/build

config/kernel/linux-rockchip64-legacy.config

master


      
          #
          # Block modes
          #
          CONFIG_CRYPTO_CBC=y
          CONFIG_CRYPTO_HEH=y
          CONFIG_CRYPTO_CTR=y
          CONFIG_CRYPTO_CTS=y
          CONFIG_CRYPTO_ECB=y
          CONFIG_CRYPTO_LRW=m
          CONFIG_CRYPTO_PCBC=m
          CONFIG_CRYPTO_XTS=y
          CONFIG_CRYPTO_KEYWRAP=m
          
          #
          # Hash modes
          #
          CONFIG_CRYPTO_CMAC=y
          CONFIG_CRYPTO_HMAC=y
          CONFIG_CRYPTO_XCBC=m
          CONFIG_CRYPTO_VMAC=m

… but still OVPN has to know somehow to use that.

On some RK3399 devices this is different:

github.com/armbian/build

config/kernel/linux-rk3399-legacy.config

master


      
          CONFIG_CRYPTO_TWOFISH=m
          CONFIG_CRYPTO_TWOFISH_COMMON=m
          
          #
          # Compression
          #
          CONFIG_CRYPTO_DEFLATE=y
          CONFIG_CRYPTO_ZLIB=m
          CONFIG_CRYPTO_LZO=m
          CONFIG_CRYPTO_842=m
          CONFIG_CRYPTO_LZ4=m
          CONFIG_CRYPTO_LZ4HC=m
          CONFIG_CRYPTO_ZSTD=m
          
          #
          # Random Number Generation
          #
          CONFIG_CRYPTO_ANSI_CPRNG=y
          CONFIG_CRYPTO_DRBG_MENU=y
          CONFIG_CRYPTO_DRBG_HMAC=y
          CONFIG_CRYPTO_DRBG_HASH=y

Current 5.4.y has it as module build/config/kernel/linux-rockchip64-current.config at master · armbian/build · GitHub

Talkabout · January 4, 2020, 2:34pm

Hi @igorp,

thanks for the suggestion but there is no difference in speed with or without the module. I have loaded it manually (modprobe rk_crypto) and added it to /etc/modules (then reboot) and still nothing changed, speed < 2MB/s…

Bye

Talkabout · January 5, 2020, 12:12pm

Hi all,

any other hint what I can do to make the RockPI 4 perform better in OpenVPN case?

Thanks!

Bye

igorp · January 5, 2020, 1:15pm

That you should widen your research out of this forum.

Talkabout · January 5, 2020, 1:57pm

Hi @igorp,

I have done a lot of research already, also on the pages mentioned by you. None of them provided a solution to the problem. Assuming that this is the forum of the RockPI’s vendor, I was hoping to get some more help here, maybe also somebody trying out if the issue is reproducible or simply an issue with only my device…

Yesterday I was playing around with the CPU frequencies and realized that those are throttled. Maybe this is one of the problems… I also realized that RockPI 4 is not supported by Armbian, they have a section in their forums claiming support for this device being still “in development”.

It seems to me the device itself is not that mature yet…

Bye

igorp · January 5, 2020, 2:12pm

Probably not, but you can get hints how to approach to the problem. Not everything can be found on the internet just like that. Especially if you are not searching for proper targets.

They are producing a board with RK3399 (or whatever) chip. Hardware crypto functions were developed by Rockchip. If there is a manual, you should look there http://opensource.rock-chips.com/wiki_Linux_SDK

Support for RK3399 is in very good shape, but forums are still in the “Development” areas for no apparent reasons. Its simply due lack of time to moderate forum. Most of users just demand things and give nothing in return (perhaps help on moderating forum).

It is supported by Armbian Rockpi 4 - Armbian (Green sign “SUPPORTED”) while support will always stay in the grey area Introduction - Armbian Documentation

The problem you have is unrelated to Radxa, Rockchip and Armbian but it is called “VPN with hw supported chippers” which I have no special knowledge and can’t give you better clues.