Recommendation for Linux Proxy

[garth1138]

Hello,

I'm getting ready to spin up a v10 Linux Proxy for my B&R on an ESXi cluster. This will be a virtual machine.
From a post on Reddit three months ago, Gostev mentions this:

...here are some curious backup proxy stress testing results with the current v10 build, with different job and compression settings.
WS2019: Most consistent performance overall regardless of proxy load.
Ubuntu 19, Debian 10: By far top performance specifically under heavy load, worse than WS2019 under lighter loads.
openSUSE 15.1: Very consistent and only slight worse performance than WS2019.
CentOS8: Most inconsistent performance (all over the place depending on proxy load).
CentOS7: Most terrible performance overall, not recommended

For CentOS he goes on to say:
"This is likely due to the default tune of CentOS. Experienced users will apply throughput oriented tuned profile to solve the issue. But since almost all users just go with the default OS settings, we need to establish some general recommendations for out of the box configurations."

From the v10 user guide, I am aware of the following requirements and limitations that apply to Linux backup proxies:
• bash shell and SSH are required
• The user account that you specify for the Linux server must be a root user or user elevated to root.
• The disk.EnableUUID parameter of the Linux server must be set to TRUE in the VMware vSphere client.
• Only the Virtual appliance transport mode is available.

So, I am further wondering:
1. What is the recommended distro for best performance?
2. What is the recommended configuration for memory/disk?
3. What is the recommended performance tuning?

anything else?

[HannesK]

Hello,
and welcome to the forums.

1. Windows You probably can tune all distros to Ubuntu / Debian level (which are obviously the recommendation according to our QA tests because you don't need to do anything).
2. Same like for Windows (1 CPU & 2 GB RAM per task)
3. I don't have that information as I use Debian since 2002

Best regards,
Hannes

[garth1138]

Update:
Decided to go with Ubuntu 19.10
Created VM with "ubuntu-19.10-live-server-amd64.iso"
• 8 Gigs RAM
• 40 Gig HD
• 4 CPU
• VMWare ParaVirtual SCSI controller and VMXNET3 NIC

Installed with no extra packages, just straight up server-only install.
Perl is installed by default so no worries.
Set the "disk.EnableUUID" advanced config parameter of the Linux server to TRUE in the VMware vSphere client before reboot

Added all the necessary port to the firewall and activated the firewall.

Code: Select all

$ sudo ufw allow OpenSSH
$ sudo ufw allow 2500:3300/tcp
$ sudo ufw allow 443/tcp
$ sudo ufw allow 902/tcp
$ sudo ufw allow 49152:65535/tcp
$ sudo ufw enable
$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
2500:3300/tcp              ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
902/tcp                    ALLOW       Anywhere
49152:65535/tcp            ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
2500:3300/tcp (v6)         ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)
902/tcp (v6)               ALLOW       Anywhere (v6)
49152:65535/tcp (v6)       ALLOW       Anywhere (v6)

Also installed tuned and set default profile to "throughput-performance"

Code: Select all

$ sudo apt install tuned
$ sudo tuned-adm profile throughput-performance
$ sudo tuned-adm active
Current active profile: throughput-performance

I added the new linux proxy server to my Veeam 10 infrastructure on the Veeam server and set it as an active proxy.

I use three total proxies in my infrastructure, including the Veeam Server, so I de-activated one of the Windows proxies. The first incremental backup went flawless.
The second day I created another Linux Proxy same as above and de-activated the other Windows proxy. Same flawless performance and incremental backups have been just as fast as the Windows ones.

I will see this weekend, when I do an active-full backup of all my servers, if the overall performance will match the previous months. (8TB read, 5.5TB transfer)

I'll try to report my result.

[garth1138]

Okay,
The results are in. It turned into a minor disaster.
First Backup Job finished with three VMs backed up. Then the next job with 16 VMs had 4 failures. Not telling if this was related to the Linux Proxies. Snapshots were failing to be removed. The vCenter server was failing with this error:

"Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-36, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce False
Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.x.x.x:443

Again, no idea if this was due to Linux proxies. 2nd and 3rd retries failed (I was not watching at this time... )
So, I stopped all jobs and rebooted the Veeam Server, both Linux proxies, found multiple Veeam snapshots on the vCenter server so I deleted those (took a while) and rebooted the vCenter server appliance (a Vm in my ESXi environment). WOW, this has never happened to me before. My normal routine is to simply reboot the Veeam servers and the gateway server (which I had done before the first run), never the vCenter appliance.

After all was rebooted I started the Active full backups again. All Jobs worked this time through. I watched the progress for about 4 hours. What I noticed is that the Linux proxies were processing my hard disks at a much slower rate than the Veeam-server-proxy (Windows). This is revealed in the overall time my full backup took.

Essentially, every backup job was slower except for a 16VM job of mostly Linux VMs (48 minutes today compared to 58 minutes last time) with roughly the same amount of data.

I have 5 backup jobs that run in sequence. I am transferring approximately 5.5TB (after dedupe and compression). My time for the total Active Full backup last month with all Windows proxies (Server 2012 R2) on v9.5 was 14:54 (HH:mm). This month on v10 with my Linux proxies, my overall time was 19:43 (HH:mm). That is an increase of 4:49 or a 32% increase in time.

Added to this I am seeing these messages on the consoles of my Linux proxies: Buffer I/O error on dev dm-[x], logical block [x], async page read. Pic here:
(https://imgur.com/ebqb0pr)

I am now also worried about the data. I never had any issues such as this with my Windows proxies.

So, I am going back to my Windows proxies. I'll be waiting to see if the Linux proxy gets better. It was worth a shot.

[nitramd]

@garth1138,

I've found that adjusting the RX & TX buffers upward helps performance issues. Also, enabling Receive Side Scaling helps.