Discussions specific to the VMware vSphere hypervisor
Post Reply
garth1138
Novice
Posts: 4
Liked: 2 times
Joined: Mar 18, 2020 1:06 pm
Contact:

Recommendation for Linux Proxy

Post by garth1138 » 1 person likes this post

Hello,

I'm getting ready to spin up a v10 Linux Proxy for my B&R on an ESXi cluster. This will be a virtual machine.
From a post on Reddit three months ago, Gostev mentions this:
...here are some curious backup proxy stress testing results with the current v10 build, with different job and compression settings.
WS2019: Most consistent performance overall regardless of proxy load.
Ubuntu 19, Debian 10: By far top performance specifically under heavy load, worse than WS2019 under lighter loads.
openSUSE 15.1: Very consistent and only slight worse performance than WS2019.
CentOS8: Most inconsistent performance (all over the place depending on proxy load).
CentOS7: Most terrible performance overall, not recommended
For CentOS he goes on to say:
"This is likely due to the default tune of CentOS. Experienced users will apply throughput oriented tuned profile to solve the issue. But since almost all users just go with the default OS settings, we need to establish some general recommendations for out of the box configurations."

From the v10 user guide, I am aware of the following requirements and limitations that apply to Linux backup proxies:
• bash shell and SSH are required
• The user account that you specify for the Linux server must be a root user or user elevated to root.
• The disk.EnableUUID parameter of the Linux server must be set to TRUE in the VMware vSphere client.
• Only the Virtual appliance transport mode is available.

So, I am further wondering:
1. What is the recommended distro for best performance?
2. What is the recommended configuration for memory/disk?
3. What is the recommended performance tuning?

anything else?

HannesK
Veeam Software
Posts: 7111
Liked: 1136 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Recommendation for Linux Proxy

Post by HannesK »

Hello,
and welcome to the forums.

1. Windows ;-) You probably can tune all distros to Ubuntu / Debian level (which are obviously the recommendation according to our QA tests because you don't need to do anything).
2. Same like for Windows (1 CPU & 2 GB RAM per task)
3. I don't have that information as I use Debian since 2002

Best regards,
Hannes

garth1138
Novice
Posts: 4
Liked: 2 times
Joined: Mar 18, 2020 1:06 pm
Contact:

Re: Recommendation for Linux Proxy

Post by garth1138 »

Update:
Decided to go with Ubuntu 19.10
Created VM with "ubuntu-19.10-live-server-amd64.iso"
• 8 Gigs RAM
• 40 Gig HD
• 4 CPU
• VMWare ParaVirtual SCSI controller and VMXNET3 NIC

Installed with no extra packages, just straight up server-only install.
Perl is installed by default so no worries.
Set the "disk.EnableUUID" advanced config parameter of the Linux server to TRUE in the VMware vSphere client before reboot

Added all the necessary port to the firewall and activated the firewall.

Code: Select all

$ sudo ufw allow OpenSSH
$ sudo ufw allow 2500:3300/tcp
$ sudo ufw allow 443/tcp
$ sudo ufw allow 902/tcp
$ sudo ufw allow 49152:65535/tcp
$ sudo ufw enable
$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
2500:3300/tcp              ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
902/tcp                    ALLOW       Anywhere
49152:65535/tcp            ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
2500:3300/tcp (v6)         ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)
902/tcp (v6)               ALLOW       Anywhere (v6)
49152:65535/tcp (v6)       ALLOW       Anywhere (v6)
Also installed tuned and set default profile to "throughput-performance"

Code: Select all

$ sudo apt install tuned
$ sudo tuned-adm profile throughput-performance
$ sudo tuned-adm active
Current active profile: throughput-performance
I added the new linux proxy server to my Veeam 10 infrastructure on the Veeam server and set it as an active proxy.

I use three total proxies in my infrastructure, including the Veeam Server, so I de-activated one of the Windows proxies. The first incremental backup went flawless.
The second day I created another Linux Proxy same as above and de-activated the other Windows proxy. Same flawless performance and incremental backups have been just as fast as the Windows ones. :mrgreen:

I will see this weekend, when I do an active-full backup of all my servers, if the overall performance will match the previous months. (8TB read, 5.5TB transfer)

I'll try to report my result.

garth1138
Novice
Posts: 4
Liked: 2 times
Joined: Mar 18, 2020 1:06 pm
Contact:

Re: Recommendation for Linux Proxy

Post by garth1138 »

Okay,
The results are in. It turned into a minor disaster.
First Backup Job finished with three VMs backed up. Then the next job with 16 VMs had 4 failures. Not telling if this was related to the Linux Proxies. Snapshots were failing to be removed. The vCenter server was failing with this error:
"Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-36, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce False
Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.x.x.x:443
Again, no idea if this was due to Linux proxies. 2nd and 3rd retries failed (I was not watching at this time... )
So, I stopped all jobs and rebooted the Veeam Server, both Linux proxies, found multiple Veeam snapshots on the vCenter server so I deleted those (took a while) and rebooted the vCenter server appliance (a Vm in my ESXi environment). WOW, this has never happened to me before. My normal routine is to simply reboot the Veeam servers and the gateway server (which I had done before the first run), never the vCenter appliance.

After all was rebooted I started the Active full backups again. All Jobs worked this time through. I watched the progress for about 4 hours. What I noticed is that the Linux proxies were processing my hard disks at a much slower rate than the Veeam-server-proxy (Windows). This is revealed in the overall time my full backup took.

Essentially, every backup job was slower except for a 16VM job of mostly Linux VMs (48 minutes today compared to 58 minutes last time) with roughly the same amount of data.

I have 5 backup jobs that run in sequence. I am transferring approximately 5.5TB (after dedupe and compression). My time for the total Active Full backup last month with all Windows proxies (Server 2012 R2) on v9.5 was 14:54 (HH:mm). This month on v10 with my Linux proxies, my overall time was 19:43 (HH:mm). That is an increase of 4:49 or a 32% increase in time.

Added to this I am seeing these messages on the consoles of my Linux proxies: Buffer I/O error on dev dm-[x], logical block [x], async page read. Pic here:
Image (https://imgur.com/ebqb0pr)

I am now also worried about the data. I never had any issues such as this with my Windows proxies.

So, I am going back to my Windows proxies. I'll be waiting to see if the Linux proxy gets better. It was worth a shot.

nitramd
Veeam Legend
Posts: 289
Liked: 81 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Recommendation for Linux Proxy

Post by nitramd »

@garth1138,

I've found that adjusting the RX & TX buffers upward helps performance issues. Also, enabling Receive Side Scaling helps.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 7 guests