Direct SAN Backup - Slow

[slewin]

Hi All

I currently have a physical Veeam Server(Dell 2x Xeon's with 32G ram) attached with a 6G SAS direct attached Dell Storage device an 8G Fibre connection to our EMC SAN. The backups are indicating that a SAN backup is being used but my speeds don't seem right. All backups are scheduled for after hours when our VMWare and SAN enviroment is "idle" . My current backups are running at about 153MB/s. I realise different variables can determine different site speeds but what is the "average speed" I should be getting?

[Vitaliy S.]

Hi Steven,

What is the bottleneck stats for your backup jobs? You can find this info in the job session details.

Thanks!

[slewin]

Hi,

Strangely enough. Different Jobs are showing different bottlenecks, Target , Proxy and Source.

What is best practice with regards to proxy servers? My physical backup server is also a proxy server but I decided to add 1 Proxy per Host(3) and performance has definitely increased.
On one of my jobs last night with the target as the bottleneck had a throughput of 1.2GB/s, nice and fast but not all jobs are running this fast.

[Vitaliy S.]

If you have physical server that has access to your SAN storage, then it's best practice to leverage Direct SAN access mode to backup your VMs. What configuration do you have for your jobs? Do they use the same proxy and repository servers?

[slewin]

All my Backup Jobs are configured the same: Automatic Selection for Proxy and they all use the same Backup Repository.

If my Physical Backup Server has direct access to my SAN should I remove the 3 other Virtual Proxies which are each on a Host?

[Vitaliy S.]

Did you mount all LUNs with your VMs to the physical proxy server? Can you check if these proxy server are actually being used? This can be done either using Veeam ONE, or by investigating individual backup job sessions. If these proxy servers were never used, then you can safely remove them.

Also I'm a bit confused about your bottlenecks stats - you say it's different for all jobs, but these jobs should be using the same proxy and the same repository based on what I understand see from your configuration.

[slewin]

Hi, Yes. All my LUNS are visible to my physical proxy server via 8G Fibre directly into our VNX5300. I can see them all in Disk Manager. I removed the virtual proxies which were on each host(3) as the backups were definitely using my physical proxy (san). I cant understand why the speeds are so inconsistent. Surely with 8G direct to san I should be getting very good throughput speeds. All drivers/firmware have been update on the server and hba card. I'm stumped with this.

[Vitaliy S.]

Do you see inconsistency for full and incremental backup jobs runs? Can you post full bottleneck stats for your jobs?

[slewin]

All my backups are reverse incrementals. An example Job. 3x SQL Servers - Data Processed: 4,9Tb Read:341,4GB Transfered:101,5 GB(3,4x)
Processed Rate: 68MB/s Throughput: 169,5 MB/s

Source: 14%
Proxy:10%
Network:39
Target: 81%

My Veeam Backup Server is a Dell R720, 96G Ram, 2x 6 Core CPu's and 2x SAS MD3200 direct Storage

The performance\figures seems to be an average for all jobs at the moment although in my earlier comments i was getting 1,2 - 2 GB/s

[Vitaliy S.]

Oh...reverse incremental and target acting as a bottleneck, it is somewhat expected for not high-end target storage. If you switch to forward incremental backup mode, then it should bump up the performance, since reversed incremental backup mode has more penalty on target storage in terms of required IOPs.

The performance\figures seems to be an average for all jobs at the moment although in my earlier comments i was getting 1,2 - 2 GB/s

That is actually very decent performance, but I assume it was calculated like this - processed size/total job time. What Veeam B&R version are you at?

[slewin]

V 7.0.0.871

[Vitaliy S.]

2 GB/s performance is way to quick, probably you had these stats on earlier versions of Veeam B&R where calculations were performed in a bit different way? Since now you see target storage as a bottleneck through out all jobs, try switching one of the jobs to forward incremental backup mode, and see what rates you have after that. Thanks!

[foggy]

Be sure not to compare full job runs with incremental ones, since they should have different processing rates. Also I would check whether CBT is working on all of the jobs and the number of concurrently running tasks, since it can affect performance as well.

[slewin]

Correct, I've just created a new Job and the throughput Rate is 700+ MB/s. Why are full faster than incrementals?

[Gostev]

Because you are using reversed incremental backup (that is heavy on random I/O) against a backup repository that is lacking IOPS capacity. Full backup on the other hand is mostly sequential I/O, which does not require much IOPS to sustain on the backup storage side.