Slow backup (San to San)

[samuk]

Hi,

Using Veeam Backup 6.5 - have just switched to SAN based backup (without network fail over)

Backup Method is Revered Incremental

Full backups already exist - I am seeing speeds of 20MB/s - is this normal?

EVA SAN using Fibre disks.


thanks

[foggy]

Sam, what is you target storage and could you also provide backup job bottleneck stats, please? The speed is indeed not high.

[samuk]

can you please tell me how to get the stats out?

Backups till now have been over the network - and i have changed to san based..

Should i maybe delete the backups and run a new full backup?

[foggy]

Bottleneck stats are available in the job session log (right-click the job and select Statistics).

Creating new full is not required.

[samuk]

this of for one of the servers:

Code: Select all

25/01/2014 21:16:44 :: Queued for processing at 25/01/2014 21:16:44
25/01/2014 21:29:51 :: Required backup infrastructure resources have been assigned
25/01/2014 21:29:52 :: VM processing started at 25/01/2014 21:29:52
25/01/2014 21:29:52 :: VM size: 440.0 GB
25/01/2014 21:29:53 :: Using source proxy VMware Backup Proxy [san]
25/01/2014 21:30:19 :: Preparing guest for hot backup
25/01/2014 21:30:23 :: Creating snapshot
25/01/2014 21:30:30 :: Releasing guest
25/01/2014 21:30:48 :: Saving '[SQL Store] Server/Server.vmx'
25/01/2014 21:30:53 :: Saving '[SQL Store] Server/Server.vmxf'
25/01/2014 21:30:58 :: Saving '[SQL Store] Server/Server.nvram'
25/01/2014 21:31:04 :: Hard Disk 1 (40.0 GB)
25/01/2014 21:35:06 :: Hard Disk 2 (20.0 GB)
25/01/2014 21:37:08 :: Hard Disk 3 (100.0 GB)
25/01/2014 22:05:49 :: Hard Disk 4 (130.0 GB)
25/01/2014 23:04:29 :: Hard Disk 5 (150.0 GB)
25/01/2014 23:10:41 :: Truncating transaction logs
25/01/2014 23:10:42 :: Removing VM snapshot
25/01/2014 23:13:28 :: Swap file blocks skipped: 22.0 MB
25/01/2014 23:13:28 :: Finalizing
25/01/2014 23:13:48 :: Network traffic verification detected no corrupted blocks
25/01/2014 23:13:48 :: Busy: Source 8% > Proxy 20% > Network 0% > Target 98%
25/01/2014 23:13:48 :: Primary bottleneck: Target
25/01/2014 23:13:48 :: Processing finished at 25/01/2014 23:13:48

[veremin]

“Target” bottleneck refers to disk writer component that, in your case, seems to spent most of its time performing I/O to backup files. So, switching proxy mode doesn't seem to make any difference.

In order to increase backup performance you might want to switch to forward incremental mode as it’s more storage-friendly than reversed incremental one.

Also, I'm wondering what type of repository you're using.

Thanks.

[samuk]

not copying the read speed for some reason

25/01/2014 21:31:04 :: Hard Disk 1 (40.0 GB) read at 17MB/s (CBT)
25/01/2014 21:35:06 :: Hard Disk 2 (20.0 GB) read at 19MB/s (CBT)
25/01/2014 21:37:08 :: Hard Disk 3 (100.0 GB) read at 23MB/s (CBT)
25/01/2014 22:05:49 :: Hard Disk 4 (130.0 GB) read at 25MB/s (CBT)
25/01/2014 23:04:29 :: Hard Disk 5 (150.0 GB) read at 22MB/s (CBT)

[foggy]

What kind of target storage do you have?

[samuk]

Backups are going to Fata storage on the same eva.

Transport Mode set to - Direct SAN access (the backup server has the esx datastores mapped)
Connected databases - manual selection ( i added the esx datastores manually)

max concurrent tasks are set to 2

throttling not set.



not sure if the above is relevant.

[samuk]

i will switch to network based backup and run a test backup and compare it to the san based..

[veremin]

According to the provided statistics, it seems like corresponding device used as a backup repository doesn't handle well the load put on it by reversed incremental mode. Switching to forward incremental mode might be the way to go.

Thanks.

[samuk]

A Test Forward incremental back is very different

25/02/2014 12:36:38 :: Busy: Source 99% > Proxy 90% > Network 0% > Target 0%

25GB vmsize - it says it read at 5MB/s - processing rate says 67MB/s



Veeam Agent is killing the cpu on the backup server.. i have one other job running at the same time. Single Processor Xeon 2.27 GHz (shall i get a second cpu installed)

[foggy]

Please note that there always will be a bottleneck. If you would run the same job you ran previously and compare the overall job duration, you'd notice the difference the backup mode change gives you.

[dellock6]

Even if it's not directly related to the thread, I've understood correctly the Veeam repository is anoher LUN inside the same HP EVA Storage? This is considered a really bad practice, if you loose the storage, you loose at the same time both production VMs and their backups.
Also, it could be in part responsible of the bad performances, since the storage processor of the array is responsible for both the reads from production luns, and read/writes into Veeam area.

Luca.