Long backup time

VMware specific discussions

Long backup time

Veeam Logoby jurriaan_cap » Wed Sep 02, 2015 8:22 am

I recently switched over from TSM for VE to Veeam for my daily backup.
I noticed that the veeam backup is taking much longer than the old TSM backup
its around 132 vm's at the moment and 11.7 TB of data, of which 3 TB is read and 1.4 TB is processed.

it took around 23 hours though and had the following bottlenecks source 70 , proxy 73 network 41 target 9
I have added another proxy and the time went down with 8 or 9 hours. but it still longer than my planned backup window of 12 hours.

I have added another proxy but unfortunaly the total process time does not seem to improve , the proxy bottleneck has gone down ofcourse but source stays around 70 %
I have been monitoring whats happening and I think the main problem is that I seems to take long amounts of time to start the backup.

for example , I now have 9 vms stuck at 0 % and one is processing data .
the 0 % vm are waiting for something the disks are mounted at my proxy however there doesn not to seem to be any activity yet.

cpu , disk and memory is all at near zero. during this period
as far as I can tell its not reading the CBT , doing a dedup or whatever. it seems to be waiting for something.

the source and the target story show no latency or extremely high read or write IOPS during the backup tasks.
even when data is processed all my source SAN's seem to be handeling it fine. no latency or high iops or rates.

once it runs I get typical around 200 mb of throughput which is normal for the backup repository and network.
when data is processed all my source SAN's seem to be handeling it fine. no latency or high iops or rates.
all my vms report normal operation during processing.

on backup proxy servers I have increased the max concurrent tasks to 8
and on the backup repository I have removed the checkbox of limit maximum tasks and limit combined data rate.
also I have enabled parallel processing

this seem to have increased the number of vm's which are waiting at 0 % but nothing else.

anyone got a clue on how to increase the speed ?
jurriaan_cap
Influencer
 
Posts: 15
Liked: 1 time
Joined: Tue Feb 03, 2015 2:22 pm
Full Name: Jurriaan Cap

Re: Long backup time

Veeam Logoby foggy » Wed Sep 02, 2015 9:52 am

Jurriaan, could you please describe your setup in a bit more detail? What kind of primary storage do you have, what transport mode is being used to retrieve VM data from it? If you select the particular VM (that shows 0%) in the list in job statistics window, what task takes the most time before VM processing is actually started?
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Long backup time

Veeam Logoby jurriaan_cap » Wed Sep 02, 2015 11:42 am

I have multiple SAN systems , 2x fiber channel SANs , and 2 x NFS datastores on a NAS
both fiber channel sans are scaled for low latency and high iops. the limits of which are not encounterd during the backup.

vsphere is 5.1 using a enterprise plus license.

the backup repository is on a MSA isci device.

I think every VM is being backuped using Hotadd I have not seen any network mode backups yet.

during the 0 % phase it will give me the message
"using backup proxy xxxx for disk harddisk x (hotadd).

when I check the proxy server I see a number of disks added in offline mode
there is 0 cpu disk or memory activity for a extended period

when the disk backup realy starts I see cpu cycles/memory and disk. increasing to indicate its deduping , writing or reading.
I see no hard bottleneck though, cpu never gets higher than 50 % and memory is around 75 % free , the 10 gb interface does not fill up.

a typical backup looks like this
Code: Select all
1-9-2015 18:06:48 :: Queued for processing at 1-9-2015 18:06:48
1-9-2015 18:06:51 :: Required backup infrastructure resources have been assigned
1-9-2015 18:06:55 :: VM processing started at 1-9-2015 18:06:55
1-9-2015 18:06:55 :: VM size: 80,0 GB (47,9 GB used)
1-9-2015 18:07:43 :: Inventorying guest system
1-9-2015 18:07:47 :: Preparing guest for hot backup
1-9-2015 18:07:52 :: Creating snapshot
1-9-2015 18:08:00 :: Releasing guest
1-9-2015 18:11:11 :: Saving [xxxx] x.vmx
1-9-2015 18:11:12 :: Saving [xxxx] x.vmxf
1-9-2015 18:11:12 :: Saving [xxxx] x.nvram
1-9-2015 18:11:52 :: Disk xxxx_2.vmdk has been skipped because it was excluded from processing by user
1-9-2015 18:11:52 :: Disk xxxx_4.vmdk has been skipped because it was excluded from processing by user
1-9-2015 18:11:52 :: Disk xxxx_3.vmdk has been skipped because it was excluded from processing by user
1-9-2015 18:11:52 :: Disk xxxx_1.vmdk has been skipped because it was excluded from processing by user
1-9-2015 18:12:19 :: Using backup proxy VMware Backup Proxy for disk Hard disk 2 [hotadd]
1-9-2015 18:12:24 :: Using backup proxy mtnl0126.intra.local for disk Hard disk 1 [hotadd]
1-9-2015 18:27:55 :: Hard disk 2 (40,0 GB) 5,0 MB read at 5 MB/s [CBT]
1-9-2015 18:28:09 :: Hard disk 1 (40,0 GB) 2,4 GB read at 135 MB/s [CBT]
1-9-2015 19:02:46 :: Removing VM snapshot
1-9-2015 19:04:08 :: Finalizing
1-9-2015 19:05:06 :: Truncating transaction logs
1-9-2015 19:05:25 :: Swap file blocks skipped: 655,0 MB
1-9-2015 19:05:26 :: Busy: Source 80% > Proxy 43% > Network 18% > Target 5%
1-9-2015 19:05:26 :: Primary bottleneck: Source
1-9-2015 19:05:26 :: Network traffic verification detected no corrupted blocks
1-9-2015 19:05:26 :: Processing finished at 1-9-2015 19:05:26
jurriaan_cap
Influencer
 
Posts: 15
Liked: 1 time
Joined: Tue Feb 03, 2015 2:22 pm
Full Name: Jurriaan Cap

Re: Long backup time

Veeam Logoby foggy » Wed Sep 02, 2015 12:24 pm

Do you see similar behavior regardless of the storage VMs being backed up reside on? Hotadd from NFS storage does not perform well due to NFS locking issues. You can either try network mode (since you're on 10GB LAN, should give you good performance and allow to eliminate or confirm hotadd impact) or make sure you have a proxy on every host running VMs residing on NFS datastores and configure EnableSameHostHotaddMode registry key to use only proxies on the same host as the VM in question.

Also, we're adding direct NFS in the upcoming v9, something else to consider.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Long backup time

Veeam Logoby jurriaan_cap » Wed Sep 02, 2015 1:27 pm

yes i see exactly the same on both NFS datastores and on FC datastores.

i have over 15 Esx hosts, running proxy's on all of them is not really a option .

i have installed patch 2030 , it reads that there were improvements to the backup process. ill see what that does.
if that does not have a positive result i will try the network mode backup.

any other suggestions are welcome though.
jurriaan_cap
Influencer
 
Posts: 15
Liked: 1 time
Joined: Tue Feb 03, 2015 2:22 pm
Full Name: Jurriaan Cap

Re: Long backup time

Veeam Logoby jurriaan_cap » Wed Sep 02, 2015 1:36 pm

after installing patch 2030 it seems that more backups are now running in parallel.

i will check it tomorrow and see if it has improved
jurriaan_cap
Influencer
 
Posts: 15
Liked: 1 time
Joined: Tue Feb 03, 2015 2:22 pm
Full Name: Jurriaan Cap

Re: Long backup time

Veeam Logoby jurriaan_cap » Thu Sep 03, 2015 8:37 am

whoa ! patch 2030 made a huge impact !!

backup time went from 15+ hours plus to 2 hours !.

throughput is basicly the same ,but it just process more tasks at the same time
jurriaan_cap
Influencer
 
Posts: 15
Liked: 1 time
Joined: Tue Feb 03, 2015 2:22 pm
Full Name: Jurriaan Cap

Re: Long backup time

Veeam Logoby foggy » Thu Sep 03, 2015 9:49 am

Yep, Update 2 has actually introduced some optimizations in performing job auxiliary tasks. Glad you were able to get such an improvement.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Long backup time

Veeam Logoby jurriaan_cap » Thu Sep 03, 2015 12:21 pm 1 person likes this post

well personally I wouldn't call going from 15+ hours to 2 hours optimization. improvements like that sound more like that something was broken and it got fixed. anyways I am glad the backup windows is finally within expected specs.
jurriaan_cap
Influencer
 
Posts: 15
Liked: 1 time
Joined: Tue Feb 03, 2015 2:22 pm
Full Name: Jurriaan Cap

Re: Long backup time

Veeam Logoby foggy » Thu Sep 03, 2015 7:41 pm 1 person likes this post

Sometimes minor optimizations lead to major results. ;)

To define the actual reasons of reduced job duration in your case, logs of the jobs that took long should be closely investigated.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 18 guests