Veeam B&R slow

[royp]

Got the very latest veeam 7 patches on.

I have put in an extra ESXi host called ESXI5 (NOT IN Vsphere as its temporary) target to run a couple of replicas over to using the seed option. (I assume this is the fastest option for the first replica run)
I have a win 7 virtual machine on the ESXI5 HOST set up as a proxy.
Getting a horrendous throughput and bottleneck is showing as Target.
The transport mode at the moment is set to auto on the proxy, is this the fastest option.
The data stores on this new ESXI5 host are local drives on the host.

The data coming from the source is on a HP LeftHand San to this ESXI5 Hosts local storage.

12/05/2014 15:37:27 :: Queued for processing at 12/05/2014 15:37:27
12/05/2014 15:37:27 :: Required backup infrastructure resources have been assigned
12/05/2014 15:37:29 :: VM processing started at 12/05/2014 15:37:29
12/05/2014 15:37:29 :: VM size: 2.0 TB (1.2 TB used)
12/05/2014 15:37:31 :: Discovering replica VM
12/05/2014 15:38:14 :: Creating VM snapshot
12/05/2014 15:38:28 :: Preparing replica VM
12/05/2014 15:38:38 :: Processing configuration
12/05/2014 15:38:49 :: Creating helper snapshot
12/05/2014 15:38:59 :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd]
12/05/2014 15:38:59 :: Using target proxy 192.9.200.23 for disk Hard disk 1 [hotadd]
12/05/2014 15:39:02 :: Calculating digests Hard disk 1 (60.0 GB)
12/05/2014 15:44:49 :: Hard disk 1 (60.0 GB) THIS ONE TOOK 17MINS TO COMPLETE
12/05/2014 16:02:26 :: Using backup proxy VMware Backup Proxy for disk Hard disk 2 [nbd]
12/05/2014 16:02:27 :: Using target proxy 192.9.200.23 for disk Hard disk 2 [hotadd]
12/05/2014 16:02:29 :: Calculating digests Hard disk 2 (500.0 GB)
12/05/2014 16:41:56 :: Hard disk 2 (500.0 GB)
HARD DISK 2 ONLY ON 8% AFTER 16 HRS AND 25 MINS

[foggy]

According to the bottleneck stats, it is clear that the issue is write speed to the target storage. You could try to force network mode for the target proxy and see whether the processing speed becomes better (hotadd can put high I/O load on the target in some cases). Btw, what is the RAID configuration on the target host? Do you have write-back caching turned on?

[royp]

Thanks Alexander speedy reply again.
Originally the target was in another building prior to writing on here.
Connected via fibre as only 100 meters apart.
Ive now put the target in with the source, same switch etc same room, made no differance, infact it timed out doing the Calculating digests.

After your reply Ive checked and Write Back Caching is turned on, and its on a Raid 5.
Just about to set the target proxy to Network Mode and see what that does.

[foggy]

Ive now put the target in with the source, same switch etc same room, made no differance, infact it timed out doing the Calculating digests.

Calculating digests step should not be performed during subsequent incremental job runs.

After your reply Ive checked and Write Back Caching is turned on, and its on a Raid 5.

It is generally recommended to have RAID10 for minimal I/O penalty.

Just about to set the target proxy to Network Mode and see what that does.

Keep us posted.

[royp]

After moving the target into same server room as source I noticed the reason the target was as you mentioned bottlenecking, was because it was on a 10/100 switch.
Plugged it into the same gig switch that the source is on and left it running yesterday thinking hmmm this looks like it may work.
Came in this morning to the below.

Code: Select all

14/05/2014 13:55:48 :: Queued for processing at 14/05/2014 13:55:48
14/05/2014 13:55:48 :: Required backup infrastructure resources have been assigned
14/05/2014 13:55:50 :: VM processing started at 14/05/2014 13:55:50
14/05/2014 13:55:50 :: VM size: 2.0 TB (1.2 TB used)
14/05/2014 13:55:52 :: Discovering replica VM
14/05/2014 13:56:30 :: Creating VM snapshot
14/05/2014 13:56:48 :: Preparing replica VM
14/05/2014 13:56:57 :: Processing configuration
14/05/2014 13:57:20 :: Using backup proxy 192.9.200.23 for disk Hard disk 2 [nbd]
14/05/2014 13:57:21 :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd]
14/05/2014 13:57:21 :: Using backup proxy VMware Backup Proxy for disk Hard disk 5 [nbd]
14/05/2014 13:57:21 :: Using backup proxy vproxyb1 for disk Hard disk 3 [nbd]
14/05/2014 13:57:21 :: Using backup proxy vproxyb1 for disk Hard disk 4 [nbd]
14/05/2014 13:57:21 :: Using target proxy 192.9.200.23 for disk Hard disk 2 [nbd]
14/05/2014 13:57:21 :: Using target proxy VMware Backup Proxy for disk Hard disk 1 [nbd]
14/05/2014 13:57:21 :: Using target proxy VMware Backup Proxy for disk Hard disk 5 [nbd]
14/05/2014 13:57:21 :: Using target proxy vproxyb1 for disk Hard disk 3 [nbd]
14/05/2014 13:57:21 :: Using target proxy vproxyb1 for disk Hard disk 4 [nbd]
14/05/2014 13:57:21 :: Calculating digests Hard disk 1 (60.0 GB)
14/05/2014 13:57:21 :: Calculating digests Hard disk 5 (500.0 GB)
14/05/2014 13:57:23 :: Calculating digests Hard disk 2 (500.0 GB)
14/05/2014 13:57:24 :: Calculating digests Hard disk 3 (500.0 GB)
14/05/2014 13:57:25 :: Calculating digests Hard disk 4 (500.0 GB)
14/05/2014 14:38:27 :: Hard disk 1 (60.0 GB)
14/05/2014 17:23:46 :: Hard disk 2 (500.0 GB)
14/05/2014 18:33:59 :: Hard disk 3 (500.0 GB)
14/05/2014 18:36:26 :: Hard disk 5 (500.0 GB)
14/05/2014 18:37:16 :: Hard disk 4 (500.0 GB)
15/05/2014 03:11:44 :: Removing VM snapshot
15/05/2014 03:12:47 :: Error: Client error: ChannelError: ConnectionReset
15/05/2014 03:12:47 :: Busy: Source 44% > Proxy 22% > Network 41% > Target 58%
15/05/2014 03:12:47 :: Primary bottleneck: Target
15/05/2014 03:12:47 :: Network traffic verification detected no corrupted blocks
15/05/2014 03:12:47 :: Processing finished with errors at 15/05/2014 03:12:47

Its showing Processing rate 39MB/s
Processed 1.4tb 100%
Read 1.4 tb
Transferred 600.8gb (2.5x)
Speed 92.4 MB/s

VMware Replica job: FILESERVER 
Created by domain\user at 12/05/2014 15:34:47. Error
1 of 1 VMs processed  
14 May 2014 13:54:37 
Success 0 Start time 13:54:37 Total size 1.4 TB Backup size 96.8 GB Processing 'esx3_Fileserver' Error: Client error: ChannelError: ConnectionReset 
Warning 0 End time 03:12:55 (+1) Data read 1.4 TB Dedupe 1.0x 
Error 1 Duration 13:18:18  Transferred 600.8 GB Compression 1.0x 
Details  
Name Status Start time End time Size Read Transferred Duration Details 
esx3_Fileserver Error 13:55:50 03:12:47 (+1) 1.4 TB 1.4 TB 600.8 GB 13:16:56  Error: Client error: ChannelError: ConnectionReset 
 

Veeam Backup & Replication 7.0.0.839

[foggy]

Bottleneck stats numbers are almost equally distributed among all the components in this example, which tells there's no real bottleneck in such configuration (and you get pretty decent speed for the network mode).

There's an existing thread regarding the error you get and I recommend contacting support directly with this. Btw, is this error 100% reproducible during each job cycle?

[royp]

Thanks for the above.
I started a fresh ie deleted the replica off the disk.
Created a completely new veeam rep job, no tick in replica mapping as I now dont have a rep, selected get seed from the following backup repository.
set it off and at the moment it is a lot further ahead than previously.
Will create a support case Monday if it fails on this job.

In the running job it has the one vm in twice,
Once as a seed and that is at 70% and the 2nd instance does not have [seed] next to it and that is pending waiting for the seed job to finish. Total percentage is 44%.
Will notify you monday if it completes or post support ticket number if I have to create a support case.

Again thanks for the quick replies.

Thanks for the above though Alexander

[foggy]

In the running job it has the one vm in twice,
Once as a seed and that is at 70% and the 2nd instance does not have [seed] next to it and that is pending waiting for the seed job to finish. Total percentage is 44%.

You're welcome, though I'm not sure regarding the quoted above - could you please elaborate on this? Are you saying you have the same VM added to the job twice?

[royp]

Not sure myself to be honest why it does this.
In the job setup the VM been backed up d10_fileserver Ive only added it once in there.
Yet if I go onto the running job statistics.
Into last 24hrs
Into running.
Then highlight that running job in the main right hand window.
Down below under Job Progress section you have your running servers been backed up.
Even though I only have one server been backed up, for some bizarre reason it is in there twice.
The one that is in progress right now states = esx3_d10_fileserver [seed]
Thats on 71%
Above that and pending is = esx3_d10_fileserver

If I DO NOT have a replica which I do not at the moment as the job is running, BUT I DO have a previous BACKUP ie veeam backup job of this same server and when running a replica veeam job and point >>Get seed from the following backup repository<< to where the veeam BACKUP (not rep) but backup files are, it adds the server in twice when running.

[foggy]

Got it, this is expected. The [seed] one refers to the first processing stage - restoring the VM from backup to the target host, while the second is actual synchronization of changes between the original VM and the seed restored from backup.

[royp]

Works fine now.
As your link above suggests, just starting with no replica from the outset seems to do the trick which of course most people would do. I guess and it is only a guess that the cause is because I originally created a replica with just the system drive selected which completed successfully
THEN
On the next job I selected ALL DISKS to be included in the replica and I had pointed to that first system drive only replica.

Original successful replica took 11 hours which was expected been the first run of a VM just over 2tb in total size.
Just completed the incremental replica in just over 20 mins.

Nice one yet again Alexander, thanks for your help.

[foggy]

Glad it is running fine now, thanks for the update. Feel free to ask for any additional clarification, if required.