Backup Copy Job Very Slow

[JaxIsland7575]

I put in 2 new Veeam Backup Servers (update 2) that are Dell R730XD with (2) Intel Xeon E5-2623 v3 @ 3GHz CPUs, 32 GB RAM, running 2012 R2 64bit, with 10TB RAID1 internal storage. There are 2 AD domains that are connected via the LAN. There is no domain trust so each VBR server is backing up a different domain set. Because the serves are on opposite sides of the campus I have setup a backup copy job between them so there are 2 copies of the backups. Yesterday was the first day I had it installed and ran a single backup job on the exchange server. At 1800 EST last night the backup copy interval started and at 0914 today (15+ hours later) the job is still running and is at 74%. Its reporting a 5MB/s rate during the job.

So site A ran exchange backup and started a copy job to site B. That transferred 104GB in 9.5 hours.

Right now site B is running the backup copy job to site A and that is at 183GB (74%) in 15+ hours.

The bottleneck says "Target" but the network is Gig with fiber backbone connecting the ends of the campus. I have never seen a backup copy job running so slow before, anyone have any ideas? There are no other jobs running on either server, and no roles added either. They are strictly VBR servers. I did add VEB to them to backup the C:\ Volume, but that job ran in a few minutes at 0400 without issue.

Cheers!

Edit: Your support case number is 01014062

[foggy]

Ryan, bottleneck target means that the write speed to the target storage is the weakest point in data processing chain. How the repository is added into Veeam B&R console?

[JaxIsland7575]

It is setup as CIFS, should it be changed to a MS Server repository?

[foggy]

Yes, that would be a better design. Let's see whether this helps.

[JaxIsland7575]

I recreated the repositories and I am only getting about 15MB/s so it looks like this will take overnight to finish. I will watch it and see how it goes. I had to cancel the site B transfer, it was going to hit the new cycle before it finished the first one.

[foggy]

Is target still the bottleneck?

[JaxIsland7575]

Says "Network" for bottleneck now.

Thanks!

[JaxIsland7575]

Ok, problem solved. This was not related to Veeam but a VLAN issue between the servers. It looks like there was a VLAN that allowed traffic it should not. In the original setup the servers should have never been able to reach each other. The resolution is to fix the VLAN but I added a 2nd network card to each server that is on each others network. Now they can talk directly to each other and the backup copy jobs went as quick as I expected.

Cheers!

For those that are interested here is the times:
9.5 hours now runs at 22 minutes
15+ hours now runs at 45 minutes

[JaxIsland7575]

I thought I would add to this post because my fix for the backup copy jobs between servers introduced a new issue. Now all my Backup jobs on my second site were slow, and I mean, really slow. 1-2MB/s slow. I had a test server that I had setup to trial and do a proof of concept at the site with, brought that up and ran some backups and everything went well. No issues. I go back to the production server and I cant get past 4 MB/s for a Backup job.

The issue was introducing 2 NICs to VBR allowed it to try and use the wrong NIC for the backup jobs, so I had to go into VBR and set the preferred network to the primary network the server was on. I did that on both sides, and last night I got expected backup behavior on both sides.

Hope it saves someone a couple days of troubleshooting

Cheers!

[leem]

Thanks for posting. That very well may have been the fix to a problem I was having awhile back.

[tommls]

I look at the current progress of a Backup Copy between two sites...and most days it works 'successfully'.

Source site has really slow internet and uses a WAN accelerator, the data is backed-up vSphere VMs on a Qnap NAS.
Destination site has fast internet and a WAN accelerator, the copied stuff goes into a Exagrid backup box.

However when it's done doing the backup copy it''s working on creating restore points...and this is why I wonder if Backup Copy is too slow:
As of today it's creating a restore point for 11/21/15!! -- and merging the backup files takes HOURS AND HOURS.
Because the upload from source is only a 3M connection I can often see in the history that the Backup Copy takes a full 24 hours -- 12 am to 12 am...

Would someone please elucidate if this is good, bad, or just something we must put up with??

Thank you, Tom

[Shestakov]

Hello Tom,
Your backup copy job performance doesn`t look good. It`s better to start from the bottleneck statistics. Could you provide it?
How much data do you transfer? 3 Mbps will not let you show good performance if you have big files to transfer.
Thanks!

[foggy]

How is your Exagrid box added to Veeam B&R console (what type of repository is used)? Also, what kind of retention do you have for the backup copy job and what are the settings for Exagrid's landing zone (when the data starts deduplicate)? My assumption is that you have retention of 14 days and the merge process into an already deduped full backup file takes long due to required rehydration.

[tommls]

@foggy @shestakov I'll copy/paste this stuff into a support case plus upload logs. I can't do anything about the network/Internet speeds though. Thank you, Tom

[Shestakov]

Tom, please provide your support case number for us to follow the case. Thanks!