Seeding remote datastore

VMware specific discussions

Re: Seeding remote datastore

Veeam Logoby pawel » Mon Apr 27, 2015 6:44 pm

I have been having horrible performance issues with the WAN accelerator as well. We work with about 1TB of data across 2 disks and I've had to give up for now and am using basic replication without the WAN accelerator. The veeam engineer suggested SSDs on both ends which is what I was eventually going to buy but looks like you are having issues with those too.

The digests/fingerprints took FOREVER. If they completed the job would still fail. It's been nothing but problematic for me. Have you tried a transfer without the WAN accelerator? For 2.2TB of data you might be able to get it done in about 8 hours (it takes 4 hours for my 1TB of data). I actually came here to see if 4 hours is normal for how little incremental changes I'm transferring but saw your thread so figured I'd chime in so I can get updated on it.

Thanks.
pawel
Enthusiast
 
Posts: 34
Liked: 3 times
Joined: Thu Nov 20, 2014 10:54 pm

Re: Seeding remote datastore

Veeam Logoby Tomsyr » Tue Apr 28, 2015 9:38 am

I'm glad I'm not the only one having issues ;)
I haven't tried without the WAN accelerators, as that would defeat the purpose. Both sites could handle it as they both have good connections. Right now it seems like one site is taking advantage of the WAN accelerator, but is incredibly slow, and the other site seems to be ignoring the WAN accelerator and is flooding my connection (50 mbs).
I had 4 TB's of SSD provisioned at the target, and have not seen much of an improvement. I do see that the 'Global Cache' folder is filling up - it's at 1.36 TB's. I set each site to use 200 GB's, so now I have something else wrong.
Tomsyr
Influencer
 
Posts: 22
Liked: never
Joined: Tue Jul 01, 2014 3:39 pm
Full Name: Tom Conklin

Re: Seeding remote datastore

Veeam Logoby foggy » Tue Apr 28, 2015 12:41 pm

This is not expected, please let your engineer know about that.

Btw, how many sites are using/concurrently using this WAN accelerator?
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Seeding remote datastore

Veeam Logoby pawel » Tue Apr 28, 2015 3:23 pm

Just to verify you are using the SSDs just for the WAN accelerator cache or for everything? As I understand it you should only use it for the WAN accelerator cache as far as the WAN acceleration goes. My engineer actually suggested I clear the cache (I had mine set at 50GB) and start the job with no cache at all letting it populate on its own. Same issues, but might be worth a try for you.

I would try without the WAN accelerator just to see, you might be surprised, Veeam does seem to do a very good job of just sending the changed data over the main data link. However, it does send a lot of processing traffic between the backup server and the esxi server so I have a private link between the main Veeam server and the vSphere server so that the production link doesn't get saturated with processing data. I don't know how much data gets changed on your end but in our production site it's about 5-10GB a day and Veeam is really good about compressing it and sending just the changes. But it does have to process the entire VM which takes way more time than I think it should (which is a separate issue that I need to ask about in another thread) and as I mentioned above does send lots of traffic on the local subnet for that processing if you don't have a dedicated link for that (which in my case is just a cross over cable).

Also, make sure you have a recent replica of your production server at the DR site and that it is mapped properly in the job settings so it knows only to send the changed data, otherwise it will try to send your entire VM.
pawel
Enthusiast
 
Posts: 34
Liked: 3 times
Joined: Thu Nov 20, 2014 10:54 pm

Re: Seeding remote datastore

Veeam Logoby Tomsyr » Tue Apr 28, 2015 8:50 pm

I have been working with a support engineer. I was told that SP 2 was just released, so I've gone ahead and upgraded all sites. We're currently replicating 6 sites without any issues - all are very small (< 500 GB) and daily changes typically less than a gig. ALL my issues revolve around the large sites (only 2 have been tried so far) and seeding. 2 small sites had the seeding work correctly. The others I just did without seeding due to their small sizes.

For 1 of the larger sites, I used SSD fr both the WAN Cache and the data storage. No real improvement. Another big issue that I came across was the WAN cache filling up the drive 2 times. The first time it killed the job, the second time I killed the job while the Engineer was on line with me. The Cache had used up 1.4 TB's, and growing. we confirmed the settings on the 2 servers using the WAN Cache as having 100 gb allocated, so it's a mystery why it was filling up. Hopefully SP 2 fixes that, but at this point I'm trying to sync up without the WAN cache.
Tomsyr
Influencer
 
Posts: 22
Liked: never
Joined: Tue Jul 01, 2014 3:39 pm
Full Name: Tom Conklin

Re: Seeding remote datastore

Veeam Logoby Tomsyr » Wed Apr 29, 2015 1:15 pm

I've been able to complete both sites WITHOUT the WAN accelerators. Now the question is should I implement the WAN accelerators, or will it have to go through the fingerprint pain (MANY hours to complete).
I would like to take advantage of not using so much bandwidth on a daily basis.

Thanks,
Tom
Tomsyr
Influencer
 
Posts: 22
Liked: never
Joined: Tue Jul 01, 2014 3:39 pm
Full Name: Tom Conklin

Re: Seeding remote datastore

Veeam Logoby foggy » Wed Apr 29, 2015 3:44 pm

On the first pass after turning WAN acceleration on, the target global cache will be rebuilt and digests will be calculated on source.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Seeding remote datastore

Veeam Logoby Tomsyr » Thu Apr 30, 2015 11:40 am

Does this mean that I'll need to plan for 48+ hours again, or did SP 2 fix this?
I've needed to disable all backups during this process since it interrupts the digest.
Tomsyr
Influencer
 
Posts: 22
Liked: never
Joined: Tue Jul 01, 2014 3:39 pm
Full Name: Tom Conklin

Re: Seeding remote datastore

Veeam Logoby foggy » Thu Apr 30, 2015 3:27 pm

There were not improvements to this process in Update 2, as far as I know.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Seeding remote datastore

Veeam Logoby pawel » Fri May 01, 2015 1:19 pm

Tom, can you keep this thread updated with your progress and I'll update you with mine? Sounds like we're having a very similar issue.

Thanks!
pawel
Enthusiast
 
Posts: 34
Liked: 3 times
Joined: Thu Nov 20, 2014 10:54 pm

[MERGED] Replication issues?

Veeam Logoby kmile » Wed Feb 03, 2016 4:19 pm

Hello,

We are running Veeam B&R 8 (ver. 8.0.0.20.84) Enterprise Plus.
We recently had problems upgrading our ESXi hosts 5.5 to 5.5 Update 3b. It turned out that the vCenter server had to be upgraded first, then the hosts.
After the upgrade all Veeam backup and replication jobs failed. (Error: No connection could be made because the target machine actively refused it vCenterServer IP address:443)

I then went to the properties of each job and re-added the VMs to process. That worked. However all replication jobs are now calculating digests and creating finger prints. It takes forever! Our file server has a 1 TB hard disk and calculating digests took over 27 hours. Now it is on to creating fingerprints. After 7h 30 min its is on 19% (it transferred only 331MB of data....

Our biggest server (exchange) is yet to be done! it will probably take a week if not more....Is there any way to skip/speed up the process
Connection at source 20MB down, 4MB up. Connection at target 20MB down, 20MB up. (data transfer over WAN) We even upgraded to the Enterprise Plus version of Veeam hoping it would speed things up with WAN accelerators but did not see any noticeable difference.

Also we have two Veeam proxies - at the production site and at the DR site.

Any ideas other than upgrading our Internet connection? Is the speed for calculating digests and creating fingerprints normal?

Your help is as always much appreciated.

Regards,
Kamil
kmile
Enthusiast
 
Posts: 33
Liked: 1 time
Joined: Fri Dec 16, 2011 10:13 am
Full Name: Kamil Smagorzewski

Re: Seeding remote datastore

Veeam Logoby Shestakov » Wed Feb 03, 2016 5:05 pm

Hello Kamil,

Having that relatively slow connection in mind, the shown performance doesn`t look abnormal. Starting from the second run everything should work much faster.

Besides the network upgrade, getting SSD disks for target WAN accelerator is to increase performance.
You could also populate global cache.

Thanks!
Shestakov
Veeam Software
 
Posts: 4861
Liked: 395 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Seeding remote datastore

Veeam Logoby kmile » Wed Feb 03, 2016 5:29 pm

Thanks.

Can you think of the reason why upgrading the vCenter server caused the backup and replication jobs to fail with Error: No connection could be made because the target machine actively refused it vCenterServer IP address:443 ?

Also, would you be able to tell me what it is actually triggering the digest calculation and creation of fingerprints? I know of two scenarios:
1. first replication
2. if I change the size od a VM disk at the source.

What has to actually happen beside the reasons I gave to trigger these lengthy processes?

Thanks,
Kamil
kmile
Enthusiast
 
Posts: 33
Liked: 1 time
Joined: Fri Dec 16, 2011 10:13 am
Full Name: Kamil Smagorzewski

Re: Seeding remote datastore

Veeam Logoby foggy » Mon Feb 08, 2016 4:20 pm

kmile wrote:Can you think of the reason why upgrading the vCenter server caused the backup and replication jobs to fail with Error: No connection could be made because the target machine actively refused it vCenterServer IP address:443 ?

Looks like some kind of a connection issue, logs should tell exactly.

kmile wrote:Also, would you be able to tell me what it is actually triggering the digest calculation and creation of fingerprints? I know of two scenarios:
1. first replication
2. if I change the size od a VM disk at the source.

Upgrade process could result in VM ID's change (this should not occur in case of a proper in-place upgrade, though). After re-adding them to the job, Veeam B&R has to re-calculate differences between them and their replicas.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Previous

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 10 guests