Seeding remote datastore

[Tomsyr]

I'm having issues with setting up an off-site target. I've been through the instructions, but still no luck.
I created a backup copy job that went to a usb external disk. It was sent to the remote datacenter, and uploaded to the target server. I've set up the WAN accelerators, and followed these instructions:
http://www.veeam.com/kb1856

The problem is that WAY too much data is being transferred. I seeded with 2.2 TB's of data, and so far, I've had to 'sync' about 500 GB over the wire, and the job progress is showing that it has a long way to go. I put in a support ticket, but I'm being told that everything seems to be working and everything is fine.
Case # 00873219
In the time between creating the backup and up till yesterday, the amount of changes based on the incrementals equal about 35 gigs.

any suggestions?
Thanks!
Tom

[Gostev]

Hi, Tom

Yes, it does not seem right from your description.

I recommend performing the seeding test locally first:
1. Create a new seed.
2. Put it into some other local storage.
3. Create a new backup job mapping into the seed, and run it (note Transferred size).

This way, you can ensure you are doing everything correctly. Obviously, if your first incremental size will be more than a few GB in a matter of hour since seed creation, then something is wrong.

Thanks!

[Tomsyr]

Hi Gostov,
I was able to do a successful test on a small VM. I sent new copies down to the datacenter, and for a smaller site (500 GB) I was able to get to work correctly. It took a very long time for the fingerprint process to finish.
I started the larger site yesterday, and it looks like the fingerprint process went very slow as well, and at midnight it failed. I suspect that it was because I have my backup are copies set to run at midnight daily. I disabled backups of all jobs (not copies). I figured that since backups didn't run, it wouldn't need to copy.
Anyway, the remote job kicked off again, and I'm back to where I was last week - massive backup going over the wire, and it's not using the seed.
I had to cancel the job.
Is there anyone at Veeam support that can help me with this? We have invested 16 sites into using the Enterprise version of the product. I'm going to upload the logs from the current job.
Also - how can I restart the process without having to create yet another new backup copy to send to the remote data center. Can I put the original metadata file with the full backup file and delete all the current time stamped files?
It seems like if I can get the fingerprint process completed, it will work as needed.
Any estimate on how long 2.2 TB's should take for the fingerprint creation?
Thanks,
Tom

[foggy]

Is there anyone at Veeam support that can help me with this?

You can ask your support engineer to take a look at this.

Also - how can I restart the process without having to create yet another new backup copy to send to the remote data center. Can I put the original metadata file with the full backup file and delete all the current time stamped files?

I believe this is possible.

[Tomsyr]

I've spoken to an engineer about seeding remote datastores, and was told that it does take a very long time to create fingerprints. Each disk needs to be created for each VM - In my particular case right now, it's 3 disks. Drive 1 is small, so just a few hours for that. Drive 2 much bigger (1.6 TB) took 24 hours. Disk 3 failed at 79%. Since it was not able to finish the digest, it reverted back to pulling all data across the wire for disk 3. 65 hours into the job, I'm at 88% completed, and now the bandwidth is impacted for production while I wait for this to complete.

We are not using SSD for the global cache on either end, which seems to be the ONLY mention about improving performance. Is there anything else to consider? As mentioned, we invested quite a bit on Veeam for 16 sites - specifically so that we can back up remotely. We've spent a a good deal of time upgrading sites to V8 to take advantage of WAN performance. What else can we do?

[foggy]

Tom, just to check, what kind of repository do you have on remote side?

[Tomsyr]

Sorry for the late response - I didn't get a notification.
We are running EMC VNX SATA disks.

[foggy]

I mean, how it is added to Veeam B&R console (what type of backup repository)?

[Tomsyr]

Not sure if I am following you...
It was a new backup copy job where the target was a USB drive. After creating, I disabled the job so it wouldn't run again. After the contents of the disk were copied to the remote target, I did a re-scan of the remote repository, then mapped it within the backup copy job I had disabled. I followed these instructions:
http://www.veeam.com/kb1856

I really didn't do anything at the remote site's Veeam console except for rescanning the repository there too. Is there anything that should be done on this console?

[foggy]

I did a re-scan of the remote repository, then mapped it within the backup copy job I had disabled.

I was juts wondering what kind of repository it is (Windows/Linux server, CIFS)?

[Tomsyr]

Gotcha - Windows...

[foggy]

And Windows server is in the same location with EMC device, right? What about WAN accelerator?

It really depends on the target storage I/O capabilities (hence, SSD recommendations), but I would say that 24 hours that it took to calculate digests for 1.6 TB of data is an acceptable speed (18 MB/s).

[Tomsyr]

I'm now trying another site - 35 hours into it, with backups turned off so that is does not interrupt the process. Processed 643 GB out of 2.5 TBs.
Obviously not acceptable.
If I were able to stage the backups on SSD in the remote datacenter, would that fix the performance issues, or is it needed on the source site as well?
I could probably do that for the remote site, but not for the source sites. The console is saying the bottleneck is the Target WAN.
I'm thinking that I could present the drive, get the site to sync, then move it to the slower storage.
The documentation only mentioned SSD for the WAN cache.
Am I missing something? What are others doing to use remote datacenters? Is the initial time to sync just something that has to happen?
Thanks,
Tom

[foggy]

If I were able to stage the backups on SSD in the remote datacenter, would that fix the performance issues, or is it needed on the source site as well?
I could probably do that for the remote site, but not for the source sites. The console is saying the bottleneck is the Target WAN.

Yes, in this case getting SSD disks for target WAN accelerator should increase performance.

[Tomsyr]

We're still having issues after a weekend of using the SSD. Is there professional services that are available for Veeam? Anyone to recommend?
Thanks,
Tom

[pawel]

I have been having horrible performance issues with the WAN accelerator as well. We work with about 1TB of data across 2 disks and I've had to give up for now and am using basic replication without the WAN accelerator. The veeam engineer suggested SSDs on both ends which is what I was eventually going to buy but looks like you are having issues with those too.

The digests/fingerprints took FOREVER. If they completed the job would still fail. It's been nothing but problematic for me. Have you tried a transfer without the WAN accelerator? For 2.2TB of data you might be able to get it done in about 8 hours (it takes 4 hours for my 1TB of data). I actually came here to see if 4 hours is normal for how little incremental changes I'm transferring but saw your thread so figured I'd chime in so I can get updated on it.

Thanks.

[Tomsyr]

I'm glad I'm not the only one having issues
I haven't tried without the WAN accelerators, as that would defeat the purpose. Both sites could handle it as they both have good connections. Right now it seems like one site is taking advantage of the WAN accelerator, but is incredibly slow, and the other site seems to be ignoring the WAN accelerator and is flooding my connection (50 mbs).
I had 4 TB's of SSD provisioned at the target, and have not seen much of an improvement. I do see that the 'Global Cache' folder is filling up - it's at 1.36 TB's. I set each site to use 200 GB's, so now I have something else wrong.

[foggy]

This is not expected, please let your engineer know about that.

Btw, how many sites are using/concurrently using this WAN accelerator?

[pawel]

Just to verify you are using the SSDs just for the WAN accelerator cache or for everything? As I understand it you should only use it for the WAN accelerator cache as far as the WAN acceleration goes. My engineer actually suggested I clear the cache (I had mine set at 50GB) and start the job with no cache at all letting it populate on its own. Same issues, but might be worth a try for you.

I would try without the WAN accelerator just to see, you might be surprised, Veeam does seem to do a very good job of just sending the changed data over the main data link. However, it does send a lot of processing traffic between the backup server and the esxi server so I have a private link between the main Veeam server and the vSphere server so that the production link doesn't get saturated with processing data. I don't know how much data gets changed on your end but in our production site it's about 5-10GB a day and Veeam is really good about compressing it and sending just the changes. But it does have to process the entire VM which takes way more time than I think it should (which is a separate issue that I need to ask about in another thread) and as I mentioned above does send lots of traffic on the local subnet for that processing if you don't have a dedicated link for that (which in my case is just a cross over cable).

Also, make sure you have a recent replica of your production server at the DR site and that it is mapped properly in the job settings so it knows only to send the changed data, otherwise it will try to send your entire VM.

[Tomsyr]

I have been working with a support engineer. I was told that SP 2 was just released, so I've gone ahead and upgraded all sites. We're currently replicating 6 sites without any issues - all are very small (< 500 GB) and daily changes typically less than a gig. ALL my issues revolve around the large sites (only 2 have been tried so far) and seeding. 2 small sites had the seeding work correctly. The others I just did without seeding due to their small sizes.

For 1 of the larger sites, I used SSD fr both the WAN Cache and the data storage. No real improvement. Another big issue that I came across was the WAN cache filling up the drive 2 times. The first time it killed the job, the second time I killed the job while the Engineer was on line with me. The Cache had used up 1.4 TB's, and growing. we confirmed the settings on the 2 servers using the WAN Cache as having 100 gb allocated, so it's a mystery why it was filling up. Hopefully SP 2 fixes that, but at this point I'm trying to sync up without the WAN cache.

[Tomsyr]

I've been able to complete both sites WITHOUT the WAN accelerators. Now the question is should I implement the WAN accelerators, or will it have to go through the fingerprint pain (MANY hours to complete).
I would like to take advantage of not using so much bandwidth on a daily basis.

Thanks,
Tom

[foggy]

On the first pass after turning WAN acceleration on, the target global cache will be rebuilt and digests will be calculated on source.

[Tomsyr]

Does this mean that I'll need to plan for 48+ hours again, or did SP 2 fix this?
I've needed to disable all backups during this process since it interrupts the digest.

[foggy]

There were not improvements to this process in Update 2, as far as I know.

[pawel]

Tom, can you keep this thread updated with your progress and I'll update you with mine? Sounds like we're having a very similar issue.

Thanks!

[kmile]

Hello,

We are running Veeam B&R 8 (ver. 8.0.0.20.84) Enterprise Plus.
We recently had problems upgrading our ESXi hosts 5.5 to 5.5 Update 3b. It turned out that the vCenter server had to be upgraded first, then the hosts.
After the upgrade all Veeam backup and replication jobs failed. (Error: No connection could be made because the target machine actively refused it vCenterServer IP address:443)

I then went to the properties of each job and re-added the VMs to process. That worked. However all replication jobs are now calculating digests and creating finger prints. It takes forever! Our file server has a 1 TB hard disk and calculating digests took over 27 hours. Now it is on to creating fingerprints. After 7h 30 min its is on 19% (it transferred only 331MB of data....

Our biggest server (exchange) is yet to be done! it will probably take a week if not more....Is there any way to skip/speed up the process
Connection at source 20MB down, 4MB up. Connection at target 20MB down, 20MB up. (data transfer over WAN) We even upgraded to the Enterprise Plus version of Veeam hoping it would speed things up with WAN accelerators but did not see any noticeable difference.

Also we have two Veeam proxies - at the production site and at the DR site.

Any ideas other than upgrading our Internet connection? Is the speed for calculating digests and creating fingerprints normal?

Your help is as always much appreciated.

Regards,
Kamil

[Shestakov]

Hello Kamil,

Having that relatively slow connection in mind, the shown performance doesn`t look abnormal. Starting from the second run everything should work much faster.

Besides the network upgrade, getting SSD disks for target WAN accelerator is to increase performance.
You could also populate global cache.

Thanks!

[kmile]

Thanks.

Can you think of the reason why upgrading the vCenter server caused the backup and replication jobs to fail with Error: No connection could be made because the target machine actively refused it vCenterServer IP address:443 ?

Also, would you be able to tell me what it is actually triggering the digest calculation and creation of fingerprints? I know of two scenarios:
1. first replication
2. if I change the size od a VM disk at the source.

What has to actually happen beside the reasons I gave to trigger these lengthy processes?

Thanks,
Kamil

[foggy]

Can you think of the reason why upgrading the vCenter server caused the backup and replication jobs to fail with Error: No connection could be made because the target machine actively refused it vCenterServer IP address:443 ?

Looks like some kind of a connection issue, logs should tell exactly.

Also, would you be able to tell me what it is actually triggering the digest calculation and creation of fingerprints? I know of two scenarios:
1. first replication
2. if I change the size od a VM disk at the source.

Upgrade process could result in VM ID's change (this should not occur in case of a proper in-place upgrade, though). After re-adding them to the job, Veeam B&R has to re-calculate differences between them and their replicas.