Is anyone successfully replicating large data offsite?

[bunger]

We currently utilize a local and cloud-based backup solution, but costs are incrementing exponentially as our storage requirements increase. To mitigate that, I am considering moving to Veeam to local onsite backups, and then replicating those backups over a site-to-site VPN connection to a remote office. The core office connection is 100mb/7mb and the remote site is 60mb/5mb.

We currently backup 7 servers with rough total backup size of about 3TB.

I have been reading the forums and seeing mixed results when trying to backup/replicate offsite with large amounts of data.

Is anyone successfully doing this, or should I be looking at a different product?

[veremin]

Actually, a lot of our customers are doing exactly that; thanks be to our backup copy job. Also, it has GFS retention functionality guaranteeing your long-time archival goals are met as well as WAN Acceleration helping to speed up the copy process in case of slow links. Thanks.

[bunger]

Performing a forum search returns multiple results with people encountering issues with WAN Acceleration and related replication, which is what prompted my initial query.

[veremin]

It might stand to reason, then, to install a trial version and see how everything goes for you. Nothing but actual tests will allow you to make a final decision regarding product implementation. Thanks.

[csinetops]

I'll throw in our experience. Running Version 8 Update 2. We backup copy about 40 VM's and 10TB ( total VM size not incremental ) worth of data over our 100MB MPLS connection between MN and TX every night. This runs in a 8HR window pretty much every night. We started by running local backup copy jobs in MN, saving them to a large NAS and shipped them to our secondary site in TX. Once there, we copied them to the TX target repository and remapped the backup copy jobs, then kicked the jobs off again and let them sync up. The initial sync's will take some time but do eventually complete. Ours took about a week to get in sync again.

The incremental jobs are pretty small. For example, our largest copy job is 3.8TB and the nighty's that get sent down are ~ 14GB. There is about 57 GB of CBT data read in this job and only 14GB sent, a 6x savings. Some of our other jobs get 17x+ savings. Of course this will vary depending on your change rate etc. Veeam is a very efficient data mover.

We did run into a issue with our Exchange backup. That VM is very large at around 7TB split between 4 disks. Veeam couldn't get through the disks in our initial re-sync window so we had to give up on getting that machine to our secondary site. This isn't a Veeam issue but a infrastructure limitation at the second site as well as a dumb design decision on our side. If the disks were smaller Veeam would have no issue getting through them in the window.

Like the others have said, demo it and see if it will work. I think if you seed the data and utilize the feature set built into Veeam you won't have any issues. Let me know if you want more detail on our setup or have more questions.

[Gostev]

We did run into a issue with our Exchange backup. That VM is very large at around 7TB split between 4 disks. Veeam couldn't get through the disks in our initial re-sync window so we had to give up on getting that machine to our secondary site. This isn't a Veeam issue but a infrastructure limitation at the second site as well as a dumb design decision on our side. If the disks were smaller Veeam would have no issue getting through them in the window.

Have you considered seeding this VM?

[Delo123]

@Bunger we also are testing veeam for replicating big vm's (small ones we are doing since forever and never had issues). We ran into a couple of issues with 4TB+ VM's and apparently most of the issues are related to offloading settings on the nics. We currently have an open ticket but offloading seems to be the root cause. I will let you know in a couple of days if this solved all our issues and then you can do some testing of your own

[nunciate]

We have a 10Tb File server which we replicate to our DR facility nightly. It consists of 5 2Tb drives on that single server. Here is my experience with that.

First I will say that going to our old DR facility replicating this server was a nightmare. We had a 100/mbps connection to that site and the SAN backend for our VMware environments were sub-par. We had older NetApp storage very slow.
To replicate such a large VM I had to backup the VM here onto tape. Ship the tape out there and then restore it. Once restored (which took a week to get it there) I ran the replication job and it took Forever! to run. It took so long to sync up that it failed. The servers run on snapshots while they replicate and those snapshots became enormous and took a long time to remove as well. Eventually the only way I was able to get it synced up was to exclude all drives but 1 on the first run and then add 1 drive at a time and run the job until the entire thing was synced. As I said our SAN was crap so I kept running into issues of snapshot removal taking too long and the jobs would fail. I had multiple times where the replica was broken and I had to resync. Resyncing like that took about a week. It was a nightmare. None of this was a Veeam problem just to be clear.

Last year we transitioned to a new DR facility. We replaced the SAN in production with a Dell Compellent and purchased a similar but slower Compellent in DR. Compared to the old NetApps these new SANs are lightning fast. We also upgraded the network and now have a gigabit link between the sites and added VMware capacity to match production which we didn't have in the old site.

The goal in the new site was to replicate all VMware VMs to the site or to have hot/hot fail over ability. For this server I again seeded with tapes. I then ran a single run to do the full sync back to here. It completed within 48hours without error and I have been running that job with no issues for many months now.

I will also mention that I seeded every VM we have using a Veeam backup server which I had here onsite. I performed backup copies of all VMs to that server, shipped it to DR and restored all of my VM to DR. I then ran replication jobs to sync up every VM nightly. If I had to estimate I would say around 60Tb of VM data was restored and synced up. As I said earlier, this was many months ago and I have been successfully running those replications jobs every night with no issues.

So yes, Veeam 8 works great with replicating large amounts of data and large VMs to DR as long as you have the resources on the network and storage to handle such an operation. I will be upgraded to v9 as soon as the first update is released.

[dellock6]

Thanks Alan for posting your experience, very appreciated.
I believe, based on your network, you are not using WAN acceleration, correct?

@bunger: in your case, replica will use up to 7 Mb/s badnwidth, which is the upload bandwidth of your primary site. With this amount of bandwidth, you should consider to use our WAN accelerator and the initial seeding.

Luca

[nunciate]

We are not using Wan Acceleration today as we do have a gigabit connection so it is not needed.

I did attempt use the Wan Accelerator with my large VM but had issues as my accelerator cache was never too big.
I never had time to tinker with it to get it working before we started moving to the new site. I did use it on some smaller VMs and it did help.

[ckent]

Im using offsite cloud connect (3rd party) as opposed to replicating to another site of my own.
my dataset is around 7TB.

Over a 20/20 line i never managed to get a copy offsite as the initial seed just took too long.
After moving to a 100/100 line it works a lot better.

Is the upload at your main site of 7mb going to be problematic ?