Large Fileserver - Replication recommendations?

VMware specific discussions

Large Fileserver - Replication recommendations?

Veeam Logoby kawiMTF » Wed Feb 10, 2016 1:04 pm

Hello together

At a customer we have about 30 VMs on four hosts and a shared NetApp NFS storage. We do a daily backup to disk (reverse incremental mode) on the productive system and an overnight replication to a second site (standalone ESXi server with its own NetApp NFS storage) over a synchronous 50 MBit link. We do also hourly storage snapshots on the productive system to lower the RTPO in case of a VM or user failure. (Backup server: Windows Server 2012 R2).

The backup works fine as expected. The replication works also fine, but with one big issue: large disks (VMDKs).

For example the fileserver of this customer. This VM has two disks (VMDKs). One small disk for operating system (Windows Server 2008 R2) and a big disk for data, which is about 1.2 TB in size.

Everytime when we do have to increase the size of this data disk, the replication takes some days to complete, because of calculation of digests and fingerprints.

I thought that i might split the data on new but smaller disks. For the enduser it makes no difference, a network share is a network share, it doesn't matter on which disks it resides.

So if then a disk has to be increased, the whole calculation of digests and fingerprints will be much faster and the replication don't need days to complete. At least that's my thought.

What do you think? How do you replicate "big" servers?

Thanks for your tipps!
Best regards,
Karl

-----------------------
Veeam Platinum Partner
Veeam Vanguard / VMCE v9 / VMTSP v9 / VMSP v9
vExpert / VCA6-DVC / VCA5-DVC / VCA5-Cloud
MCP / MCSA 2012
Personal blog: https://www.driftar.ch
Twitter: @widmerkarl
kawiMTF
Veeam Vanguard
 
Posts: 116
Liked: 20 times
Joined: Tue Jan 06, 2015 10:03 am
Location: Switzerland
Full Name: Karl Widmer

Re: Large Fileserver - Replication recommendations?

Veeam Logoby foggy » Wed Feb 10, 2016 2:36 pm

Karl, I'm not sure digests calculation will involve the resized disk only. Need to confirm, but my understanding was that the entire VM is recalculated. How about simply adding a new disk, instead of resizing existing one? This will preserve all existing replica restore points, btw.
foggy
Veeam Software
 
Posts: 14765
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Large Fileserver - Replication recommendations?

Veeam Logoby kawiMTF » Wed Feb 10, 2016 4:23 pm

Hi foggy,

Yes that is one idea, adding additional disks to the existing disk. Or as i mentioned, migrate all the data and split them up to a bunch of smaller disk. I don't see another way to improve the replicaton.

Thats correct, if any change in disk size is made, the whole disk will be recalculated. I saw that so many times, i can clearly confirm that :)
Best regards,
Karl

-----------------------
Veeam Platinum Partner
Veeam Vanguard / VMCE v9 / VMTSP v9 / VMSP v9
vExpert / VCA6-DVC / VCA5-DVC / VCA5-Cloud
MCP / MCSA 2012
Personal blog: https://www.driftar.ch
Twitter: @widmerkarl
kawiMTF
Veeam Vanguard
 
Posts: 116
Liked: 20 times
Joined: Tue Jan 06, 2015 10:03 am
Location: Switzerland
Full Name: Karl Widmer

Re: Large Fileserver - Replication recommendations?

Veeam Logoby foggy » Wed Feb 10, 2016 4:25 pm

I meant that it should re-calculate digests for the entire VM, not just the resized disk.
foggy
Veeam Software
 
Posts: 14765
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Large Fileserver - Replication recommendations?

Veeam Logoby kawiMTF » Wed Feb 10, 2016 4:30 pm

Yes, thats correct. I checked the logs and there you can see in the history that every disk (so the whole VM) is recalculated).
Best regards,
Karl

-----------------------
Veeam Platinum Partner
Veeam Vanguard / VMCE v9 / VMTSP v9 / VMSP v9
vExpert / VCA6-DVC / VCA5-DVC / VCA5-Cloud
MCP / MCSA 2012
Personal blog: https://www.driftar.ch
Twitter: @widmerkarl
kawiMTF
Veeam Vanguard
 
Posts: 116
Liked: 20 times
Joined: Tue Jan 06, 2015 10:03 am
Location: Switzerland
Full Name: Karl Widmer

Re: Large Fileserver - Replication recommendations?

Veeam Logoby foggy » Wed Feb 10, 2016 4:37 pm

Then splitting to smaller disks will not help.
foggy
Veeam Software
 
Posts: 14765
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Large Fileserver - Replication recommendations?

Veeam Logoby albertwt » Mon Dec 12, 2016 11:10 pm

Hi Karl,

So how did you manage to replicate the big VM server ?

I'm curious to know :o
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 607
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Large Fileserver - Replication recommendations?

Veeam Logoby antipolis » Tue Dec 13, 2016 3:03 pm

me too ! :?
antipolis
Enthusiast
 
Posts: 45
Liked: 6 times
Joined: Wed Oct 26, 2016 9:17 am

Re: Large Fileserver - Replication recommendations?

Veeam Logoby Superkikim » Mon Jan 02, 2017 12:44 pm

Would be interesting to have some more details from Veeam on how it works.

We have 2 3TB file server we try to replicate to a distant site with w 20Mb/s WAN.

Seeding procedure was not done correctly, and digest had to be recalc... too 6 days per server. One of them is now ok, and replication takes now about 4 hours. For the second one, I'm struggling ... The seeding backup is now old, and I have restarted/change settings) multiple times hoping to get better results.

I'm now trying to get a local backup, and I will use a backup copy to sync the seeding backup at destination... Still... struggling hard.

Would be great to have a best practice taking in account the whole chain: Local Backup - Remote Backup - Remote Replication - Local Backup Archiving - Remote Backup Archiving...

Because Ultimately, that's what should be the ideal architecture. It should contain all of these element. And if you don't understand perfectly how Veeam works, it makes a hell of a difference on performance.

As un example, if your primary storage for backup is a dedup box, well good luck. All post processing tasks will be very slow... And if you want to make backup copies, it will read form the Dedup box... rehydrating data etc....

So far, to understand all this, I had to read many best practices document and other publications. It's a shame. Doesn't make the job easy.
Superkikim
Influencer
 
Posts: 16
Liked: 1 time
Joined: Thu Jul 15, 2010 9:26 pm

Re: Large Fileserver - Replication recommendations?

Veeam Logoby foggy » Tue Jan 03, 2017 10:51 am

With such a slow link, it is typically recommended to perform seeding by physically moving the locally created backup to remote location, importing it there, and mapping the remote job to it. And, of course, using WAN acceleration is a must.
foggy
Veeam Software
 
Posts: 14765
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Large Fileserver - Replication recommendations?

Veeam Logoby CMBMike » Wed Jan 04, 2017 2:29 pm

This is why I use DFS for file server replication. IMO using native technologies suchs as DFS, SQL Always On, Exchange DAG, etc. are the way to go for large servers.
CMBMike
Novice
 
Posts: 3
Liked: 1 time
Joined: Wed Nov 09, 2011 7:14 pm
Full Name: Mike Chmiel

Re: Large Fileserver - Replication recommendations?

Veeam Logoby Superkikim » Thu Jan 05, 2017 5:43 pm

foggy wrote:With such a slow link, it is typically recommended to perform seeding by physically moving the locally created backup to remote location, importing it there, and mapping the remote job to it. And, of course, using WAN acceleration is a must.


This has been done obviously. Imagine 6TB of data through 20Mb/s WAN ....

no, the slowness is due to calculating digest. And I don't get why. The job says Target WAN is the bottleneck, but when I check the target WAN accelerator performances stats, they are OK. hard to tell what is really the bottleneck here.

Perhaps changing the block size on the cache partition would help ? any advice ?
Superkikim
Influencer
 
Posts: 16
Liked: 1 time
Joined: Thu Jul 15, 2010 9:26 pm

Re: Large Fileserver - Replication recommendations?

Veeam Logoby antipolis » Fri Jan 06, 2017 8:43 am

Superkikim wrote:no, the slowness is due to calculating digest. And I don't get why.

from what I understand veeam needs to checksum the blocks to figure out which data is already present at destination, but yes, it is a painfully slow process
antipolis
Enthusiast
 
Posts: 45
Liked: 6 times
Joined: Wed Oct 26, 2016 9:17 am

Re: Large Fileserver - Replication recommendations?

Veeam Logoby PaulITGuy » Mon Jul 10, 2017 8:58 pm

I too struggle in this area. I would love to see better veeam procedures on best practices for replicating a large VM that has it's disks resized from time to time. Based on my forum searching, this seems to be an area where many users struggle.
PaulITGuy
Novice
 
Posts: 8
Liked: 2 times
Joined: Fri May 08, 2015 12:16 am
Full Name: Paul S

Re: Large Fileserver - Replication recommendations?

Veeam Logoby antipolis » Tue Jul 11, 2017 8:35 am

here is what I did in the end : created a big 10 TB VMDK, and for capacity extensions I plan on adding more VMDK and extending the existing volume with the windows LVM (spanned dynamic volume)

from my testing this setup has the advantage of not triggering a checksum
antipolis
Enthusiast
 
Posts: 45
Liked: 6 times
Joined: Wed Oct 26, 2016 9:17 am


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 18 guests