Backing up large file shares

[BarryH]

We have a file server that has about 10 TB of data on one drive. I have removed that drive from the Veeam backup and I am backing up the data with a different system. I was under the impression that it was not a good idea to snapshot that large of a disk, but it would be nice to use Veeam for all of my backups. What is the best practice for backing up that much file data?
Thanks

[DaveWatkins]

I don't have anything that big but I don't see why it would be a problem. The major concern I would expect is filling up the LUN it's hosted on if the change rate is high, but given it's a file server that's probably fairly unlikely. If the change rate is fairly low the snapshots shouldn't be around that long, the hardest part will be the initial full backup and the time it takes to do that

I assume you can't break it down into smaller drives by splitting the data up?

[csinetops]

I have a couple large file servers with multiple large disks around 4TB each. While they aren't 10TB like yours , Veeam doesn't seem to mind bigger disks. One thing to keep in mind is recovery speed, the bigger the disk(s) the longer it's going to take to recover.

[foggy]

VM of that size shouldn't be a problem provided you're meeting the backup window. Please see another thread regarding similar matter.

[nunciate]

I have several file servers that I backup using reverse incremental with Veeam. One of the servers has 5x 2Tb drives so a total of 10Tb usable. Windows 2012 Dedup is turned on for all these file servers so on this big one I am maybe using 5Tb of that space. The other servers are pretty large as well. I backup with no issues. If you are using Windows 2012 servers for your repositories are are using 2012 dedup on those repositories it is a good idea to exclude the folder that contains your vbk file as it can get corrupted when deduping a file that large.

[BarryH]

Thanks everyone, great information. Hopefully I can archive off some of that data before using Veeam. I'll be happy to have all of my backups under one roof!

[Delo123]

Hi Barry,

we have multiple 10TB+ Fileservers (even with Windows Dedupe enabled so actual data is about 3-5x that much). Due tue Dedupe changerate is quite high but still we have no issues at all backing them up with veeam since i can remember.
Restoring the VM in an emergency is certainly not the easiest (quickest), but that is what replica's are for If your underlying Storage / Repository is performant there should be no issues, but since it seems like you are doing some kind of agent backup and that's working Veeam should be your littlest problem. Have all components up to date always makes a big difference. ReFS and upcoming vmfs-6 when veeam support is released should be a great combination

[smd32]

I haven't had >5TB for a VMDK backup, but these have been working reliably for a couple of years now up to that size. Lots of 1TB Exchange volumes that even on a busy server, can get backed up successfully.

What I could not get to work with a 15TB volume (about 10TB of data) was VEB. It would BSOD the server running VEB before getting to half completed. This is an iSCSI disk on a 2012 cluster for file shares, not VMDK. We went back to using the original product from a different vendor to back things up, but I'm hoping the upcoming Veeam Agent for Windows will be able to handle this.

[ejfarrar]

I have MANY VMs with over 10TB drives - most of my NFS servers use 8TB volumes, but I have one which uses 41x 16TB vmdk's

All of my VMs back up daily. Granted, the larger VMs (60+TB) take about 1 day to do full backups using storage integration - if using NBD and I get good speeds for the throughput, it usually takes about 2-3 days. Daily incremental backups using CBT take about 10-30 minutes for these big ones. The only potential failure point I encounter with any frequency (one big VM every few months) is with vmdk consolidation issues (similar to VMware's phantom snapshot issue).

Enjoy!

[Mike Resseler]

Scott,

Did you logged a support case for that problem with VEB? We have customers that use VEB to backup those data sizes without any issues...

Mike

[FSSC]

After seeing the sizes of some of the backups here, i'm now convinced something isn't right in our set-up. Our larger backup jobs are 1tb and 2tb or so. On avg, it takes 9 and 12 hours respectively for each incremental job to complete. I see that roughly half the data for each job is being read with actual data transfer amounts of 2.3gb and 12gb being made (not much considering)... I'm going to have my admin open a ticket, but what if anything can i have him check first?

Thanks

[foggy]

Bottleneck stats.

[Mike Resseler]

FSSC,

It might be a bit more difficult to get information from VEB compared to what VBR is giving

Let your admin check this: https://helpcenter.veeam.com/endpoint/1 ... urces.html and see if the throttle is active (might be a busy system and therefore the backup goes to the background)

But because most information can be found in the logs, it is advised to create a support case (through the VEB UI) so they can deep-dive into it. Please post your Case ID here and the results of the investigation. (But don't forget: Support call is FREE, but comes with no SLA, on a best effort basis)

Thanks

[FSSC]

Bottleneck stats.

Load: Source 98% > Proxy 54% > Network 1% > Target 0%
Duration: 12:49:21 - Processing Rate: 28 MB/s - Processed 1.8 TB / Read: 1 TB /Transferred: 5.8 GB

[FSSC]

It might be a bit more difficult to get information from VEB compared to what VBR is giving

Hi Mike and thanks for the reply. I should clarify that my question relates to VBR not VEB. I have no issues at all with VEB and wish we'd see the same BU performance from VBR that we do from VEB. I guess i'll have my team get a case open.

Thanks again

[tsightler]

If you are reading lots of data, but transferring very little, this usually indicates one of two things, something wrong with CBT, or something in-guest is touching lots of blocks, making VMware believe the blocks have been changed but, for whatever reason, there's no actual changed data. I'd suggest starting with a CBT reset, verify that CBT is actually working (i.e. no surprise HWv4 VMs), and work from there.

[ddrrdd]

10 To on only one disk can be huge for you (depend of the performance of your hardware / storage bay)

If all your data of that server is only on one drive, I kindly recommend you to split that drive in multiple smaller disk/vmdk an aggregate them in one drive/mount point using the OS disk manager / lvm.

I administer many DB and you should expect better performance with multiple stream to read/write your data than only one for your 10 To.

You should expect better virtualisation performance in many points: you can use multiple LUN / datastore in your storage, multiple virtual HBA in vmware, your backup can read multiple disk in parallel (restore will also be faster), your snapshot commit will be more efficient…

In linux lvm usage is very common, no so with Windows but there is also a performant disk manager. Pay attention to disk resizing if you plan to grow your volume and use the stripping aggregation (test the resizing need with small virtual disks before your 10 To)

Best Regard,
David

[Seve CH]

Hello.

I am against too many disks. In my experience, the more disks the VM has, the longer the stunt time is while taking or removing snapshots. That could lead to time-outs (and cluster failovers), sockets closing and so on.

For instance, we gave up with frequent incrementals or replications in our file servers (7-8 vDisk, 7-8TB) because the stunt time was 6 to 8 seconds and every snapshot operation (especially taking snapshot) generated 2000-4000 handles closed by clients and Citrix problems with profiles being inaccessible. Now we make only daily backups at night of these servers (way far from the RPO of 15 minutes for all application Veeam announces though).

We have an image archive (PACS) with the same problem wich must be backuped frequently and what we have done is to consolidate data in less disks via extend disk, add mirroring of the old partition and then remove the old disk mirror. With that we removed 10 vDisks and the stunt time of that server (8TB) went down to approx. 2 seconds and the application doesn't crash anymore (8-10 seconds stunt was too much). We didn't consolidate more because vmware doesn't allow to do hot-extend over 2TB.

In all cases the problem was not the size of the VM, but the number of disks and marginally the load on the VM and host. Tested on ESX 5.1, 5.5, 6.0. I have no idea if the stunt times are better in 6.5. It is not a Veeam problem, but a vmware "feature" (reproductible while taking manual snapshots from vSphere client of test VMs).

Best regards.

[mkreitzer]

Just don't expect to be backing them up to tape. Even though 9.5 finally made it possible to avoid having tape copy jobs terminate when the primary backup job starts, I don't think Veeam is capable of parallel tape drive usage when copying a single VM even in 9.5. We backup multiple 10TB+ VM's daily, but a single LTO6 drive takes days to copy a full backup of just one of them.

It's extraordinarily aggravating.

Also, Seve CH, you're adorable. It's "stun time", btw. It's a shame that with the huge head start VMware had that they could not implement a ReFS 3.0/NetApp/Everybody but VMware style snapshot system before Microsoft. We still have to consider snapshot deletes a potentially disruptive event.

[Delo123]

A SAN with Veeam Snapshot integration would be nice for these cases, but i stopped trying figuring out how Veeam decided who to integrate. Hp/EMC and then of all possible brands Veeam choose Nimble? Anyway, a lot of Vendors tell me Veeam has no resources and they are all ready blabla, time will tell i guess