Backup of really big vm

[okrehan]

Hi all,

next week we have a proof-of-concept installation at one of our custtomer's site. They have 6 esx server (vSphere 4.0) attached to a HP EVA8100.
We will face the challenge to backup three really huge VMs with 8-10TB size each. Backup will be done because the fileservers inside have millions of small files and normal backup solution takes forever to backup this amount of data.
As I've read in the forum and other ressources, you should normally not exceed the limit of 2tb within a single job (I never found an explanation why but probably because of the time the backup will take and size of the corresponding snapshot file) but this won't be possible for the filers.
Has some experience with that amount of data within a single vm? From some other tests I found that backing up such huge vms starts quite fast but gets slower and slower after ~1TB until it nearly stops. This was with version 4 of VBR. No idea if this will also happen in v5.

Thanks for any comments on that issue.

Regards,
Oliver

[Gostev]

As I've read in the forum and other ressources, you should normally not exceed the limit of 2tb within a single job

This recommendation is not related to our product. Or may be you are confusing this with VMware limitation of max 2TB VMDK size.

Has some experience with that amount of data within a single vm? From some other tests I found that backing up such huge vms starts quite fast but gets slower and slower after ~1TB until it nearly stops. This was with version 4 of VBR. No idea if this will also happen in v5.

We have tested this scenario specifically after it was suggested that this issue exists, and confirmed that it does not. The speed is absolutely constant throughout the whole large VMDK.

Thanks.

[okrehan]

Hi,

thanks for your quick reply. I git that 2TB limitation from that post: http://www.veeam.com/forums/viewtopic.php?f=2&t=406
It seems that this issue is resolved in the meanwhile - perfect!

We have tested this scenario specifically after it was suggested that this issue exists, and confirmed that it does not. The speed is absolutely constant throughout the whole large VMDK.

Hope this "problem" is specially solved with version 5 because we have had definetly issues with big VMs in version 4. Even after asking Veeam support for help we couldn't solve that problem and had to switch back to standard backup mode via DataProtector. But nice to hear that the problem shouldn't arise in the latest version.

Regards,
Oliver

[Gostev]

Uh, yes, that topic is from 3 years ago

We did not "fix" anything specifically around big VMDK support, but we do not see this issue in our lab with v5.

[jbsengineer]

We have over 200 machines being backed up by VEEAM. We definitely see an issue with slow downs on the longer backups. It doesn't have to be a large VMDK. If the backup runs long it tends to get slower over time. Unexplainable other than VEEAM itself. We looked into all over bottlenecks (I/O, CPU, Mem, etc).

This is VEEAM 4. Hopefully VEEAM5 fixes this.

[bc07]

If you backup big VM's make sure that the source and destination storage handles single-threaded squential reads/writes well. Most storages systems are designed for multi-threaded applications (if you have RAID configured other the RAID3).

Keep in mind when you have big VM's you cannot split the backup in multiple jobs for this VM, only one job can run on each VM at the same time(only one snapshot allowed).

[SteveNeruda]

We are completing our POC and one of the test cases is a VM with a 4TB filesystem of very small tiff images. We only see about 16G of changed data to this filesystem a day. Our current backup system is bogging down due to too many small files (and probably too few directories) resulting in an 8-10 hour backup time. This data is stored on virtual mode RDMs and configured in windows as a single drive letter.

The initial Veeam backup took 31 hours across a 1GB link. I didn't try SAN mode due to fear of masking the LUNs to the Veeam server and having it either resignature or result in some kind of SCSI contention issue. Daily Veeam backups using CBT take just under 20 minutes consistently. It's really incredible.

What I have noticed with the having several larger luns is a significant pause during the snapshot and snapshot removal. The machine is stunned for almost a minute when the snapshot is taken and sometimes as long as 1.5 minutes when it is removed. This is almost assuredly a VStorage API issue not technically a Veeam one, which means any product that uses the VStorage API will see this. I'm currently testing with a VSphere 4.x host, I haven't had a chance to test with an ESXI host.

Make sure you budget for snapshot space since with 10TB of data the changes that will need to be written out to the snap during the initial backup could be considerable.

[TrevorBell]

It is common to lose a few "pings" when releasing snapshots my servers drop 1 or maybe 2 on a 1TB vm.. also my 1Tb file server takes between 30 secs and 1.5 mins to release the snapshot too

[Gostev]

Yes, few pings and few seconds stun is normal, 1.5 minutes stun is not.
1. ESX(i) 4.1 reduces stun times.
2. Using NFS storage and CBT makes stun times skyrocket on ESX(i) 4.1 (VMware bug)
3. Having existing snapshot on the same VM makes stun times skyrocket on ESX(i) 4.0 (don't know if this is fixed in 4.1)

[bc07]

@Steve
31 hours for 4TB is quite a long time (around 36MB/s). If you want/have to do an initial full backup again try the settings no compression and WAN target and use SAN as backup method (you don't have to finish the backup test just let it run for 20minutes you can also disabled CBT, index, app aware in this test job). Also disable dedpulication when you do/test the full backup.
I was concerned about SAN backup method but so far I had no issues but I don't have virtual mode RDM only normal vmdk's.

[SteveNeruda]

I will try no compression the next time I have a chance to test. Why would WAN target make any difference in throughput?

[Gostev]

In most cases, WAN target would actually reduce full backup throughtput noticeably because of significant processing overhead (4x more blocks to fingerprint). In fact, I have not seen WAN setting NOT dropping performance significantly with local backup target yet (under normal circumstances).

Theoretically though, under certain artificial circumstances (no compression AND very slow backup target), better deduplication caused by the fact that WAN setting is using 4x smaller block size could reduce the backup size so significantly, that it will have noticeable positive backup performance effect due to slow target (less data to write to slow target means the job completes faster due to less time spent waiting for target to accept new portion of data).

From my side, I definitely recommend leaving default settings (LAN target) when looking for best performance backing up locally.

[bc07]

Just try it with one vmdk file with no compression and WAN target. Let it run for 30min and check the processing rate compared to what you normaly have. And let us know the result.

[Reimold]

[merged]

Hi, right now we are installing a new Windows 2008 Fileserver with a capacity of 10-12 TB of Data. The data stored on that server will be acronis-images, so that i think there is not much more compression of that data possible. Could it be a problem when i backup that server because it will create a big backup-file of a few TB.

Dirk