Veeam 3.1 with ESX 3.5/ESXi 3.5

[stevenfoo]

We tested and implemented the backup for about 12 VMs over network (agentless) to NetApp 2050 sata disk NOT SAN.

We are using most default setting recommended - optimal compression.

Test 1) The Veeam 3.1 is install in a VM and backup to CIFS

VM 1, 1st time backup 103GB with 1:34hrs, 2nd incremental 7.1GB with 1:21hrs, 3rd 6GB incremental with 1:34hrs

Test 2) The Veeam 3.1 is install in a VM and backup to iSCSI (LUN)

VM 1 (same as Test 1, we removed everything and change from CIFS to iSCSI), 1st time backup 128GB with 2:00hrs

Questions:

1~ With the same backup VM 1, why there is difference of 25GB whereas we don't make any changes to the server ?
This server is sql 7.0 database. Only 1 database was drop and the size was about 5GB.

2~ Why is the second incremental backup take almost the same time as the first full backup as in Test 1) ?
Shouldn't be the backup timing is less than 5 minutes and just scan through whether any changes takes place and backup that only ?

3~ Why is the size of the incremental backup is so large, whereas there are no such large changes in the VM ?
It takes too long to backup and the size of the incremental backup does not seems to reduce it size.

Overall Veeam 3.1 backup huge vbk files even for incremental backup. Also it take too long for incremental which is the same as full backup.

Any fix to this issue as our management is not happy about it.

thanks
Steven

[Gostev]

Hello Steven,

1. These are changes made by Windows (swap file, registry files update, logs) and SQL (database and transaction logs activity). Since incremental backup is done on block level, single bit change in the 1024KB block makes whole block market as "dirty" and it will be processed as a part of incremental sync.

2. Scanning through VMDK changes requires pulling whole VMDK from the storage, so incremental synchronization speed is mostly determined by how fast the VMD files can be pulled from storage, and not by the amount of changes. For your ESX servers, I recommend enabling SSH connection and agent mode. This way, VMDK scan will be performed locally on ESX host, as opposed to Veeam Backup server in case of ESXi (agentless mode), when whole VM has to be pulled to Veeam Backup server over the network.

3. See (1) above. Defragmenting your VMs usually helps to reduce incrementals in a few times, because disk changes are becoming more physically consolidated. On fragmented VMs, when new files are created, they scatter over multiple parts of disk, and thus making more disk blocks "dirty".

Hope this helps!

[stevenfoo]

Hi Gostev,

We have defrag in Test 1) and rerun backup few times again. Initially after that the backup was huge in size properly because of the defrag.
We then move from Test 2) using iSCSI LUN to backup again. We don't see any diffrence at all in the speed.

Also the size of the backup should not include the swap file in the backup.

The incremental backup size also take too long to backup.

We have one VM 387GB size took 2hrs 34minutes to backup a 2.4GB incremental backup. This is very slow just for 2.4GB changes.
In fact the changes is less than 200MB for that VM.

Will Veeam 4.0 able to rectify this issue and reduce the incremental backup file significantly and improve the speed for ESX 3.5 / ESX 3.5i ?

Currently our management is concern as the incremental backup size is consider large for Window VM compare to tiny fraction of Linux VM.
It about 20GB per day for 13 VM (windows + linux). 14 days = 280GB. 30 = 540GB. So it take a huge amount of space to keep for 1 year = 6.5TB of space. We don't have that luxury of owning 8TB storage.

So we hope Veeam could help to overcome this issue facing by all customer like me.

Thanks
Steven

[Gostev]

Steven, yes - Veeam Backup 4.0 will have nearly instant incremental backup, with backup time mostly defined by the amount of changes. However, this will require vSphere and ESX/ESXi 4.0 hosts. Previous versions of ESX simply do not provide any change tracking capabilities, so the only way to determine the changed block for incremental backup cycle, is to pull whole VM from the disk (as I explain in point 2 above).

Thank you.

[tsightler]

Hi Gostev,
Also the size of the backup should not include the swap file in the backup.

I don't represent Veeam, just another customer, but I think you may have an incorrect idea of how Veeam works. Veeam makes a complete image of the VM just as VMDK is on your disk. The VMDK file includes the swap file so Veeam is going to back it up. I suppose you could create a dedicated VMDK file to contain you're windows swap file and choose not to backup that VMDK volume with Veeam (Veeam has the ability to select the individual volumes you want to backup) but this would add to your effort after a full VM restore, although nothing too major.

The incremental backup size also take too long to backup.

We have one VM 387GB size took 2hrs 34minutes to backup a 2.4GB incremental backup. This is very slow just for 2.4GB changes.
In fact the changes is less than 200MB for that VM.

Will Veeam 4.0 able to rectify this issue and reduce the incremental backup file significantly and improve the speed for ESX 3.5 / ESX 3.5i ?

Once again you seem to not understand how Veeam currently works. VMware ESX prior to ESX 4.0 has no block change tracking. For a product to perform an "incremental" backup the product has to scan the entire VMDK file to look for the changes so an incremental will not be significantly faster than a full backup since it requires reading the same amount of data.

Currently our management is concern as the incremental backup size is consider large for Window VM compare to tiny fraction of Linux VM.
It about 20GB per day for 13 VM (windows + linux). 14 days = 280GB. 30 = 540GB. So it take a huge amount of space to keep for 1 year = 6.5TB of space. We don't have that luxury of owning 8TB storage.

So you want to keep a year of backups on disk? Once again I don't represent Veeam, but I don't think Veeam Backup is very well suited for this case. If this is really a requirement I would suggest a different solution that only does byte level changes at the file level, or at least much smaller block level changes, there are such products out there. Veeam's block level changes are fairly large (1MB) so it doesn't take very many blocks changing to create large VBR files. I'm sure they do this for performance reasons (the smaller the units in which you track changes the more overhead, both in processing and memory, in performing rollbacks for restores).

Veeam backup is an excellent product for it's use scenario, allowing you to make easily restorable backups for your VM's with rollback capability and decent, reasonably fast, file level restore, for the rare occasion when such a restore is needed, but I personally would never use it for keeping archive type backups on disk for such a long period. If the main VBK file becomes corrupt you basically loose all of the previous backups. For archive I'd suggest keeping a shorter period of Veeam backups (we've found 31 days to work reasonably well for most VM's) and spooling those backups to tape every so often, or using a product designed for disk based archiving.

[Gostev]

I personally would never use it for keeping archive type backups on disk for such a long period. If the main VBK file becomes corrupt you basically loose all of the previous backups.

To address long-term backup retention requirement and this concern, we have added an option for Veeam Backup to perform full backup every so often (you can set this to happen periodically, or initiate it manually from UI).

[stevenfoo]

Thanks for the reply.

So I guess we have to upgrade to ESX/ESXi 4.0 to take advantage of the block incremental backup.
This will cost another huge $ to do that and having difficulty to justify.

The separation of swap file into a separate disk will then help in this case. However it still does scan the entire VM for the image.
May be some speed improvement of small %.

Could we see the long term backup retention for full backup in Veeam 4.0 ?

When will be tentative date that Veeam 4.0 be release to their customer ?

[tsightler]

And I would add another question, with block change tracking will you use smaller blocks or will VBR files continue to be based on changes in 1MB blocks? If the block size continues to be 1MB then backups will be faster with ESX 4.0 but the size will still be the same. Also, being able to run a full is a nice feature, but still doesn't really change my opinion that Veeam is not really a great archive solution.

Sevenfoo -- Did you not purchase maintenance for ESX/ESXi? If so then you upgrading to 4.0 shouldn't cost a "huge $", it really shouldn't cost anything at all, well, except for the effort of planning and implementation.

[stevenfoo]

Yes we have 1 ESX 3 with license the others are ESXi free edition. We brought Veeam before it discountinue the support.

So we planning of moving into VMware Essential lincese which will give us 3 host ESX/ESXi license at USD995.
This is a cheaper alternative for us.

normal standard version is 1-CPU USD795. if you have 2-CPU per ESX/ESXi host = USD1590.
for 3, USD4770.

tsightler, FYI we are not a huge multi national company that give us the advantage to purchase stuff.
our currency is not USD. so to convert and purchase USD cost us more and huge $.

[Gostev]

And I would add another question, with block change tracking will you use smaller blocks or will VBR files continue to be based on changes in 1MB blocks? If the block size continues to be 1MB then backups will be faster with ESX 4.0 but the size will still be the same.

We will most likely continue using 1MB block sizes for now, until we keep capability of performing service console agent backup and replication for ESX 3.5 hosts. Without going into much details, decreasing block size will affect service console agent memory usage significantly.

Steven, Veeam Backup 4.0 RTM is tentatively scheduled for the end of Q3.

[tsightler]

tsightler, FYI we are not a huge multi national company that give us the advantage to purchase stuff.
our currency is not USD. so to convert and purchase USD cost us more and huge $.

Understood, I didn't really think about the ESXi Free version and the recent changes in support I guess because I don't really consider ESXi free to be a very good product if your looking for low cost/free virtualization but I guess I can see the advantages if you have some "paid" and some "free" servers then it's nice to keep them all the same basic environment. As far as I'm concerned the ESXi free product is mostly just an attempt at grabbing customers/market share and leading them up the chain into the paid product. I'm not saying it's a particularly good strategy, but I think that's they're real goal because there are certainly far more functional "free" virtualization products out there.

[stevenfoo]

So you want to keep a year of backups on disk? Once again I don't represent Veeam, but I don't think Veeam Backup is very well suited for this case. If this is really a requirement I would suggest a different solution that only does byte level changes at the file level, or at least much smaller block level changes, there are such products out there. Veeam's block level changes are fairly large (1MB) so it doesn't take very many blocks changing to create large VBR files. I'm sure they do this for performance reasons (the smaller the units in which you track changes the more overhead, both in processing and memory, in performing rollbacks for restores).

I still don't agree with the performance reason here. I compare again with one of the full backup of one VM using Data Protector (full backup)

Backup take for 3 volume.

Total Used
C:\ 28Gb 18Gb
D:\ 300GB 197GB
E:\ 50GB 34GB

Total Used: 129GB

The total time is taken is 3hrs 5mins for full backup. Average incremental backup is 12mins.

The Veeam backup - full is about 2hrs 35mins. Incremental backup the same as full backup.

So I don't see any advantages using Veeam to backup incremental backup here. Our management rated is as very bad.

[Gostev]

Well, comparing Veeam Backup to Data Protector is really comparing apples to oranges. Data Protector does not even support incremental image-level backups. It can only perform incremental backups on file-level. While this indeed allows for faster incrementals, you loose the benefits of full image backup. You no longer have latest state of VM in form of full, ready-to-go VM image. And you know, it is not backup when clock are ticking loudest - it is restore.

[tsightler]

I still don't agree with the performance reason here. I compare again with one of the full backup of one VM using Data Protector (full backup)

Backup take for 3 volume.
Total Used
C:\ 28Gb 18Gb
D:\ 300GB 197GB
E:\ 50GB 34GB
Total Used: 129GB

The total time is taken is 3hrs 5mins for full backup. Average incremental backup is 12mins.
The Veeam backup - full is about 2hrs 35mins. Incremental backup the same as full backup.

So I don't see any advantages using Veeam to backup incremental backup here. Our management rated is as very bad.

So you're comparing an image level backup product to a file level backup product? That's like comparing apples to planets. Of course the file level backup is going to have faster incrementals, it can use the file system to determine what files have changed and what it needs to backup. An agentless image level backup product with ESX 3.5 and lower has to scan the ENTIRE VMDK FILE to find out what blocks have changed. The advantage for Veeam incrementals isn't in time (well, maybe a little, their usually a little faster), it's in the fact that you can keep 14 or more days of "complete system images" of your VM but only use an incrementally small fraction of disk space to do so.

If you don't want an image based backup solution, then Veeam is not the product you should be looking at. Of course, now the VMware has included an incremental block tracking feature in the ESX 4 product Veeam is now implementing this feature and once Veeam 4 is released incrementals should be MUCH faster.