VM Backup Speed Optimization

[sidavid]

Hi,

I would like to know if Veeam Backup v2 backup the Used Space or the Total Space of VMs?

Thanks

David

[Gostev]

Hello sidavid, could you clarify your question please?

[sidavid]

Hi Gostev,

My question is :

I have Windows 2003 Server VM. The size disk is 90GB and the Windows Used Space is 18GB (OS + Data). My veeam backup size is 78GB. So I don't understand why my veeam backup is 78GB???
I tried with other servers, Veeam backup size are the size of the Windows Used Space. (a VM with a disk size 50GB, I used 10GB and the Veeam backup size is 9GB).

Thanks

David

[Gostev]

sidavid, with the default job settings (no compression), the size of backup is only reduced by deduplication engine, which finds and removes all blocks which are completely empty (or exactly same, which happens rarely in jobs with only 1 VM).

In your case, from the results you are getting it looks like this VM has pretty badly fragmented disks, thus the de-duplication engine could not find many blocks which are completely empty. Thus, the size of backup was not reduced as significantly as for other VMs.

To reduce the size of backup of this specific machine, you need to either defragment the disks, or enable backup compression in the backup job options. Defragmenting the disks might be a good thing to do anyway, since it also increases VM performance (VMware recommends to defragment the VM disks from time to time).

[sidavid]

Thanks for your clarification.

David

[sidavid]

I defragment the disk on my Windows VM and the backup had the same size 71GB.
I did a defragment, a Veeam Full Backup, an incremental Backup.

The size disk is 90GB and the Windows Used Space is 23GB (OS + Data), I have 66GB of Free Space. My veeam backup size is 71GB.

Thanks for your help

David

[Gostev]

Hi sidavid,

Actually I've also been researching this last few days, and I found that most typical defrag tools do not wipe the unused disk space during defrag. So all the data blocks still remain "dirty" after defrag, not containing all zeroes. Thus, to achieve the desired result, you also need to wipe the unused disk space after defrag is done.

I found that VMware Tools Shrink functionality can be used to wipe the free disk space (wiping is the first step of disk shrinking), however it appears that Shrink does not support some VM disk types.

So I've done more research and found that most people are using this free tool to optimize the virtual hard drives:
http://technet.microsoft.com/en-us/sysi ... 97443.aspx

Usage: sdelete [-p passes] [-s] [-q] <file or directory>
sdelete [-p passes] [-z|-c] [drive letter]

-c Zero free space (good for virtual disk optimization).
-p passes Specifies number of overwrite passes.
-s Recurse subdirectories.
-q Don't print errors (quiet).
-z Cleanse free space.

Note the first parameter description.
You need to run the following command inside your virtual machine:

sdelete -c C:
sdelete -c D:
etc. for all drives on your VM

Please let me know if this helped.
Thanks!

[sidavid]

Thanks you Gostev.
It's working good, my size backup is 21 GB.

[Gostev]

Thank you for letting us know, this is great.
I will talk to devs to see if we can build the disk wiping optimization into the product down the road.

[tizou]

I want to warn anyone willing to use sdelete, that while running the available disk space seen by the OS may go down very low and could therefore put the running applications to hazard.
So be cautious and make tries before you schedule sdelete !

[Gostev]

Yep, it is good to know - we would need to implement wiping differently.

Meanwhile, tizou, could you recommend any other existing tools which have a better wiping implementation than sdelete?
Thank you.

[tizou]

Unfortunately, I didn't find any alternative to sdelete. I tried and older version (when sdelete source was still available) and found the same problem. My workaround is to schedule sdelete when the VMs utilization is low and to monitor the remaining disk space. Also I need to mention the the space seen is back to normal as soon as sdelete is stopped.
I'd be grateful if someone has another experience to share on this one.

[mpozar]

We have used the esXpress product and it also had the same issue. The PHD people provide a tool called PHD Zerofill that creates a file equivelant to the FREE space on the drive and fills it with Binary Zeroes. Thus when the next backup occurs the compression is very affective..


Have FUN!

Michael Pozar

[Gostev]

Thank you mpozar for the tip. To me, from your description it sounds like Zerofill is exactly identical to sdelete tool mentioned above, as it also has the same issue of taking up all the free space. We will need to consider this, and implement this feature a bit differently, if possible.

[mpozar]

Yes, ZeroFill fills up the free space and soon as it has done so it deletes the file it has created. I have been using it for a while and found that it works and has not caused me any issues.

Looking forward to see what you guys come up with.

Have FUN!

Michael Pozar

[Gostev]

Script for automated VM disks optimization, courtesy of Even Glemmestad (glemmestad)
http://forum.veeam.com/forums/viewtopic ... 2563#p2563

Note that this script addresses the lacking-free-space issue mentioned above.

[Mike]

Gostev,

Thx for the advise!

before sDelete backup size 33Gb, full backup 29MB/s

After sDelete backup size 16Gb, full backup 38MB/s

Thats a quick win!

[Gostev]

Exactly! Besides, defragmentation is also good for VM performance in general.

I especially recommend to perform this operation on newly created VM templates BEFORE producing new VMs from them (this is needed for best deduplication results with subsequent backups).

[samuk]

Never done a defrag on our disks, though have thought about it but never found the right product.

Using the Tools to shrink:

Is this an effective method?
Is there any way to know if the disks require a defrag?


Sam

[Gostev]

I would use built-in Windows defragmentator to understand whether the disk requires defragmentation, and to perform the actual defrag. Then, use the above-mentioned script. This should be best solution.

[duhaas]

How long are you folks see that batch file run? I just started it last night around 4:00 pm and its still running this morning as of 9 am

[MadDog2K]

Running SBS2003, I see more then 1 fragmentation number.
What numbers I should look for to determine if running this script is useful ?

For example :

[Volume fragmentation]
* Total fragmentation = 29%
* File fragmentation = 59%
* Free space fragmentation = 0%

Windows says I should defragment, but will it have a positive effect on Backup you think ?
Also ... since free space fragmentation is 0% (on all my disks), is running sdelete (via the script) still useful ?

Last question : the script has a variable 'StopZeroWipeAtFreeSpace' (2000MB).
Why let it stop at 2GB, and not let it go to 0MB, what are the downsides of letting it go to 0MB ?
Trying to understand the reason behind it, so I can make a desiscion on it

[Gostev]

High file fragmentation makes the same file scatter across multiple disk blocks, this makes all of those blocks "dirty" when updating a single file and thus increases the size of incremental backups (which are done on block level).

High total fragmentation makes free space chunks scatter across disk and thus reduces the amount of empty 1MB blocks that can be dropped by Veeam Backup.

SDELETE is always useful no matter of fragmentation level, because it wipes unused blocks with zeroes, making the blocks zeroed so that Veeam Backup can drop those from processing. Remember that deleting a file in Windows only update FAT but does not delete actual file content from disk (this is why you can always UNdelete any file).

The reason for stopping at 2GB is that some applications (specifically Exchange server) will automatically freeze all activities when free disk space becomes low. If you do not have such applications, then it makes sense to reduce this number.

Thanks!

[MadDog2K]

Hi Gostev,

Thanks again for your explanation.

SDELETE is always useful no matter of fragmentation level, because it wipes unused blocks with zeroes, making the blocks zeroed so that Veeam Backup can drop those from processing. Remember that deleting a file in Windows only update FAT but does not delete actual file content from disk (this is why you can always UNdelete any file).

The reason for stopping at 2GB is that some applications (specifically Exchange server) will automatically freeze all activities when free disk space becomes low. If you do not have such applications, then it makes sense to reduce this number.

Mm, so sdelete will write out zero's to unused blocks... ack.
But how is that related to applications seeing 'free disk space' as low (< 2GB for instance) ?
(I'm really into newbie space regarding blocks, etc)
Will Windows show me a free disk space of 2GB, when I leave the setting at 2GB ? (I'm probably wrong, cause then you could not write anything else to it as well?)

Thanks!

[Gostev]

The way SDELETE works is it allocates all available disk space in one large file, leaving almost no free disk space to other applications, zeroes the file's content, and deletes it, freeing up all the disk space the file used to take. So you are correct, Windows will show that your disk only has 2GB of free disk space left at the end of the SDELETE process, just before the temp file gets deleted. This limit prevents SDELETE from allocating more disk space than [available disk space] minus 2GB.

[MadDog2K]

Now I understand, thanks

[MadDog2K]

The sdelete runs for hours, this is normal ? (see post of duhaas)
(I've accepted the EULA ofcourse)

05/06/2009 10:01 x Starting zeroout of freespace of drive C:
05/06/2009 10:01 - Not finished - Free space 80371.29 MB
05/06/2009 10:02 - Not finished - Free space 73450 MB
05/06/2009 10:03 - Not finished - Free space 64181 MB
05/06/2009 10:04 - Not finished - Free space 55666 MB
05/06/2009 10:05 - Not finished - Free space 48503 MB
05/06/2009 10:06 - Not finished - Free space 41606 MB
05/06/2009 10:07 - Not finished - Free space 34362 MB
05/06/2009 10:08 - Not finished - Free space 27451 MB
05/06/2009 10:09 - Not finished - Free space 20383 MB
05/06/2009 10:10 - Not finished - Free space 13392 MB
05/06/2009 10:11 - Not finished - Free space 6711 MB
05/06/2009 10:12 - Not finished - Free space 80371 MB
05/06/2009 10:13 - Not finished - Free space 80371 MB
...
05/06/2009 16:48 - Not finished - Free space 80365 MB

[Gostev]

Sounds like the default 1 minute (60000 ms) time out for periodic free space check is too large.
SDELETE does 7GB per minute in your case, this makes the script miss the 2GB checkpoint point every time

Try reducing the value to 10000 or 1000 in this script line: WScript.Sleep 60000.
That would make disk space check happen every 10 sec or 1 sec respectively.

[MadDog2K]

Hi Gostev,

I've set it to 5000 (exactly in the middle of your suggestions :>), and it seems to run better now
It's not fully finished yet, but have good faith.

Thanks (again) !

[AJ83]

Is it possible to get the optimization tool somewhere? When following the link, it leads to a dead end.