Backup job - amount of data "read" - what does it mean?

[Fiskepudding]

Hello

I have two VMs, one SQL server (we just started to use) and one exchange server with a few 100 GB.

The SQL has one os disk and one 1,9TB data disk.


The Exchange server has one os disk, and two 1,9TB data diks (1 for mailboxes and one for public folders)


Both jobs runs fine. When i look at the job for the SQL server it looks like the number of read GB's is close the currently used space on the SQL server:

This makes sense

Not even close to all data on both 1,9TB disks on the exhange is used, and still it reads 1,9TB on those disks?...
Disk usage on exchange:


On the 1,9TB exchange disks it seems like Veeam reads all data, but not on the smaller OS disk. (and not on the 1,9TB datadisk on the SQL server)

All disks are Thin provisioned and formated with NTFS.. dont know if that would matter anyway.

Can anyone explain this?

[homerjnick]

Does that suggest that at one time all the space on the Exchange 1.9TB disk was used?

[Fiskepudding]

I dont know what it suggests, but there have never been more data on these volumes then there is now.
And I think I would have noticed if my exchange disk would hav ebeen completly full

Even if that had been the case, should that even matter? Why read the entire voulume when only 1/4 is used, at the moment?

I would also like to add that the SQL and exchange job is set up the same way.

[Fiskepudding]

I thought that cold be the case myself, so i renamed the SQL backup folder on the repository and deleted it from V B&R. So V B&R has no records of the backup even existing.
I run the backup job once more and the reults on the SQL server (full backup) were the same as before (screenshot).(It did not read the enitere 1,9TB volume)
So even a brand new backup of the SQL vm never reads the entire 1,9 TB volume.

To make some sense of the combination of what your saying and my observations:
Even if i Delete the exchange backup files or remove them from veeam.
Then run the backup once more, the results should be different, (it should READ the amobut of data that actually is used)?
The records on what blocs are beeing used resides on the datastore on the ESX host? (CBT?)

[foggy]

Actually I've deleted my post when I realized that this was not the reason. The real reason is in CBT, right. During the first full job run Veeam B&R queries VMware CBT for the blocks that should be processed - those occupied by data (not zeroed ones). And sometimes this CBT query does not work and Veeam B&R has to read the whole disks to verify those blocks that need to be processed. This is covered here also in some more detail. Sorry for confusion.

[Fiskepudding]

aha, and that can be different from disk to disk, sinde CBT is located on each disk individually?
Because the C drive is red "correct"/only used data on the exchange VM.

Right now I am running a new backup of the exchange (after having removed traces of the previous backup from Veeam B&R).. Thanx for the link.

[foggy]

aha, and that can be different from disk to disk, sinde CBT is located on each disk individually?

Yes, exactly.

[Fiskepudding]

The problem is the same, it still reads the entire disk.
So much for makeing the disk ready for the future

It seems like veeam reads the blocks with no data a bit faster. But since there is much unsued space the backup takes a whole lot longer then necesarray.

My plan(http://forums.veeam.com/viewtopic.php?f=2&t=11120) was to run 1 job of Reversed incrementals weekdays and
Active full every saturday to antoher repository.
This could cause a problem, since the full job now actually will take longer then I hoped for.

Can this be some special behaviour on volumes containing exchange data?....

The voulmes are thin provisioned as mentioned, and this is their sizes, if it helps at all:
C:, F:, E: vmdk files (in that order from top)


They are not "inflated".

[foggy]

Have you tried disabling/enabling CTK for this VM? The step-by-step procedure can be found here.

[Fiskepudding]

No I have not.

I actually red that post before, but did not find it directly related to my problem, since i have no problem with

causing "Cannot use CBT" warning in the Veeam job log

Since this is an exchange server i cant take it down whenever I like, I wil see if I can do it tonight tho.

[dellock6]

try also to run sdelete on the NTFS partition to effectively remove once-used space, since CBT tracks this blocks as used but maybe right now they are no more. Defragmentation of the NTFS has a huge role on this (but please do not run defrag since it will move all blocks around, and in the next run CBT will mark many blocks has modified, extendy further the duration of the job).

[Fiskepudding]

Thanx for the tip
Just wonder, should this not made the VMDK files larger then they are?
They are now the same size as the used space?

I have never run sddelte, but seems quite straigt forward.

Code: Select all

sdelete -c e:

and

Code: Select all

sddlete -c f:

in my case?

Is there any risks/issues with this / wil it impact an production exchange server?

[dellock6]

It will for sure increase the I/O load on the storage while sdelete runs, but apart from that, never seen any issue using it.
No, the vmdk in not going to grow. It will only clean unused ntfs blocks.

[Fiskepudding]

Waht I ment with the VMDK size was that:
IF it was "filled" with unused blocks, should it not been bigger in the rist place.

My consern was not that it would grow AFTER sdelete.

[Gostev]

sdelete works by creating one large temporarily file and filling it up with zeroes. This means that at the end of the process (before this temporarily file is delete), there will be no free disk space available on the volume. So, you may want to stop sdelete when remaining free disk space is 100 MB or something. You can create a short script that would start the process, query disk free space and stop the process when needed.

[EDIT] Actually, another user had already created such script earlier > Defragmentation

[Fiskepudding]

This migt be a bit off topic, but, i downloded the lates versjon (1.6) of sdelete but it wont accept my syntax:

I tried this on a testserver:

Code: Select all

C:\tools>sdelete.exe /?

SDelete - Secure Delete v1.6
Copyright (C) 1999-2010 Mark Russinovich
Sysinternals - www.sysinternals.com

usage: sdelete.exe [-p passes] [-s] [-q] <file or directory> ...
       sdelete.exe [-p passes] [-z|-c] [drive letter] ...
   -a         Remove Read-Only attribute
   -c         Clean free space
   -p passes  Specifies number of overwrite passes (default is 1)
   -q         Don't print errors (Quiet)
   -s or -r   Recurse subdirectories
   -z         Zero free space (good for virtual disk optimization)


C:\tools>sdelete -c e:

SDelete - Secure Delete v1.6
Copyright (C) 1999-2010 Mark Russinovich
Sysinternals - www.sysinternals.com


Invalid drive letter: e
Argument must be formatted like d:

Since that id not work i tried :

Code: Select all

C:\tools>sdelete.exe -c e

SDelete - Secure Delete v1.6
Copyright (C) 1999-2010 Mark Russinovich
Sysinternals - www.sysinternals.com

SDelete is set for 1 pass.
Cleaning free space on C:\: 3%

Verry cute?...it runs the tool my c drive...

Is this software thrustworthy?

It seems that it runs on the drive where the tool is located.

Code: Select all

E:\>sdelete.exe -c
SDelete - Secure Delete v1.6
Copyright (C) 1999-2010 Mark Russinovich
Sysinternals - www.sysinternals.com

SDelete is set for 1 pass.
Cleaning free space on E:\: 8%

So i guess, if it does not crap out my test server I wil try it on my exchagne.

[dellock6]

I think you have to pass the drive letter as e: or e:\, do not have a sdelete at hand to check.

[Fiskepudding]

I just copied sdelete to the drive I wanted sdelete to work on, and it ran.
Anyway.. This wil mess up my thin provision disks, on my testserver, after runnig sdelete on the the E disk, it filled up in windows and emptied it self afterwards.
Now the vmdk size = provisioned size on my datastore.

Runing a vmwware standalone converter on the VM afterwards should schrink the disks according some forums, but I have not tried.

I did try this, witch was suggested on another forum:

Code: Select all

vmkfstools -i source_VMDK -d thin target_VMDK

but the disk clone/copy is the same size on the datastore as teh source.

Not too eager to try a converter on my exchange server with vmware standalone converter.

I have not had the oportunity to "Fix broken CBT" on the exchange server data disks yet, but wil try tat hopefully tonight.
I dont want to run sdelete on my exchange just yet. Because of our thin provision usage.

[Gostev]

Does not make sense to use with thin provisioned disks for sure. Was meant for flat disks.

[dellock6]

Sorry, did not know they were thin disks, obviously it's aimed to thick disks.
A trick you can use could be, after a sdelete, to run storage vmotion on the VM and force to thin again, it could eventually work. Worth a try? never done myself but I would try.

[Fiskepudding]

To make it clear i did NOT run sdelete on my production server, just my testserver.

I did however try "fix CTB" provided in a link here earlier.
Every FULL backup of the exchange still reads the entire provisioned size of the disks (1,9TB x2 disks)
(C drive on 250 GB reads perfectly about 14 GB on the same VM)

As far as I can tell this only happens on this one VM. All other VMs reads only the "actuall used space".

Considdering this is a thin provision disk, should it not even be easer for veeam to know witch blocs have been used?

[foggy]

If you are talking about disabling CBT, then it will not change anything.

[Fiskepudding]

I am talking about how many bits, or TB for that matter Veeam reads.

[foggy]

It has to read the full image (provisioned space), which is exactly what you see in your case.

[Fiskepudding]

so you would think sdelete would fix this (and introduce me to another problem, filling up my tin disk to its provisioned size)?

[foggy]

Regardless of what you do inside your VM (sdelete or whatever), with CBT disabled (or not working) Veeam B&R would have to read the whole image. Running sdelete will only speed up the processing a bit (unused blocks will be zeroed, and this will not have to be compressed, transferred and saved to the backup repository).

[Fiskepudding]

ok.. BUT CBT IS working and Veeam still reads the entire disk.
The images on my first post shows :
DISK1 .... [CBT]
DISK2 .... [CBT]
DISK3 .... [CBT]

No errors.
On disk 1 it reads only the used space, on disk 2 and 3 it READS the entire disks.
I dont understand why. I am verry close to just giving up,and let read whatever it wants...

[dellock6]

Wait, got a doubt, have you installed Patch 3? Because it sounds really similar to this thread:

http://forums.veeam.com/viewtopic.php?f=24&t=9755

and this was solved with the latest patch.

[Fiskepudding]

sorry to say, latest patch was installed before we even started using veeam for backups. Version is 6.0.0.181

[foggy]

On disk 1 it reads only the used space, on disk 2 and 3 it READS the entire disks.
I dont understand why.

Because CBT does not work for disk 2 and disk 3.