5-6GB incrementals, zero change on the volume.

[m@x]

Hello,

Can anyone chime in and explain why my incrementals are so huge, despite the fact there have been zero changes made on the backup source volume?

Source: data volume running on the top of storage spaces and ReFS.
Backup type: volume-level.
Volume size: 2TB (shows 170GB used).
Actual data size: 150GB.

Results below:

Initial Backup (full): 160GB
1st incremental (launched immediately after full): 5.56GB
2nd incremental (launched immediately after 1st inc): 6.36GB
3rd incremental (scheduled the same night): 6.05GB
4th incremental (scheduled the next night): 5.13GB

Could it be the ReFS? Anyone having similar results in NTFS?

[HannesK]

Hello,
just to make sure: it's a plain data disk where no process or human being is doing any changes?

There is no automatic Windows defragmentation or any other REFS optimization process running? Are optimization / re-balancing processes for storage spaces running? I know that StoragePool optimization is possible.

Maybe some storage spaces experts can add ideas if you give more details about the setup (mirror / parity / stripe and the amount of disks).

Best regards,
Hannes

[Andreas Neufert]

That is roughly a 3% change rate and is pretty usual on any application server.
What does the Server do ?
Remember there are many processes running in the background of Windows that optimize data, write to pagefiles, work with profiles when windows users are logged in, Antivirus (that caches things like compressed files).
What services and Software do you run there?

[m@x]

This is a file server. The data volume has my personal data that is accessed only by me over the share. I've just rebuilt it, no applications installed. Plain vanilla Server 2019. No one is using it. Even the share isn't created yet. I've installed Windows, built storage spaces pool across 4 drives (mirror), created a vdisk and a volume. Formatted as ReFS, dumped all my 150 GB on it (plain data files and pictures), installed Veeam and started evaluating the solution. Ran full backup, then an incremental immediately (2 seconds apart). 5.5GB of change - I don't believe it. Yes, storage spaces could be running some magic under the hood, but this is completely abstracted from the vdisk/volume level. Things don't move around on the virtual disk level (not to be confused with VHDX, it has a different meaning in the context of storage spaces, gotta love Microsoft naming). IMHO.

[Andreas Neufert]

In the end we just monitor the changes made to the disk and use those for backup. So the changes really happened. We have basically a transparent filter between the OS filesystem and the storage stack and just collect change block information. There is not really something we can do here.

Specifically Storage Spaces will optimize in the background data placements. Those can take multiple days, specifically if you have created a new disk. You see declined change rate above which is maybe an indication for this. Maybe enable Disk Monitoring in the Windows Performance Monitor and see what services you can identify that write to the disk.

[m@x]

Offtopic: it looks like I don't have an option to reply to your latest post and the ["] quote button is missing. Not sure if that's by design. I had to construct the quote manually.

In the end we just monitor the changes made to the disk and use those for backup. So the changes really happened. We have basically a transparent filter between the OS filesystem and the storage stack and just collect change block information. There is not really something we can do here.

Specifically Storage Spaces will optimize in the background data placements. Those can take multiple days, specifically if you have created a new disk. You see declined change rate above which is maybe an indication for this. Maybe enable Disk Monitoring in the Windows Performance Monitor and see what services you can identify that write to the disk.

I might need to test the backup of the same data set placed on a regular physical disk and compare the results. You mentioned about the filter between the filesystem and storage stack. I am curious what is the "storage stack" from Veeam Agent's perspective. Storage Spaces technology consolidates physical drives into a pool, then you create a Virtual Disk (also referred as Storage Space, and presented to OS as a regular physical disk), and then you create volumes out on vdisk. Do you know if the change block information is collected from the Virtual Disk, or the underlying layer?

[Andreas Neufert]

HI Max, thanks for letting us know about the quote thing. We will check.

FileSystem Filter drivers are similar to device drivers for disks. I do not know if the optimzation is done on what level and if the filter driver would see it or not. So I might be wrong. Nevertheless we only backup data that was reported as changed from the IO stack of windows.

[m@x]

HI Max, thanks for letting us know about the quote thing. We will check.

Offtopic continued: It looks like the quotes are missing only for the most recent reply. As soon as another comment is placed and your comment now becomes #2, the quotes are showing up.

[Andreas Neufert]

Regarding the Offtopic one. This was disabled on purpose to prevent people from quoting the immediate post in its entirety.

[Gostev]

In the end we just monitor the changes made to the disk and use those for backup. So the changes really happened. We have basically a transparent filter between the OS filesystem and the storage stack and just collect change block information. There is not really something we can do here.

Moreover, if the write I/O happened, but the content written was the same as the existing content (which is quite common), then this block will not be included into the incremental backup.

[m@x]

Regarding the Offtopic one. This was disabled on purpose to prevent people from quoting the immediate post in its entirety.

Interesting approach. It doesn't really work when there are just 2 participants.

[Andreas Neufert]

The point was that the quote system was abused: many people tend to use Quote button as their default Reply button, and so they spammed the forum with useless repeats of entire previous posts, forcing everyone to read through the same content twice.

[m@x]

Moreover, if the write I/O happened, but the content written was the same as the existing content (which is quite common), then this block will not be included into the incremental backup.

I created a Test virtual disk (not to be confused with VHD/X, Microsoft likes to confuse people, it's basically a storage space) and created a Test volume on it (T:\).

Formatted with ReFS (4096K)
Copied over a collection of my iso files (11.9GB)
Full backup: 13,314,412 KB
Incremental1: 905,532 KB
Incremental2: 943,488 KB

Formatted the same volume with NTFS (4096K)
Copied over the same collection of my iso files (11.9GB)
Full backup: 12,077,472 KB
Incremental1: 19,128 KB (!!!)
Incremental2: 18,876 KB (!!!)

It's a 50x difference.

I know that ReFS in conjunction with Storage Spaces unlocks a set of proprietary features for additional redundancy and resiliency. I am curious to see the results on a ReFS volume without Storage Spaces underneath.

[m@x]

Tested volume-level backup of ReFS volume on a single drive (no Storage Spaces).

Formatted the volume with ReFS (4096K)
Copied over the same collection of my iso files (11.9GB)
Full backup: 12,077,020 KB
Incremental1: 18,788 KB
Incremental2: 18,788 KB

Looks like huge incrementals are a result of ReFS + Storage Spaces only.
NTFS + Storage Spaces as well as ReFS without Storage Spaces result in 50x smaller incrementals.

Any thoughts?

[Andreas Neufert]

Can you try again with an Empty Storage Space volume and ReFS. Again my guess is that you have transportet data to the volume and windows will optimize then things in the background which is maybe above the filter driver level that we use.

[m@x]

Can you try again with an Empty Storage Space volume and ReFS. Again my guess is that you have transportet data to the volume and windows will optimize then things in the background which is maybe above the filter driver level that we use.

I will try that.

One thing worth mentioning is that during the testing phase I am running the incrementals back to back. As soon as the [Backup Now] button becomes available. It takes 10-11 seconds to complete Initialization, Backup Preparation and VSS Snapshot creation. Meaning that the 9-10GB change must happen during these first 10 seconds. That's a change rate of 900-1000MB/s on the volume. Not only do I see zero activity on the source drive (perfmon, diskperf), my 4 drive mirrored pool (SATA3 7200rpm) is not even capable of such cosmic speeds.

[m@x]

Can you try again with an Empty Storage Space volume and ReFS. Again my guess is that you have transportet data to the volume and windows will optimize then things in the background which is maybe above the filter driver level that we use.

Test volume - empty, ReFS (4096K) on Storage Spaces.
Results:
Full: 1,211,876 KB
Inc1: 896,280 KB
Inc2: 1,080,832 KB

All backups triggered manually back to back.
1GB change in 10 seconds is 100MB/sec of continuous change. Disk monitoring tools show 0KB/s reads and writes.

[Andreas Neufert]

Thanks we will discuss internally.

[kokishin]

Any update on this issue?

I'm running Win 10 Pro on a desktop with the OS on a 512GB SSD (C: drive) and a 6TB HDD and a 4TB HDD for the data drives, all formatted with NTFS. I run incremental backups very night. Even when there is not much activity on the desktop, I get incremental backups of at least 17GB and typically greater.

I have a laptop running Win 10 Home with a single 512GB SSD formatted NTFS. The laptop can be in hibernate mode for days. Then when the incremental backup occurs after exiting hibernate mode, the incremental backup is at least 1.5GB to 3GB.

Fortunately, I have a shared folder on a 14TB HDD on another machine dedicated for the VEEAM backups.

I ran MS WHS for years, which also uses VSS, and never had these large incremental backups.

In the VEEAM control panel -> Edit Job -> Shared Folder -> Advanced -> Storage Tab, I have Compression Level set to Optimal and Storage Optimization set to LAN Target. Most backups achieve ~1.7x Compression but Dedupe is always 1.0x. Not sure what else I can do to make backups more efficient in size.

Is there someway to tell if VSS is only presenting essential modified blocks for incremental backup?

Is there known task or activity which is creating needless modified blocks? For example, does Windows Defender or other antivirus software create modified blocks when it's sniffing files?

Has VEEAM studied the issue?

[Dima P.]

Hello Koki,

Even when there is not much activity on the desktop, I get incremental backups of at least 17GB and typically greater.

Can you please clarify the backup mode you are using (entire computer/volume level or file level backup mode)? Thanks!

[kokishin]

Hi Dima,

Here is the summary from the VEEAM control panel:

General ------------------------------------------
Backup job name: C, E, F Drive BU 01/01/2020
Backup job description:
Created by DT\User at 12/3/2019 4:01 PM.


Source -------------------------------------------
Backup mode: entire computer
The entire computer will be backed up


Destination --------------------------------------
Shared folder: \\VEEAMSERVER\Veeam Backup
Keep restore points for the last 181 days
Compression level: Optimal (recommended)
Storage optimization: LAN target
Storage encryption: disabled


Schedule -----------------------------------------
Schedule: workstation
Daily at [1:00 AM]
If computer is shutdown at this time: [backup once powered on]
Once backup is taken, computer should: [keep running]
At storage attach
Backup no more often than every [24 Hours]


Command line to start the job:
"C:\Program Files\Veeam\Endpoint Backup\Veeam.EndPoint.Manager.exe" "backup" "d2e3fa6d-8be9-4b23-98d7-70a807d6ca20"

[kokishin]

Dima,

Here is info on my last four backups:

[Dima P.]

Koki,

Set up looks good, thanks for the update! Veeam Agent processes only the changed blocks from the volumes in case of entire computer/volume level backup. Such change rate means that some software or process modifies data on disk: that could be periodic defrag operation, software updates, new product installations or data download. AV software usually does not affect the change rate in it's day-to-day activities (it does not modify the data unless it's infected).