Hyper-V 1.3 TB vs Windows Agent Backup 0.5 TB

[mattdwyerva]

I have a single physical server with 2 VMs, all Win2016. Main VM has two disks, and using properties, I see that Drive C is 200GB partition with 27 GB used. Drive E is 1.8 TB with about 488 GB used. (D is emply DVD drive).

I have one physical server in this office, and I run VBR from that physical server (the one you see when you boot up before drilling down to VMs).

I was running a backup using Windows Agent (from VBR trial) and backup size was about 500 GB, which makes sense to me. I purchased a VBR Essentials license and tech support explained I need to backup using Hypervisor mode, so I set up a new backup that way with their assistance. Great, except.... that sucker is about 1.3 TB. Job summary reads that it "Processed 1.3 TB (100%), Read 1.3 TB and Transferred 1.2 TB (1.1x)"

I just ordered a license for the Windows Agent for server and will use that. Once I install the license file for the Agent, I will rerun the backup to verify results using the agent. Perhaps I was doing something different, although I can't even imagine what.

Is this by design? Am I missing something? I am copying this job to the cloud, so doubling or tripling the size of the backup is expensive, plus the copy immediately failed because I lacked adequate space.

[Gostev]

Normally, the backup sizes should not be much different. Something is off for sure, even just because backing up average VMs you should expect to see 50% compression ratio, but in your case it is almost non-existent, almost like it is disabled completely in the job settings.

[mattdwyerva]

All set to defaults (e.g, Compression is set to "Optimal (recommended). I can change Storage Optimization: Local Target to be "WAN target" however, and guess I should do that since ultimately that is where it is going (USB first, then copy to cloud).

edit: Also, Hyper-V tab has a default checkbox Volume Snapshots that could lead to excess storage? Should I deselect that option?

[foggy]

All set to defaults (e.g, Compression is set to "Optimal (recommended). I can change Storage Optimization: Local Target to be "WAN target" however, and guess I should do that since ultimately that is where it is going (USB first, then copy to cloud).

With WAN target backups will be smaller due to using smaller block size, however, not that dramatically (and you should compare backups done with the same block size setting anyway).

edit: Also, Hyper-V tab has a default checkbox Volume Snapshots that could lead to excess storage? Should I deselect that option?

This setting is not related to backup size.

I recommend asking support to review your job configuration.

[mattdwyerva]

Opened a support case (Case #03159063) and will report back findings. Also, hope to get my Windows agent licenses today (ordered earlier today), so should be able to re-run that backup and see what I learn.

[mattdwyerva]

Ran windows agent version again and it "Processed 515.8 GB." Checking Server properties, drive C is 27 GB and drive E is 488 GB, which is about 515.
Did a restore of a randomly selected segment of this windows agent backup and it was correct.

Tech Support (?) sent me an email saying that they will get Windows Agent team to fix the problem and sent me an FTP link for all the logs from kb2404 that was actually an insecure godaddy.com site. Was that a phishing attempt? Or do you really use insecure ftp site that is not inside your domain? I am trying to decide which is worse, as either one is scary. I uploaded logs using the support portal instead on the veeam site and requested they rethink "fixing" the windows agent backup since it is the HyperV version that appears bloated beyond recognition.

Not sure what is going on, so, meanwhile, I will use the Windows Agent backup. Small branch office, so I also use a simple TakeCommand batch COPY of recently changed files to a remote Azure disk, so I have another backup as well, just in case.

[foggy]

Tech Support (?) sent me an email saying that they will get Windows Agent team to fix the problem and sent me an FTP link for all the logs from kb2404 that was actually an insecure godaddy.com site. Was that a phishing attempt? Or do you really use insecure ftp site that is not inside your domain? I am trying to decide which is worse, as either one is scary.

It's one of the largest Internet domain registrars, not sure why it was detected by your client as insecure, must be a false positive. The actual FTP is in our domain.

[mattdwyerva]

Not sure I recall how I mis-figured the ftp site, although chrome complains a bit when non-secured sites are used now. Support portal is easy solution for a reasonably sized file, but I guess the secure ftp procedure is this: https://www.veeam.com/kb1661

Meanwhile, tech support reports they think they found the bug for unused/deleted blocks on dynamic disks, but they are still investigating. Windows agent works though, so not catastrophic, just a little more money.

[mattdwyerva]

Tech Support says:

The problem appears to be differences in how VBR and VAW see deleted data within the guest OS. What I mean by this, we know when a file is "deleted" in Windows, the data still exists on disk, but the space is marked 'free' for overwrite. Aka it becomes a 'dirty block'. VBR uses what we call 'bitlooker' to exclude these dirty blocks. I think the problem here, is a limitation of bitlooker... Veeam Backup & Replication can exclude deleted file blocks only on the VM guest OS with Microsoft NTFS (https://helpcenter.veeam.com/docs/backu ... tml?ver=95)
We have developed a method for excluding dirty blocks in VAW, which has not yet been implemented into VBR.

... the problem appears to be a limitation with bitlooker being unable to exclude deleted file blocks on the ReFS partition. Let them know bitlooker on ReFS is a feature you would like.

Yes, sounds like a feature I want. Is there another place I should submit this request? Poking around the forums, I am not sure how smart I was to select ReFS. Not entirely why I did that.

[Gostev]

ReFS is still in the active development phase, so it's too early to build advanced file system parsing functionally around it.

Also, current ReFS use cases are outside VMs to the most part (hypervisor hosts and backup repositories), so few users are actually using ReFS in guests.

[mattdwyerva]

I am always skeptical when a vendor tells me I am the only one in the universe that has this problem. Yeah, maybe, but that's amazing if so.

In any case, I gritted my teeth, moved the data, erased the disk, reformatted as NTFS, and moved the data back. Happy Sunday. Then I ran VBR for HyperV and results were "Processed 1.2TB (100%), Read: 490.2 GB, Transferred: 452.2 GB (1.1x)" Pretty sure thumbs.db files did not move over with the rest of the data, so I think those numbers are correct and that I did not lose any data, so all good now I think.

Set up a similar job with a cloud destination to run once first backup is done and got similar results. If I understand, this is a separate backup entirely independent of the first.

[Gostev]

Glad to hear you have sorted this.

And you definitely have rights to be suspicious to vendor claims! However, I've been Microsoft MVP at Cloud & Datacenter Management for the past 6 years, and really into ReFS in particular for the past 3 years - that is, working directly with the development team behind etc. So I know a bit about ReFS