Last night's Incremental looks like a Full

[jadams159]

I was looking over last nights backup statistics and saw that my Exchange server's backup job(supposed to be incremental) took over 8 hours to complete(normally30-40 min). When I looked at the report, last night's job has identical statistics to a Full backup, although the report does not have the indicator "(Full)" on the report.

Now, I started a clone process on my Exchange server yesterday, but cancelled it before it could complete. I'm not sure if that had an effect on it, but if it did, should cloning procedures influence Veeam backup jobs?

[veremin]

Hi, Justin.

What about the size of resulting file? Was it comparable with the size of typical full backup? Also, was the CBT used or not (you can check it in the backup session statistics by searching for “CBT” metrics)?

Thanks.

[jadams159]

CBT is enabled.

My previous Fulls have the following:
Data Size = 1.8TB
Backup Size = 1.6TB

Last night's incremental:
Data size = 1.6TB
Backup size = 1.6TB

[jadams159]

Not sure if this is helpful, but I thought I'd add that looking at the statistics on my Data Domain, the ingestion and compression/dedupe looks just like a Full backup, there isn't a higher amount of "new" or "unrecognized" data as far as data domain is concerned.

[Vitaliy S.]

Justin, what about actual usage of the CBT? You say it is enabled, but do you see a [CBT] tag next to processed disk in the backup log session?

[jadams159]

Tell me if I'm looking in the wrong spot, but I'm looking at the "Statistics" of the job, highlight the VM's name, and I see "[CBT]" next to each disk on the VM along with the rate.

[Gostev]

CBT is enabled.

My previous Fulls have the following:
Data Size = 1.8TB
Backup Size = 1.6TB

Last night's incremental:
Data size = 1.6TB
Backup size = 1.6TB

Are you using reversed incremental backup mode?

[jadams159]

No, I'm using forward incrementals.

[Gostev]

What are the "transferred" amounts? Do you possibly have compression disabled in the job settings, or backup repository option to uncompress data before saving enabled?

[jadams159]

Transferred amount = 1.6TB. Compression level is "None", and "Decompress Data Blocks before storing" IS checked for the repository.

Both of these options are found in Data Domain's best practices for Veeam settings, and in my testing yielded the best balance for dedupe and performance.

[dellock6]

Any chance that was actually a full backup? Maybe the job is configured to have active full in that day, or it was manually started?

Luca.

[jadams159]

The report does not say "(Full)" as all Full backups have. It started as scheduled, not manually. The Full(non-synthetic) is set for Sunday.

[Gostev]

I am curious then what files do you see in the repository folder. I agree with Luca, it does sounds like a full backup.

[jadams159]

Last nights backup file carries the .vib extension.

[foggy]

How about Veeam inline deduplication? Is it enabled in the job settings?

[jadams159]

Inline dedupe is not enabled.

[foggy]

In this case, Veeam B&R does not use its proprietary change tracking algorithm and takes what CBT returns for granted. If CBT was reset for some reason (for example, after svMotion), this results in the entire VM data transfer.

[jadams159]

I guess I'll see what happens over the weekend. The good news is that it really isn't costing me any storage.

[foggy]

If this is the case, the next increment should have its usual size. Anyway, you can always ask technical support to research what's happening.

[jadams159]

That's what I'm hoping, perhaps just a fluke caused by the cancelled clone operation. Should it occur again, I will open a case and post the case number in this thread. Thanks to all for the insight.

As always, Veeam rules.

[jadams159]

OK, it appears that if I clone my servers, this scenario occurs on the next backup. It is not as bad(noticeable) on smaller servers though. If I clone a VM, the amount of data transferred on the following backup increase dramatically on an incremental backup.

Is this expected behavior?

[Gostev]

No, I would not expect that from the cloning process.

[jadams159]

When I get some time I will test to see if this happens with my Veeam server that has version 6.

[tsightler]

Yes, and I can't reproduce this in my environment. I cloned several VMs and it made no difference in the amount of data on the next backup. It certainly sounds like that for some reason CBT is being impacted by the cloning process. Are you just using the clone feature in vCenter?

[Gostev]

Even if CBT was impacted, our source-side dedupe would detect that the actual content of most blocks is unchanged, and transfer amount would be minimal. Large transfer amount can only happen if every single block had its content changed (not just overwritten with the same data).

This sounds almost like a virus activity to me.

[jadams159]

Gostev - Inline Dedupe is not enabled, so would Veeam source-side dedupe still be a factor?

tsightler - Yes I'm cloning with vCenter, although with powerCli scripts.

[tsightler]

@Gostev - It was mentioned earlier in the thread that he had inline dedupe disabled, which will cause Veeam to read/send every block if CBT fails or is reset by some event. This is one of the reasons why I say it's best practice to leave Veeam dedupe enabled even when using a dedupe target. Veeam source side dedupe still has plenty of benefit during things like CBT failures or resets, and as you well know these can be triggered by all types of things, including some that are just part of regular operations. There is really almost no reason to ever disable Veeam dedupe.

Of course, I have no idea why a simple clone operation causes this, unless it's some strange bug with certain builds of ESXi, but I certainly can't reproduce it. Perhaps it is a virus of some sort, but it sounds like your seeing this consistently with any clones.

[jadams159]

OK, so I finally got around to looking at this in detail. I picked a significantly smaller server (time constraints and I didn't want to be running multiple backups on my exchange server all day). As I mentioned, the impact is not as significant on a small server, but none the less the pattern is visible. Below is a log from Veeam EM with some events injected in there to show when things took place. Enabling inline dedupe definitely resolved some of the symptoms, but there is still a jump in data transfer and especially Performance Rate(assuming Veeam is having to look at more "changed" data) after performing a clone operation.

I thought that perhaps my PowerCLI script had something to do with it since you could not duplicate it in your lab, but as you can see, if I use my vSphere client to manually clone the VM, the pattern continues.

[jadams159]

@Gostev and @tsightler

According to this KB: http://www.veeam.com/kb1745 Veeam suggests disabling inline Dedupe. Tom's comment about leaving the inline dedupe enabled makes sense. I also think it helps skipping 0 bits on the vmdks. I have left Inline Dedupe enabled on many Data Domains and have continued to see good dedupe numbers.

Any official Veeam response to Tom's suggestion?

[Gostev]

KB1745 is the official "one size fits all" Veeam recommendation when using deduplicating storage for backup repository. You can keep inline dedupe enabled if it does not negatively impact YOUR deduplicating storage make and model, but we cannot make such a blanket recommendation to everyone based on our observations in support. Thanks!