Comprehensive data protection for all workloads
Post Reply
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Last night's Incremental looks like a Full

Post by jadams159 »

I was looking over last nights backup statistics and saw that my Exchange server's backup job(supposed to be incremental) took over 8 hours to complete(normally30-40 min). When I looked at the report, last night's job has identical statistics to a Full backup, although the report does not have the indicator "(Full)" on the report.

Now, I started a clone process on my Exchange server yesterday, but cancelled it before it could complete. I'm not sure if that had an effect on it, but if it did, should cloning procedures influence Veeam backup jobs?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Last night's Incremental looks like a Full

Post by veremin »

Hi, Justin.

What about the size of resulting file? Was it comparable with the size of typical full backup? Also, was the CBT used or not (you can check it in the backup session statistics by searching for “CBT” metrics)?

Thanks.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

CBT is enabled.

My previous Fulls have the following:
Data Size = 1.8TB
Backup Size = 1.6TB

Last night's incremental:
Data size = 1.6TB
Backup size = 1.6TB
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

Not sure if this is helpful, but I thought I'd add that looking at the statistics on my Data Domain, the ingestion and compression/dedupe looks just like a Full backup, there isn't a higher amount of "new" or "unrecognized" data as far as data domain is concerned.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Last night's Incremental looks like a Full

Post by Vitaliy S. »

Justin, what about actual usage of the CBT? You say it is enabled, but do you see a [CBT] tag next to processed disk in the backup log session?
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

Tell me if I'm looking in the wrong spot, but I'm looking at the "Statistics" of the job, highlight the VM's name, and I see "[CBT]" next to each disk on the VM along with the rate.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Last night's Incremental looks like a Full

Post by Gostev »

jadams159 wrote:CBT is enabled.

My previous Fulls have the following:
Data Size = 1.8TB
Backup Size = 1.6TB

Last night's incremental:
Data size = 1.6TB
Backup size = 1.6TB
Are you using reversed incremental backup mode?
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

No, I'm using forward incrementals.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Last night's Incremental looks like a Full

Post by Gostev »

What are the "transferred" amounts? Do you possibly have compression disabled in the job settings, or backup repository option to uncompress data before saving enabled?
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

Transferred amount = 1.6TB. Compression level is "None", and "Decompress Data Blocks before storing" IS checked for the repository.

Both of these options are found in Data Domain's best practices for Veeam settings, and in my testing yielded the best balance for dedupe and performance.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Last night's Incremental looks like a Full

Post by dellock6 »

Any chance that was actually a full backup? Maybe the job is configured to have active full in that day, or it was manually started?

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

The report does not say "(Full)" as all Full backups have. It started as scheduled, not manually. The Full(non-synthetic) is set for Sunday.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Last night's Incremental looks like a Full

Post by Gostev »

I am curious then what files do you see in the repository folder. I agree with Luca, it does sounds like a full backup.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

Last nights backup file carries the .vib extension.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Last night's Incremental looks like a Full

Post by foggy »

How about Veeam inline deduplication? Is it enabled in the job settings?
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

Inline dedupe is not enabled.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Last night's Incremental looks like a Full

Post by foggy »

In this case, Veeam B&R does not use its proprietary change tracking algorithm and takes what CBT returns for granted. If CBT was reset for some reason (for example, after svMotion), this results in the entire VM data transfer.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 » 1 person likes this post

I guess I'll see what happens over the weekend. The good news is that it really isn't costing me any storage.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Last night's Incremental looks like a Full

Post by foggy »

If this is the case, the next increment should have its usual size. Anyway, you can always ask technical support to research what's happening.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

That's what I'm hoping, perhaps just a fluke caused by the cancelled clone operation. Should it occur again, I will open a case and post the case number in this thread. Thanks to all for the insight.

As always, Veeam rules.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

OK, it appears that if I clone my servers, this scenario occurs on the next backup. It is not as bad(noticeable) on smaller servers though. If I clone a VM, the amount of data transferred on the following backup increase dramatically on an incremental backup.

Is this expected behavior?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Last night's Incremental looks like a Full

Post by Gostev »

No, I would not expect that from the cloning process.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

When I get some time I will test to see if this happens with my Veeam server that has version 6.
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Last night's Incremental looks like a Full

Post by tsightler »

Yes, and I can't reproduce this in my environment. I cloned several VMs and it made no difference in the amount of data on the next backup. It certainly sounds like that for some reason CBT is being impacted by the cloning process. Are you just using the clone feature in vCenter?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Last night's Incremental looks like a Full

Post by Gostev »

Even if CBT was impacted, our source-side dedupe would detect that the actual content of most blocks is unchanged, and transfer amount would be minimal. Large transfer amount can only happen if every single block had its content changed (not just overwritten with the same data).

This sounds almost like a virus activity to me.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

Gostev - Inline Dedupe is not enabled, so would Veeam source-side dedupe still be a factor?

tsightler - Yes I'm cloning with vCenter, although with powerCli scripts.
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Last night's Incremental looks like a Full

Post by tsightler »

@Gostev - It was mentioned earlier in the thread that he had inline dedupe disabled, which will cause Veeam to read/send every block if CBT fails or is reset by some event. This is one of the reasons why I say it's best practice to leave Veeam dedupe enabled even when using a dedupe target. Veeam source side dedupe still has plenty of benefit during things like CBT failures or resets, and as you well know these can be triggered by all types of things, including some that are just part of regular operations. There is really almost no reason to ever disable Veeam dedupe.

Of course, I have no idea why a simple clone operation causes this, unless it's some strange bug with certain builds of ESXi, but I certainly can't reproduce it. Perhaps it is a virus of some sort, but it sounds like your seeing this consistently with any clones.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

OK, so I finally got around to looking at this in detail. I picked a significantly smaller server (time constraints and I didn't want to be running multiple backups on my exchange server all day). As I mentioned, the impact is not as significant on a small server, but none the less the pattern is visible. Below is a log from Veeam EM with some events injected in there to show when things took place. Enabling inline dedupe definitely resolved some of the symptoms, but there is still a jump in data transfer and especially Performance Rate(assuming Veeam is having to look at more "changed" data) after performing a clone operation.

I thought that perhaps my PowerCLI script had something to do with it since you could not duplicate it in your lab, but as you can see, if I use my vSphere client to manually clone the VM, the pattern continues.

Image
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: Last night's Incremental looks like a Full

Post by jadams159 »

@Gostev and @tsightler

According to this KB: http://www.veeam.com/kb1745 Veeam suggests disabling inline Dedupe. Tom's comment about leaving the inline dedupe enabled makes sense. I also think it helps skipping 0 bits on the vmdks. I have left Inline Dedupe enabled on many Data Domains and have continued to see good dedupe numbers.

Any official Veeam response to Tom's suggestion?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Last night's Incremental looks like a Full

Post by Gostev »

KB1745 is the official "one size fits all" Veeam recommendation when using deduplicating storage for backup repository. You can keep inline dedupe enabled if it does not negatively impact YOUR deduplicating storage make and model, but we cannot make such a blanket recommendation to everyone based on our observations in support. Thanks!
Post Reply

Who is online

Users browsing this forum: Google [Bot], legil.miguel and 187 guests