Comprehensive data protection for all workloads
Post Reply
barberj66
Novice
Posts: 7
Liked: 1 time
Joined: Dec 07, 2015 12:06 pm
Full Name: James Barber
Contact:

Incremental backups grown from 120Gb to 1.2TB overnight

Post by barberj66 »

Hi
I have an issue which has been logged using case ID 01214300, our backups have been running without any issues for over a year incremental backups are gernerally around 120GB and Full backups around 2.1TB. For some reason over last 2 incremental backups have jumped from 120GB to 1.2TB overnight and we have not had that much data change.

We have a B&R server onsite backing upto a NAS device and this has now filled up due to these incrementals. We also had a ESXI box at the other side of the site with Veeam B&R on there replicating all VM's onto that box. Recently we have moved this replication server offsite in to a datacenter and had issues with replication due to not having a proxy set up. I then configured our on site B&R server which is just doing backups to act as the proxy for the replication and since that date the incremental backups went through the roof.

I have disabled the replication jobs not only because of this but we do not have enough bandwidth on the line to replicate effectively until we get that resolved. Log's have been passed on to Veeam but I just wanted to check if anyone has had this happen before and what the cause was?

Is our setup wrong? Veeam said we required a proxy close to the production site to aid the offsite box which I provided but i wasn't expecting it to interfere with the on site backups at all.

Any advice is more than welcome!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by PTide »

Hi,

Were any of VMs that show a dramatic growth of their incremental backups recently moved/migrated/restored from server to server?

Thank you.
barberj66
Novice
Posts: 7
Liked: 1 time
Joined: Dec 07, 2015 12:06 pm
Full Name: James Barber
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by barberj66 » 1 person likes this post

No VM's have been moved or migrated or restored at all. We haven't actually had a sucessfull replication due to the line so the replication jobs are now disabled. So I'm hoping now we have freed up some space on the repository we get a normal sized back up and it will prove to be that which caused the issue.

I can't see anywhere which shows me how much size each individual VM is taking in the incremental backups maybe I'm looking in the wrong place but under backup and replication>>Backups> then doing a properties on that I can see what size each full and incremental backup is but only get the original size of the VM itself not each backup.
skrause
Veteran
Posts: 487
Liked: 105 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by skrause »

You can't see what each individual machine takes up inside of a job because Veeam's in-job deduplication makes it impossible/impractical to show that.
Steve Krause
Veeam Certified Architect
barberj66
Novice
Posts: 7
Liked: 1 time
Joined: Dec 07, 2015 12:06 pm
Full Name: James Barber
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by barberj66 »

Cheers for that Skrause I didn't think I could find it. None of the VM's themselves have had any real increase in usage definately nothing which would increase by that much.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by PTide »

No VM's have been moved or migrated or restored at all.
Did you perform anything that might cause blocks to change? I mean if any kind of defragmentation/deduplication routine took place prior to backup?
barberj66
Novice
Posts: 7
Liked: 1 time
Joined: Dec 07, 2015 12:06 pm
Full Name: James Barber
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by barberj66 »

Nothing at all has changed apart from adding this backup server as a proxy for our offsite replication server. So it must have something to do with that but I can't see why it would make the incremental backups grow?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by foggy »

If you open job session, you can click every VM to the left and see how much data was actually read for this particular VM disks to compare with the previous "normal" job runs.
barberj66
Novice
Posts: 7
Liked: 1 time
Joined: Dec 07, 2015 12:06 pm
Full Name: James Barber
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by barberj66 »

Had a look through the session reports our main file server had 9.8GB transferred in a normal operation then the next job it failed as it "failed to delete VSS snapshot" as it was currently in use by a replication job which had overrun,then that job re-run and it transfered 1.1TB and had a warning about exchange transaction logs could not be truncated( we do not run exchange here and have never had exchange anywhere near that server) then after that the incremental backup for that 1 VM jumped to transfering 1.1TB in the next job and suceeded without errors.

Would it be something to do with the replication job that was running at the time and then the backup job has tried backing up the VM and a snapshot for some reason?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by foggy »

I'd let support engineers take a look at this. Normally, replication job should not affect backup job of the same VM if both are created on the same Veeam B&R instance. However, two different instances can make some mess in this case, since they do not now about each other's activities.
barberj66
Novice
Posts: 7
Liked: 1 time
Joined: Dec 07, 2015 12:06 pm
Full Name: James Barber
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by barberj66 »

Thanks Foggy, in normal circumstances the replication jobs run at the weekend and never interfere with our production backups, since moving our replica box offsite we have since realised our Internet line is not quick enough to replicate all our VM's in the available windows causing them both to run at the same time in this instance.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by foggy »

What kind of link do you have? Haven't you considered using WAN acceleration for replication jobs?
barberj66
Novice
Posts: 7
Liked: 1 time
Joined: Dec 07, 2015 12:06 pm
Full Name: James Barber
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by barberj66 »

It's only a 10MB link, not really looked in to the WAN acceleration as we plan to upgrade the line due to other projects that require it so hoping this will help to resolve the issue.

We went from completing all replication jobs within around 5hrs on a 1GB link internally to estimating it taking around 70hours since we moved it offsite via the 10MB link. Would WAN acceleration help enough to bring that down over a 10MB line or would I be spending money just to probably have to upgrade the link anyway going forward?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by foggy »

10Mb is a good candidate for WAN acceleration. Worth to test.
tpayton
Veeam Software
Posts: 137
Liked: 81 times
Joined: Jan 16, 2013 5:27 pm
Full Name: Tyler Payton
Contact:

Re: Incremental backups grown from 120Gb to 1.2TB overnight

Post by tpayton » 1 person likes this post

Just as an update on this.

This case has since been closed. The software detected a change in the location of the VMDK and caused the job to recapture all of the VMDK's content.

The following error was found within the logs:

Code: Select all

Warning Disk path has been changed since last transfer. Disk '<redacted>.vmdk' will be transferred as full 
Resetings CBT and running a new active full was advised. The customer has reported that the job is now behaving as expected.
Tyler Payton
Customer Experience Manager / KB Manager
Post Reply

Who is online

Users browsing this forum: bct44, Bing [Bot] and 206 guests