Comprehensive data protection for all workloads
Post Reply
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Backup Copy Job process

Post by nmdange »

I've noticed that while our backup copy jobs are running, particularly when the copy job is merging the oldest incremental into the full, the Veeam B&R server is very slow. Using the console or even just doing anything within Windows on the server takes forever.

Looking at Resource Monitor, I see that there is a lot of I/O happening to files with names like this:
C:\ProgramData\Veeam\Backup\<copy job name>\<date>_<copy job name>.zip
and in the same folder a "DotNetZip" tmp file.

I think because all this I/O is happening on the system drive of the Backup & Recovery server, it is causing the entire operating system to respond very slow

Is there any way to change where this activity happens? I would think that it would make more sense for the copy to go straight from the source backup repository to the destination backup repository. At least, in cases like mine where both repositories are Windows servers. This seems to only be an issue with Copy jobs and not the actual Backup jobs.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Backup Copy Job process

Post by Shestakov »

Hello,
It`s always better to start troubleshooting from the bottleneck analysis. Could you provide the stats?
Where do you see high IOPs, on the source or target site?
Thanks!
veremin
Product Manager
Posts: 20415
Liked: 2302 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Copy Job process

Post by veremin »

C:\ProgramData\Veeam\Backup\<copy job name>\<date>_<copy job name>.zip
This archive contains Target datamover and Backup Sync logs, so, it shouldn't be the root cause of degraded performance. As Nikita's suggested, it's better to start with bottleneck statistics review. Thanks.
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy Job process

Post by foggy »

I also wonder what are the source and target repositories for this copy job and how they are added into Veeam B&R?
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Backup Copy Job process

Post by nmdange »

The bottleneck is almost always listed as "Source" for all of our backup jobs and copy jobs. Very rarely, the jobs will show "Throttling" as the bottleneck. I have a throttling rule set to 8000 Mbps to prevent the backups from saturating our 10gb links. I have never seen the bottleneck show "Target". In terms of percentages, the Source is usually 70%+, Proxy is 10% or less, and network can vary. Target always shows 0%.

Both the source and target repositories are of type "Windows" in Veeam. They are physical servers running Windows Server 2012 R2 with Hyper-V enabled, with a bunch of RAID 50 disks with 4TB 7.2k SATA drives. They also act as an Off-Host Proxy for our Hyper-V backups. The performance of the actual backups themselves seems to be ok.

The High IOPs I see are not on either the source or target repository server, but the C drive of the Veeam B&R server itself. Looking in Performance Monitor while the Copy Jobs are running, I see 1000+ write IO/sec on the disk, and a queue length with 15+ outsanding IOs. The Veeam B&R server is a VM so I moved the C drive to a different physical disk array and it seems to have helped somewhat. It just seems like the Copy Jobs are generating a huge amount of disk activity on the C drive of the Veeam B&R server when the activity should really be Source Repository -> Target Repository.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Backup Copy Job process

Post by Shestakov »

Queue of 15+ IO requests is a big one. What is the delay?

There can be several reasons of high IO rate on the source. I would start with next steps:
1) Check if you have any Undersized VMs by memory, which can use disk space to extend physical memory size.
2) Disable all non-related to virtualization roles on the HV host.

What version of Windows Server are you at? Thanks!
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Backup Copy Job process

Post by Shestakov »

In addition to those general solutions of high IOPs, if the reason is
nmdange wrote:C:\ProgramData\Veeam\Backup\<copy job name>\<date>_<copy job name>.zip
and in the same folder a "DotNetZip" tmp file.
You might need to set a proper retention for your log files. Since looks like during backup copy job merge lots of new logs are generated which causes the logs compression.
Please review the KB to adjust related settings. Thanks!
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Backup Copy Job process

Post by nmdange » 1 person likes this post

Everything is 2012 R2. I am looking at the IOPs and Disk Queue inside the Veeam B&R VM, not at the host. Looking at Resource Monitor at the disk detail is how I determined those files as being the cause.

So I'm looking at the log files and the size of the .zip files for the backup copy jobs that are using most of the IOPs are 6gb each. In addition, there are over 5,000 10MB log files created per day.

It looks like a new zip file is created every 2 weeks? Looking at that KB article, it doesn't look like there's a way to limit the size of the compressed logs or how often they are created? Any recommendations on how to change the settings? Right now everything is the default.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Backup Copy Job process

Post by Shestakov »

50 GBs of logs a day is a huge number. How big is your infrastructure (#of hosts, VMs, jobs)?
You can postpone the log compression process by increasing a Maximum file size to 102400 and Total size of uncompressed logs to 5120000. This way you will have more uncompressed logs, instead of archived ones, which should decrease number of IOps on the VBR VM.
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Backup Copy Job process

Post by nmdange »

We have 15 backup jobs and 15 backup copy jobs, though that 50gb of log files is being generated by a single backup copy job. That particular job has about 50 VMs in it, which total just over 2tb in size.

I noticed another registry entry called "LoggingLevel" which was set to 6. I found the KB article for that https://www.veeam.com/kb1247 and it looks like that is the highest level of detail, and the default is 4. I don't remember changing this setting, but it's possible an engineer on a previous support case asked me to change it. I'm going to change the logging level back to the default and hopefully that fixes it! Does Veeam have to be restarted for the registry changes to take effect?
veremin
Product Manager
Posts: 20415
Liked: 2302 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Copy Job process

Post by veremin »

As mentioned in the referenced KB article, you need to restart Veeam Backup Service. In order to be on the safe side, you might want to restart all Veeam specific services. Thanks.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Backup Copy Job process

Post by Shestakov »

nmdange wrote:We have 15 backup jobs and 15 backup copy jobs, though that 50gb of log files is being generated by a single backup copy job. That particular job has about 50 VMs in it, which total just over 2tb in size.
50GB of logs for your infrastructure that`s a lot. High logging level explains it. So set back the logginglevel, if it will not help, make changes I described above, if you still have problems after all let us know.
Thanks!
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Backup Copy Job process

Post by nmdange »

Ok I changed the logging level and restarted the server. Magically the system is running so much better! So lesson learned, if the logging level is set to the maximum possible, it can generate a huge amount of IO and impact the system performance.

Thank you everyone for your input!
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Backup Copy Job process

Post by Shestakov »

Glad that the issue is found!
Once you have a question, don`t hesitate asking.
fmprojecting
Lurker
Posts: 1
Liked: never
Joined: Oct 12, 2015 9:07 am
Full Name: fmprojecting
Contact:

[MERGED]: task log under C:\ProgramData\Veeam\Backup size

Post by fmprojecting »

We have size issue on C:\ProgramData\Veeam\Backup ( 15 Gb folder size ), every run job create a folder with own name and under this all machine on job create file task."machinename"...,
this file are 43 Mb for every machine, the log write in to this is verbose ...
Is possible to reduce the log level to this task ?
Or applay any fix to this process for reduce the size ?
regards

Our Backup machine is veeam Backup 8.0.1
Post Reply

Who is online

Users browsing this forum: No registered users and 27 guests