Backup Copy Job process

[nmdange]

I've noticed that while our backup copy jobs are running, particularly when the copy job is merging the oldest incremental into the full, the Veeam B&R server is very slow. Using the console or even just doing anything within Windows on the server takes forever.

Looking at Resource Monitor, I see that there is a lot of I/O happening to files with names like this:
C:\ProgramData\Veeam\Backup\<copy job name>\<date>_<copy job name>.zip
and in the same folder a "DotNetZip" tmp file.

I think because all this I/O is happening on the system drive of the Backup & Recovery server, it is causing the entire operating system to respond very slow

Is there any way to change where this activity happens? I would think that it would make more sense for the copy to go straight from the source backup repository to the destination backup repository. At least, in cases like mine where both repositories are Windows servers. This seems to only be an issue with Copy jobs and not the actual Backup jobs.

[Shestakov]

Hello,
It`s always better to start troubleshooting from the bottleneck analysis. Could you provide the stats?
Where do you see high IOPs, on the source or target site?
Thanks!

[veremin]

C:\ProgramData\Veeam\Backup\<copy job name>\<date>_<copy job name>.zip

This archive contains Target datamover and Backup Sync logs, so, it shouldn't be the root cause of degraded performance. As Nikita's suggested, it's better to start with bottleneck statistics review. Thanks.

[foggy]

I also wonder what are the source and target repositories for this copy job and how they are added into Veeam B&R?

[nmdange]

The bottleneck is almost always listed as "Source" for all of our backup jobs and copy jobs. Very rarely, the jobs will show "Throttling" as the bottleneck. I have a throttling rule set to 8000 Mbps to prevent the backups from saturating our 10gb links. I have never seen the bottleneck show "Target". In terms of percentages, the Source is usually 70%+, Proxy is 10% or less, and network can vary. Target always shows 0%.

Both the source and target repositories are of type "Windows" in Veeam. They are physical servers running Windows Server 2012 R2 with Hyper-V enabled, with a bunch of RAID 50 disks with 4TB 7.2k SATA drives. They also act as an Off-Host Proxy for our Hyper-V backups. The performance of the actual backups themselves seems to be ok.

The High IOPs I see are not on either the source or target repository server, but the C drive of the Veeam B&R server itself. Looking in Performance Monitor while the Copy Jobs are running, I see 1000+ write IO/sec on the disk, and a queue length with 15+ outsanding IOs. The Veeam B&R server is a VM so I moved the C drive to a different physical disk array and it seems to have helped somewhat. It just seems like the Copy Jobs are generating a huge amount of disk activity on the C drive of the Veeam B&R server when the activity should really be Source Repository -> Target Repository.

[Shestakov]

Queue of 15+ IO requests is a big one. What is the delay?

There can be several reasons of high IO rate on the source. I would start with next steps:
1) Check if you have any Undersized VMs by memory, which can use disk space to extend physical memory size.
2) Disable all non-related to virtualization roles on the HV host.

What version of Windows Server are you at? Thanks!

[Shestakov]

In addition to those general solutions of high IOPs, if the reason is

C:\ProgramData\Veeam\Backup\<copy job name>\<date>_<copy job name>.zip
and in the same folder a "DotNetZip" tmp file.

You might need to set a proper retention for your log files. Since looks like during backup copy job merge lots of new logs are generated which causes the logs compression.
Please review the KB to adjust related settings. Thanks!

[nmdange]

Everything is 2012 R2. I am looking at the IOPs and Disk Queue inside the Veeam B&R VM, not at the host. Looking at Resource Monitor at the disk detail is how I determined those files as being the cause.

So I'm looking at the log files and the size of the .zip files for the backup copy jobs that are using most of the IOPs are 6gb each. In addition, there are over 5,000 10MB log files created per day.

It looks like a new zip file is created every 2 weeks? Looking at that KB article, it doesn't look like there's a way to limit the size of the compressed logs or how often they are created? Any recommendations on how to change the settings? Right now everything is the default.

[Shestakov]

50 GBs of logs a day is a huge number. How big is your infrastructure (#of hosts, VMs, jobs)?
You can postpone the log compression process by increasing a Maximum file size to 102400 and Total size of uncompressed logs to 5120000. This way you will have more uncompressed logs, instead of archived ones, which should decrease number of IOps on the VBR VM.

[nmdange]

We have 15 backup jobs and 15 backup copy jobs, though that 50gb of log files is being generated by a single backup copy job. That particular job has about 50 VMs in it, which total just over 2tb in size.

I noticed another registry entry called "LoggingLevel" which was set to 6. I found the KB article for that https://www.veeam.com/kb1247 and it looks like that is the highest level of detail, and the default is 4. I don't remember changing this setting, but it's possible an engineer on a previous support case asked me to change it. I'm going to change the logging level back to the default and hopefully that fixes it! Does Veeam have to be restarted for the registry changes to take effect?

[veremin]

As mentioned in the referenced KB article, you need to restart Veeam Backup Service. In order to be on the safe side, you might want to restart all Veeam specific services. Thanks.

[Shestakov]

We have 15 backup jobs and 15 backup copy jobs, though that 50gb of log files is being generated by a single backup copy job. That particular job has about 50 VMs in it, which total just over 2tb in size.

50GB of logs for your infrastructure that`s a lot. High logging level explains it. So set back the logginglevel, if it will not help, make changes I described above, if you still have problems after all let us know.
Thanks!

[nmdange]

Ok I changed the logging level and restarted the server. Magically the system is running so much better! So lesson learned, if the logging level is set to the maximum possible, it can generate a huge amount of IO and impact the system performance.

Thank you everyone for your input!

[Shestakov]

Glad that the issue is found!
Once you have a question, don`t hesitate asking.

[fmprojecting]

We have size issue on C:\ProgramData\Veeam\Backup ( 15 Gb folder size ), every run job create a folder with own name and under this all machine on job create file task."machinename"...,
this file are 43 Mb for every machine, the log write in to this is verbose ...
Is possible to reduce the log level to this task ?
Or applay any fix to this process for reduce the size ?
regards

Our Backup machine is veeam Backup 8.0.1