Compact Full Backup feature for Backup Copy Job

[dmitri-va]

Hi,
I'm noticing that during backup copy jobs, the task of "Merging oldest incremental backup into full backup file" is taking a really long time.

I'm guessing this is due to expected fragmentation of the vbk file?

If I enable "compact Full Backup' feature for this backup copy job via the GUI, what happens if the next scheduled day happens to be during the copy cycle? will it interfere with the VM backup copying or incremental file merging which may still be going on?

The guide mentions that 'compact full backup' feature will only work if 'remove deleted VMs data from backup" is not set for the job? What's my option if I do want to remove deleted VMs?

Thanks.

[kte]

why not have the option to shedule an active full copy to start a new backup chain, but still do the muges on the previuos backup chane so you just need the flace if 1 additional full

[dmitri-va]

it's going to be a royal pain to transfer a full backup over our WAN again

[Gostev]

I'm guessing this is due to expected fragmentation of the vbk file?

No, it because poor backup storage performance (in terms of IOPS). Fragmentation has little to no impact on merge process, because it is already mostly random I/O anyway. Thanks!

[foggy]

If I enable "compact Full Backup' feature for this backup copy job via the GUI, what happens if the next scheduled day happens to be during the copy cycle? will it interfere with the VM backup copying or incremental file merging which may still be going on?

Compact operation is performed after applying retention at the end of successful job cycle, so does not interfere with the normal copy job activity.

The guide mentions that 'compact full backup' feature will only work if 'remove deleted VMs data from backup" is not set for the job? What's my option if I do want to remove deleted VMs?

This limitation relates only to the mechanism that handles data for VMs that have only one restore point, not to the entire compact operation.

[dmitri-va]

Interestingly, I didn't see (or didn't notice) this issue before, and we've been doing these backup copy jobs for quite a while now on the same hardware (11x 7k SAS disks in RAID6 behind PercH730P controller).

The merging is still going on and I just jumped on the repository server and ran resource and perf monitors.

Resource monitor showed disk working at 985MB/sec, even though sometimes it goes down to single digits.

Perf monitor showed very little IOPS (unless I'm looking at the wrong counters or something): Avg. Disk sec/Read and Avg. Disk sec/Write were both at 0.011 average, and 0.051 max (with scale set to 1000)

resource monitor: https://onedrive.live.com/redir?resid=B ... hoto%2cjpg

perf mon: https://onedrive.live.com/redir?resid=B ... hoto%2cjpg

I can provide more counters or info, if someone is interested to dig deeper, or should I just create a support case?

[dmitri-va]

ok, have only been collecting for a small amount of time, but here are the numbers from proper counters.

read iops: 53 average, 712 max
write iops: 39 average, 3,900 max (only one spike that big)

current disk queue length: 0.73 average, 4 max

it doesn't look like the merging is IOPS intensive, and it seems our setup is adequate? Also, as I mentioned, most of the time disk throughput is single digits (MB/sec), so what is it doing during those times, while CPU is also at 1% ?

[dmitri-va]

could this be a scalability issue? this one is a copy of the pretty big backup job (180VMs, ~10TB full backup, ~0.5-1.8TB incrementals)?

it seems the slow merging issue started around the time we virtualized file and print server which added ~2.5TB to the backup. Maybe it crossed some limit there?

[Gostev]

Unlikely (not due to backup size at least). Are you using v9 by the way? It had tons of improvements around merge process.

[dmitri-va]

no, but we are close to upgrading it to v9. looking forward to bitlooker and standalone backup admin client.

btw, this may explain why we did not notice the full backup merge was taking that much time:

I know for a fact (saw the job while it was in progress) that the incremental file merge started yesterday at ~4am and was chugging along through the day.
while running, the ongoing even name was "merging the last incremental into the full backup"

however, now that the job is complete, the log says:
"full backup file merge completed successfully" 3/14/2016 11:18:32pm
"starting full backup file merge" 3/14/2016 11:18:32 pm

so the log does not reflect the actual time the merge started and let you believe it was instantaneous, while in fact it took ~19 hours

in the screenshot below you can see that the actual time the last incremental merge started is reflected by this event:

"required backup infrastructure resources have been assigned" 3/14/2016 04:19:08 am

https://onedrive.live.com/redir?resid=B ... hoto%2cjpg

[foggy]

Do you see similar divergence in the job session displayed in Veeam B&R console itself?

[dmitri-va]

as I mentioned, while the job was still running and I was looking at it in the console it was clearly sitting at the "merging last incremental backup into the full backup' or similar and the percentage was ever so slowly increasing. I didn't have a chance to look at it again in B&R console after it was complete, so the screenshot is from the Enterprise manager. I think, however, that Enterprise manager shows exactly the same log as the B&R console would.

[foggy]

Not necessarily, but I will ask QA to pay attention to it.

[dmitri-va]

btw, pushed it as an emergency change here and upgraded to v9 yesterday
will see if this helps with the merge process.

[dmitri-va]

nope, same slow merging, 10% done after 2.5 hours

perfmon shows average of 59 read iops, 64 write iops, and average of 0.6 disk queue length, so I cant say storage is stressed out. Maybe that's usual/expected, ~400MB/hour for merging on average (10TB vbk)? I know this is not storage bottleneck since I saw sustained >900MB/sec during merging per MS resource monitor

[Gostev]

No, your numbers are not normal at all. I guess it's time to open a support case and have our engineers enable advanced performance parameters logging mode to see where is the issue.

[RISIT]

I am having the same issue with 10.0.0.4461
One compact running presently for 41 hours 35 minutes and only 39% complete.
I thought this may have been WAN speed but then was corrected that this process should all be taking place on the remote machine.

has anyone got advice to resolve this issue? I know this is an older post but the problem seems to be the same. Since I don't see a resolution in this post I thought I'd ask.

Thanks.

[Gostev]

Indeed, it's been a while since we had any known issues with this functionality. Please open a support case to determine the bottleneck - keep in mind it could as well be the backup storage performance. Thanks!