merge time

[jurriaan_cap]

I started using veeam backup to backup a vsphere environment.

we are now starting to hit our restore points limit (31) and the incremental backups are starting to merge.

however it is already taking a very long time to finish.

we have around 81 VM's in the daily backup

since we have migrated vms over a period of time to veeam only 5 of those have 31 restore points. all the other vm have fewer restore points.

it now takes around 9 hours for those 5 vms to merge.

the total job has transferred 235 GB , processed 12.3 TB, read 1.6 TB.
the load is as follows : source 84 % , proxy 40 % network 29 % target 9 %

I noticed no disk activity during the merge.

is this a normal merge time ?
does it increase if more vms are hitting the restore point limit ?
if so how can we ensure that it stays within the 24 hour time limit.

can we for example offload merge to proxy servers ?

[Shestakov]

Hello,
Could you specify the kind of merge we are talking about? Is that Synthetic Full backup for 31 restore points of 5VMs?
Does it also include previous chains into rollbacks transformation?

does it increase if more vms are hitting the restore point limit ?

Yes it does. Also number of restore points matters.

if so how can we ensure that it stays within the 24 hour time limit.
can we for example offload merge to proxy servers ?

Merge process requires more Disk IO capabilities rather than Proxy CPU/Memory. To keep within 1 day you can either use storage with better performance, or make it more often(decrease number of restore points to merge) or switch to another mode, like Forever-Forward Incremental.

By the way, you can set an alarm using Veeam ONE to be notified if the job takes too long.
Here is a related thread.

[foggy]

Jurriaan, what kind of target storage do you have and how it is added to Veeam B&R?

[PTide]

Hi,

Poor merge performance might be also caused by lack of RAM on repository - in this case metadata cannot fit into cache thus frequent random I/O occur during work with metadata. Please see this thread for details.

Thank you.

[jurriaan_cap]

some answers
its a incremental backup , with NO full synthetic full backups

I am unsure what you mean with "Does it also include previous chains into rollbacks transformation?"

during the merge I see between 150 and 200 Read and write IOPS. and below 50 MBps
during the backup phase itself I het between 200 and 400 IOPS. I then hit just below 100 MBps

those numbers seem to indicate that the max of that SAN is not being hit.

the storage is a ISCSI connect MSA SAN. its connect as a VMware raw device mapping to the backup repository server

the memory usage of the backup repository VM during the merge is around 50 % (around 10 gb of the 16 GB allocated). if the issue is memory should it not be around 100 % ?

[Shestakov]

Thanks for the answer.

I am unsure what you mean with "Does it also include previous chains into rollbacks transformation?"

It`s an option related to synthetic fulls.

if the issue is memory should it not be around 100 % ?

It should. So it`s about IOPs, not memory.
And by the way the bottleneck of the job is source. So the question is what kind of transport mode do you use?
Thanks!

[alanbolte]

Are you running the latest build, 8.0.0.2030? There are known issues with merge performance in previous versions.

It is normal for merge throughput to be much lower than incremental backup throughput, because merging is a complex read-write operation, while an incremental backup is a mostly sequential write operation.

http://www.veeam.com/kb2014 describes a method for benchmarking merge performance (under heading "Transforms, merges, and other synthetic operations") using a Microsoft command-line tool run on the repository server. If you are running the latest version and your merge throughput is considerably lower than the benchmark (this is not listed in the UI so you will need to estimate merge performance by dividing the size of your incremental backup file by the time taken to merge it), and no other resources seem stressed, you might want to open a support case and post the ID in this thread.

[jurriaan_cap]

yes I am running 2030 .

I will check out the benchmark.

I already openend a support Case # 01069768

stupid question since I am not used to veeam terminology

what kind of calculation must I do on the result of the benchmar.

Code: Select all

Command Line: c:\tools\diskspd.exe -c1G -b512K -w50 -r4K -h -d600 D:\testfile.da
t

Input parameters:

        timespan:   1
        -------------
        duration: 600s
        warm up time: 5s
        cool down time: 0s
        random seed: 0
        path: 'D:\testfile.dat'
                think time: 0ms
                burst size: 0
                software and hardware write cache disabled
                performing mix test (write/read ratio: 50/100)
                block size: 524288
                using random I/O (alignment: 4096)
                number of outstanding I/O operations: 2
                thread stride size: 0
                threads per file: 1
                using I/O Completion Ports
                IO priority: normal



Results for timespan 1:
*******************************************************************************

actual test time:       600.00s
thread count:           1
proc count:             8

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   9.88%|   3.17%|    6.71%|  90.12%
   1|   2.58%|   2.02%|    0.57%|  97.42%
   2|   1.83%|   1.42%|    0.41%|  98.17%
   3|   3.57%|   2.79%|    0.78%|  96.43%
   4|   0.77%|   0.59%|    0.18%|  99.23%
   5|   4.36%|   3.59%|    0.77%|  95.64%
   6|  11.82%|   9.72%|    2.09%|  88.18%
   7|   6.44%|   4.17%|    2.27%|  93.56%
-------------------------------------------
avg.|   5.16%|   3.43%|    1.72%|  94.84%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |    174688567296 |       333192 |     277.66 |     555.32 | D:\testfile.dat (1024MB)
------------------------------------------------------------------------------
total:      174688567296 |       333192 |     277.66 |     555.32

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     87412965376 |       166727 |     138.94 |     277.88 | D:\testfile.dat (1024MB)
------------------------------------------------------------------------------
total:       87412965376 |       166727 |     138.94 |     277.88

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     87275601920 |       166465 |     138.72 |     277.44 | D:\testfile.dat (1024MB)
------------------------------------------------------------------------------
total:       87275601920 |       166465 |     138.72 |     277.44

I am running the recommended incremental backups , no syntethic full , with 31 restore points. (to be able to go back 31 days).

should I divde by 2 or 4 ?

[jurriaan_cap]

also what is my filesize that I should use to determine how long it should take ?

Code: Select all

03-09-2015  20:17   240.433.788.416 Daily Veeam Backup2015-09-03T180503.vib
04-09-2015  20:47   265.381.003.264 Daily Veeam Backup2015-09-04T180433.vib
05-09-2015  21:02   225.054.353.408 Daily Veeam Backup2015-09-05T180442.vib
06-09-2015  19:50   221.872.875.008 Daily Veeam Backup2015-09-06T180444.vib
07-09-2015  20:36   223.999.100.928 Daily Veeam Backup2015-09-07T180512.vib
08-09-2015  20:20   241.616.845.312 Daily Veeam Backup2015-09-08T180500.vib
09-09-2015  20:11   281.485.987.328 Daily Veeam Backup2015-09-09T180442.vib
10-09-2015  20:08   245.725.071.360 Daily Veeam Backup2015-09-10T180444.vib
11-09-2015  20:09   258.687.170.048 Daily Veeam Backup2015-09-11T180454.vib
12-09-2015  20:47   222.265.007.616 Daily Veeam Backup2015-09-12T180447.vib
13-09-2015  20:04   230.552.311.296 Daily Veeam Backup2015-09-13T180455.vib
14-09-2015  20:43   233.424.156.672 Daily Veeam Backup2015-09-14T180457.vib
15-09-2015  20:35   251.814.814.720 Daily Veeam Backup2015-09-15T180500.vib
16-09-2015  20:36   248.320.360.960 Daily Veeam Backup2015-09-16T180454.vib
17-09-2015  20:13   282.000.748.032 Daily Veeam Backup2015-09-17T180500.vib
18-09-2015  20:06   235.731.714.048 Daily Veeam Backup2015-09-18T180507.vib
19-09-2015  20:07   221.142.274.560 Daily Veeam Backup2015-09-19T180454.vib
20-09-2015  20:49   225.276.778.496 Daily Veeam Backup2015-09-20T180442.vib
21-09-2015  21:11   241.509.989.376 Daily Veeam Backup2015-09-21T180504.vib
22-09-2015  20:13   259.040.740.352 Daily Veeam Backup2015-09-22T180500.vib
23-09-2015  20:55   240.046.140.928 Daily Veeam Backup2015-09-23T180506.vib
24-09-2015  20:19   239.458.876.416 Daily Veeam Backup2015-09-24T180510.vib
25-09-2015  20:25   238.466.822.144 Daily Veeam Backup2015-09-25T180458.vib
26-09-2015  20:17   225.807.164.928 Daily Veeam Backup2015-09-26T180449.vib
27-09-2015  20:23   231.995.854.848 Daily Veeam Backup2015-09-27T180508.vib
28-09-2015  20:13   232.227.792.384 Daily Veeam Backup2015-09-28T180445.vib
29-09-2015  20:25   248.277.630.976 Daily Veeam Backup2015-09-29T180449.vib
30-09-2015  06:39    68.781.267.968 Daily Veeam Backup2015-09-02T201753.vib
30-09-2015  06:39   192.359.017.984 Daily Veeam Backup2015-09-02T180541.vib
30-09-2015  06:40     7.563.494.912 Daily Veeam Backup2015-09-02T152215.vib
30-09-2015  06:40 4.882.731.300.352 Daily Veeam Backup2015-09-01T180614.vbk

if I use the VBK which is 4 tb and use half the mb throughput from the previous benchmark I get the following

4 TB = 4 000 000 MB
half of 277.66 MBps is 135 MBps

4 000 000 / 135 = 29629 Seconds = 8 hours.

is this about right ?

[alanbolte]

You can take the lower of the read and write data benchmark, or divide the combined benchmark by 2; the result is usually similar and I'm not quite sure which is the more accurate method.

You then compare that to what your job is doing; that is, how quickly the oldest .vib file is copied into the .vbk. This requires that you either note the file size of the oldest VIB before it's deleted by the merge, or ask your support engineer to check the logs for the size of a past file. For example, if a .vib file is 200 GB, and it takes 300 minutes to complete the merge, that's 11 MB/s (you can use Wolfram Alpha to make the calculation, but I think you have the math right). If the benchmark read or write rate isn't massively larger than that, then the performance limitation is probably not a Veeam bug. The VBK size is not particularly important to merge performance.

Given your output, there is some cause for concern, because you're benchmarking at around 135 MB/s, but you report seeing only 50 MB/s from the performance monitoring on your storage during the merge. However, I should mention that on a good SAN the benchmark is sometimes invalidated by caching, so if your initial result is very high it may take someone with more advanced knowledge of the tools to determine whether there's a Veeam bug or if the benchmark is incorrect. Also, the -b parameter might not match your job setting; this is explained near the top of the KB, and your support engineer can verify that for you from the logs.

By the way: single-file read/write operations have some inherent bottlenecks in both OS and storage, so if this is the only job running at the time, it might make more sense to split the VMs among several jobs, so that you have several sets of backup files; the resulting parallel I/O can give better overall performance. In v9 you will have the option to keep each VM in a separate chain of files, which sacrifices deduplication for improved performance.

[jurriaan_cap]

hmm

the full backup merge now lasted 1:17:05

I am unsure if I fully understand how this works.
but if the merge is getting the second last VIB file into the VBk file than I think that merging 7.5 GB (30-09-2015 06:40 7.563.494.912 Daily Veeam Backup2015-09-02T152215.vib)
took over 1 hour and 17 minutes

if I have around 135 MBps of throughput than it should have been around 60 seconds ((7,5 x 1014) /135 = 60 seconds).

if I look at the throughput of the vm is very low during the merge

less than 10 MBps

which would explain why it takes a hour.

[ITP-Stan]

Correct if I'm wrong, but (forward incremental) merging I/O is all about the target storage where the backups are stored.The source storage is no longer in play at that point.
I think the job statistics don't count merging, but only the incremental backup part itself.

I've had issues with merging performance as well at a client, and worked with support for several weeks. Doing the benchmarks, testing config changes and private fixes.

In the end it improved the merge performance slightly, but I think the clients target storage for backups (a NAS with 4x 5400rpm disks in RAID5) can't handle more.

[lando_uk]

If your target is a Windows server, open resource monitor and look at disk queue lengths when a merge is taking place, if it's over 2, then your target storage is suffering.

Think yourself lucky, some of our merges take 35 hours... The best way to speed it up is to scrap RAID5 or 6 and use RAID10. If I could go back in time and build the whole system again, I would have made the business pay more and use RAID10.

[foggy]

Correct if I'm wrong, but (forward incremental) merging I/O is all about the target storage where the backups are stored.The source storage is no longer in play at that point.

I'd say, it is all about the server, where target data mover runs (which is, for example, in case of CIFS repository, not the target storage itself, but rather server specified as a gateway for it) and target storage performance.

I think the job statistics don't count merging, but only the incremental backup part itself.

That's correct.

[ITP-Stan]

Well since the backup servers has to read the incremental from target storage and then write also to target storage, you have twice the amount of I/O on that target storage than when just creating an incremental. My point was that the source storager where the VM's live, is no longer used when a merge runs.
I'm sure the backupserver itself is also important in terms of RAM and CPU power, but that's always the case if it's also a proxy like it mostly is.

@lando_uk: How can your job run daily if the merge takes 35 hours?

[foggy]

Well since the backup servers has to read the incremental from target storage and then write also to target storage, you have twice the amount of I/O on that target storage than when just creating an incremental. My point was that the source storager where the VM's live, is no longer used when a merge runs.

That point is correct.

I'm sure the backupserver itself is also important in terms of RAM and CPU power, but that's always the case if it's also a proxy like it mostly is.

Right, you can see the requirements in the corresponding user guide section.