-
- Influencer
- Posts: 15
- Liked: 1 time
- Joined: Feb 03, 2015 2:22 pm
- Full Name: Jurriaan Cap
- Contact:
merge time
I started using veeam backup to backup a vsphere environment.
we are now starting to hit our restore points limit (31) and the incremental backups are starting to merge.
however it is already taking a very long time to finish.
we have around 81 VM's in the daily backup
since we have migrated vms over a period of time to veeam only 5 of those have 31 restore points. all the other vm have fewer restore points.
it now takes around 9 hours for those 5 vms to merge.
the total job has transferred 235 GB , processed 12.3 TB, read 1.6 TB.
the load is as follows : source 84 % , proxy 40 % network 29 % target 9 %
I noticed no disk activity during the merge.
is this a normal merge time ?
does it increase if more vms are hitting the restore point limit ?
if so how can we ensure that it stays within the 24 hour time limit.
can we for example offload merge to proxy servers ?
we are now starting to hit our restore points limit (31) and the incremental backups are starting to merge.
however it is already taking a very long time to finish.
we have around 81 VM's in the daily backup
since we have migrated vms over a period of time to veeam only 5 of those have 31 restore points. all the other vm have fewer restore points.
it now takes around 9 hours for those 5 vms to merge.
the total job has transferred 235 GB , processed 12.3 TB, read 1.6 TB.
the load is as follows : source 84 % , proxy 40 % network 29 % target 9 %
I noticed no disk activity during the merge.
is this a normal merge time ?
does it increase if more vms are hitting the restore point limit ?
if so how can we ensure that it stays within the 24 hour time limit.
can we for example offload merge to proxy servers ?
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: merge time
Hello,
Could you specify the kind of merge we are talking about? Is that Synthetic Full backup for 31 restore points of 5VMs?
Does it also include previous chains into rollbacks transformation?
By the way, you can set an alarm using Veeam ONE to be notified if the job takes too long.
Here is a related thread.
Could you specify the kind of merge we are talking about? Is that Synthetic Full backup for 31 restore points of 5VMs?
Does it also include previous chains into rollbacks transformation?
Yes it does. Also number of restore points matters.jurriaan_cap wrote:does it increase if more vms are hitting the restore point limit ?
Merge process requires more Disk IO capabilities rather than Proxy CPU/Memory. To keep within 1 day you can either use storage with better performance, or make it more often(decrease number of restore points to merge) or switch to another mode, like Forever-Forward Incremental.jurriaan_cap wrote:if so how can we ensure that it stays within the 24 hour time limit.
can we for example offload merge to proxy servers ?
By the way, you can set an alarm using Veeam ONE to be notified if the job takes too long.
Here is a related thread.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: merge time
Jurriaan, what kind of target storage do you have and how it is added to Veeam B&R?
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: merge time
Hi,
Poor merge performance might be also caused by lack of RAM on repository - in this case metadata cannot fit into cache thus frequent random I/O occur during work with metadata. Please see this thread for details.
Thank you.
Poor merge performance might be also caused by lack of RAM on repository - in this case metadata cannot fit into cache thus frequent random I/O occur during work with metadata. Please see this thread for details.
Thank you.
-
- Influencer
- Posts: 15
- Liked: 1 time
- Joined: Feb 03, 2015 2:22 pm
- Full Name: Jurriaan Cap
- Contact:
Re: merge time
some answers
its a incremental backup , with NO full synthetic full backups
I am unsure what you mean with "Does it also include previous chains into rollbacks transformation?"
during the merge I see between 150 and 200 Read and write IOPS. and below 50 MBps
during the backup phase itself I het between 200 and 400 IOPS. I then hit just below 100 MBps
those numbers seem to indicate that the max of that SAN is not being hit.
the storage is a ISCSI connect MSA SAN. its connect as a VMware raw device mapping to the backup repository server
the memory usage of the backup repository VM during the merge is around 50 % (around 10 gb of the 16 GB allocated). if the issue is memory should it not be around 100 % ?
its a incremental backup , with NO full synthetic full backups
I am unsure what you mean with "Does it also include previous chains into rollbacks transformation?"
during the merge I see between 150 and 200 Read and write IOPS. and below 50 MBps
during the backup phase itself I het between 200 and 400 IOPS. I then hit just below 100 MBps
those numbers seem to indicate that the max of that SAN is not being hit.
the storage is a ISCSI connect MSA SAN. its connect as a VMware raw device mapping to the backup repository server
the memory usage of the backup repository VM during the merge is around 50 % (around 10 gb of the 16 GB allocated). if the issue is memory should it not be around 100 % ?
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: merge time
Thanks for the answer.
And by the way the bottleneck of the job is source. So the question is what kind of transport mode do you use?
Thanks!
It`s an option related to synthetic fulls.jurriaan_cap wrote:I am unsure what you mean with "Does it also include previous chains into rollbacks transformation?"
It should. So it`s about IOPs, not memory.jurriaan_cap wrote:if the issue is memory should it not be around 100 % ?
And by the way the bottleneck of the job is source. So the question is what kind of transport mode do you use?
Thanks!
-
- Veteran
- Posts: 635
- Liked: 174 times
- Joined: Jun 18, 2012 8:58 pm
- Full Name: Alan Bolte
- Contact:
Re: merge time
Are you running the latest build, 8.0.0.2030? There are known issues with merge performance in previous versions.
It is normal for merge throughput to be much lower than incremental backup throughput, because merging is a complex read-write operation, while an incremental backup is a mostly sequential write operation.
http://www.veeam.com/kb2014 describes a method for benchmarking merge performance (under heading "Transforms, merges, and other synthetic operations") using a Microsoft command-line tool run on the repository server. If you are running the latest version and your merge throughput is considerably lower than the benchmark (this is not listed in the UI so you will need to estimate merge performance by dividing the size of your incremental backup file by the time taken to merge it), and no other resources seem stressed, you might want to open a support case and post the ID in this thread.
It is normal for merge throughput to be much lower than incremental backup throughput, because merging is a complex read-write operation, while an incremental backup is a mostly sequential write operation.
http://www.veeam.com/kb2014 describes a method for benchmarking merge performance (under heading "Transforms, merges, and other synthetic operations") using a Microsoft command-line tool run on the repository server. If you are running the latest version and your merge throughput is considerably lower than the benchmark (this is not listed in the UI so you will need to estimate merge performance by dividing the size of your incremental backup file by the time taken to merge it), and no other resources seem stressed, you might want to open a support case and post the ID in this thread.
-
- Influencer
- Posts: 15
- Liked: 1 time
- Joined: Feb 03, 2015 2:22 pm
- Full Name: Jurriaan Cap
- Contact:
Re: merge time
yes I am running 2030 .
I will check out the benchmark.
I already openend a support Case # 01069768
stupid question since I am not used to veeam terminology
what kind of calculation must I do on the result of the benchmar.
I am running the recommended incremental backups , no syntethic full , with 31 restore points. (to be able to go back 31 days).
should I divde by 2 or 4 ?
I will check out the benchmark.
I already openend a support Case # 01069768
stupid question since I am not used to veeam terminology
what kind of calculation must I do on the result of the benchmar.
Code: Select all
Command Line: c:\tools\diskspd.exe -c1G -b512K -w50 -r4K -h -d600 D:\testfile.da
t
Input parameters:
timespan: 1
-------------
duration: 600s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'D:\testfile.dat'
think time: 0ms
burst size: 0
software and hardware write cache disabled
performing mix test (write/read ratio: 50/100)
block size: 524288
using random I/O (alignment: 4096)
number of outstanding I/O operations: 2
thread stride size: 0
threads per file: 1
using I/O Completion Ports
IO priority: normal
Results for timespan 1:
*******************************************************************************
actual test time: 600.00s
thread count: 1
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 9.88%| 3.17%| 6.71%| 90.12%
1| 2.58%| 2.02%| 0.57%| 97.42%
2| 1.83%| 1.42%| 0.41%| 98.17%
3| 3.57%| 2.79%| 0.78%| 96.43%
4| 0.77%| 0.59%| 0.18%| 99.23%
5| 4.36%| 3.59%| 0.77%| 95.64%
6| 11.82%| 9.72%| 2.09%| 88.18%
7| 6.44%| 4.17%| 2.27%| 93.56%
-------------------------------------------
avg.| 5.16%| 3.43%| 1.72%| 94.84%
Total IO
thread | bytes | I/Os | MB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 174688567296 | 333192 | 277.66 | 555.32 | D:\testfile.dat (1024MB)
------------------------------------------------------------------------------
total: 174688567296 | 333192 | 277.66 | 555.32
Read IO
thread | bytes | I/Os | MB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 87412965376 | 166727 | 138.94 | 277.88 | D:\testfile.dat (1024MB)
------------------------------------------------------------------------------
total: 87412965376 | 166727 | 138.94 | 277.88
Write IO
thread | bytes | I/Os | MB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 87275601920 | 166465 | 138.72 | 277.44 | D:\testfile.dat (1024MB)
------------------------------------------------------------------------------
total: 87275601920 | 166465 | 138.72 | 277.44
should I divde by 2 or 4 ?
-
- Influencer
- Posts: 15
- Liked: 1 time
- Joined: Feb 03, 2015 2:22 pm
- Full Name: Jurriaan Cap
- Contact:
Re: merge time
also what is my filesize that I should use to determine how long it should take ?
if I use the VBK which is 4 tb and use half the mb throughput from the previous benchmark I get the following
4 TB = 4 000 000 MB
half of 277.66 MBps is 135 MBps
4 000 000 / 135 = 29629 Seconds = 8 hours.
is this about right ?
Code: Select all
03-09-2015 20:17 240.433.788.416 Daily Veeam Backup2015-09-03T180503.vib
04-09-2015 20:47 265.381.003.264 Daily Veeam Backup2015-09-04T180433.vib
05-09-2015 21:02 225.054.353.408 Daily Veeam Backup2015-09-05T180442.vib
06-09-2015 19:50 221.872.875.008 Daily Veeam Backup2015-09-06T180444.vib
07-09-2015 20:36 223.999.100.928 Daily Veeam Backup2015-09-07T180512.vib
08-09-2015 20:20 241.616.845.312 Daily Veeam Backup2015-09-08T180500.vib
09-09-2015 20:11 281.485.987.328 Daily Veeam Backup2015-09-09T180442.vib
10-09-2015 20:08 245.725.071.360 Daily Veeam Backup2015-09-10T180444.vib
11-09-2015 20:09 258.687.170.048 Daily Veeam Backup2015-09-11T180454.vib
12-09-2015 20:47 222.265.007.616 Daily Veeam Backup2015-09-12T180447.vib
13-09-2015 20:04 230.552.311.296 Daily Veeam Backup2015-09-13T180455.vib
14-09-2015 20:43 233.424.156.672 Daily Veeam Backup2015-09-14T180457.vib
15-09-2015 20:35 251.814.814.720 Daily Veeam Backup2015-09-15T180500.vib
16-09-2015 20:36 248.320.360.960 Daily Veeam Backup2015-09-16T180454.vib
17-09-2015 20:13 282.000.748.032 Daily Veeam Backup2015-09-17T180500.vib
18-09-2015 20:06 235.731.714.048 Daily Veeam Backup2015-09-18T180507.vib
19-09-2015 20:07 221.142.274.560 Daily Veeam Backup2015-09-19T180454.vib
20-09-2015 20:49 225.276.778.496 Daily Veeam Backup2015-09-20T180442.vib
21-09-2015 21:11 241.509.989.376 Daily Veeam Backup2015-09-21T180504.vib
22-09-2015 20:13 259.040.740.352 Daily Veeam Backup2015-09-22T180500.vib
23-09-2015 20:55 240.046.140.928 Daily Veeam Backup2015-09-23T180506.vib
24-09-2015 20:19 239.458.876.416 Daily Veeam Backup2015-09-24T180510.vib
25-09-2015 20:25 238.466.822.144 Daily Veeam Backup2015-09-25T180458.vib
26-09-2015 20:17 225.807.164.928 Daily Veeam Backup2015-09-26T180449.vib
27-09-2015 20:23 231.995.854.848 Daily Veeam Backup2015-09-27T180508.vib
28-09-2015 20:13 232.227.792.384 Daily Veeam Backup2015-09-28T180445.vib
29-09-2015 20:25 248.277.630.976 Daily Veeam Backup2015-09-29T180449.vib
30-09-2015 06:39 68.781.267.968 Daily Veeam Backup2015-09-02T201753.vib
30-09-2015 06:39 192.359.017.984 Daily Veeam Backup2015-09-02T180541.vib
30-09-2015 06:40 7.563.494.912 Daily Veeam Backup2015-09-02T152215.vib
30-09-2015 06:40 4.882.731.300.352 Daily Veeam Backup2015-09-01T180614.vbk
4 TB = 4 000 000 MB
half of 277.66 MBps is 135 MBps
4 000 000 / 135 = 29629 Seconds = 8 hours.
is this about right ?
-
- Veteran
- Posts: 635
- Liked: 174 times
- Joined: Jun 18, 2012 8:58 pm
- Full Name: Alan Bolte
- Contact:
Re: merge time
You can take the lower of the read and write data benchmark, or divide the combined benchmark by 2; the result is usually similar and I'm not quite sure which is the more accurate method.
You then compare that to what your job is doing; that is, how quickly the oldest .vib file is copied into the .vbk. This requires that you either note the file size of the oldest VIB before it's deleted by the merge, or ask your support engineer to check the logs for the size of a past file. For example, if a .vib file is 200 GB, and it takes 300 minutes to complete the merge, that's 11 MB/s (you can use Wolfram Alpha to make the calculation, but I think you have the math right). If the benchmark read or write rate isn't massively larger than that, then the performance limitation is probably not a Veeam bug. The VBK size is not particularly important to merge performance.
Given your output, there is some cause for concern, because you're benchmarking at around 135 MB/s, but you report seeing only 50 MB/s from the performance monitoring on your storage during the merge. However, I should mention that on a good SAN the benchmark is sometimes invalidated by caching, so if your initial result is very high it may take someone with more advanced knowledge of the tools to determine whether there's a Veeam bug or if the benchmark is incorrect. Also, the -b parameter might not match your job setting; this is explained near the top of the KB, and your support engineer can verify that for you from the logs.
By the way: single-file read/write operations have some inherent bottlenecks in both OS and storage, so if this is the only job running at the time, it might make more sense to split the VMs among several jobs, so that you have several sets of backup files; the resulting parallel I/O can give better overall performance. In v9 you will have the option to keep each VM in a separate chain of files, which sacrifices deduplication for improved performance.
You then compare that to what your job is doing; that is, how quickly the oldest .vib file is copied into the .vbk. This requires that you either note the file size of the oldest VIB before it's deleted by the merge, or ask your support engineer to check the logs for the size of a past file. For example, if a .vib file is 200 GB, and it takes 300 minutes to complete the merge, that's 11 MB/s (you can use Wolfram Alpha to make the calculation, but I think you have the math right). If the benchmark read or write rate isn't massively larger than that, then the performance limitation is probably not a Veeam bug. The VBK size is not particularly important to merge performance.
Given your output, there is some cause for concern, because you're benchmarking at around 135 MB/s, but you report seeing only 50 MB/s from the performance monitoring on your storage during the merge. However, I should mention that on a good SAN the benchmark is sometimes invalidated by caching, so if your initial result is very high it may take someone with more advanced knowledge of the tools to determine whether there's a Veeam bug or if the benchmark is incorrect. Also, the -b parameter might not match your job setting; this is explained near the top of the KB, and your support engineer can verify that for you from the logs.
By the way: single-file read/write operations have some inherent bottlenecks in both OS and storage, so if this is the only job running at the time, it might make more sense to split the VMs among several jobs, so that you have several sets of backup files; the resulting parallel I/O can give better overall performance. In v9 you will have the option to keep each VM in a separate chain of files, which sacrifices deduplication for improved performance.
-
- Influencer
- Posts: 15
- Liked: 1 time
- Joined: Feb 03, 2015 2:22 pm
- Full Name: Jurriaan Cap
- Contact:
Re: merge time
hmm
the full backup merge now lasted 1:17:05
I am unsure if I fully understand how this works.
but if the merge is getting the second last VIB file into the VBk file than I think that merging 7.5 GB (30-09-2015 06:40 7.563.494.912 Daily Veeam Backup2015-09-02T152215.vib)
took over 1 hour and 17 minutes
if I have around 135 MBps of throughput than it should have been around 60 seconds ((7,5 x 1014) /135 = 60 seconds).
if I look at the throughput of the vm is very low during the merge
less than 10 MBps
which would explain why it takes a hour.
the full backup merge now lasted 1:17:05
I am unsure if I fully understand how this works.
but if the merge is getting the second last VIB file into the VBk file than I think that merging 7.5 GB (30-09-2015 06:40 7.563.494.912 Daily Veeam Backup2015-09-02T152215.vib)
took over 1 hour and 17 minutes
if I have around 135 MBps of throughput than it should have been around 60 seconds ((7,5 x 1014) /135 = 60 seconds).
if I look at the throughput of the vm is very low during the merge
less than 10 MBps
which would explain why it takes a hour.
-
- Expert
- Posts: 214
- Liked: 61 times
- Joined: Feb 18, 2013 10:45 am
- Full Name: Stan G
- Contact:
Re: merge time
Correct if I'm wrong, but (forward incremental) merging I/O is all about the target storage where the backups are stored.The source storage is no longer in play at that point.
I think the job statistics don't count merging, but only the incremental backup part itself.
I've had issues with merging performance as well at a client, and worked with support for several weeks. Doing the benchmarks, testing config changes and private fixes.
In the end it improved the merge performance slightly, but I think the clients target storage for backups (a NAS with 4x 5400rpm disks in RAID5) can't handle more.
I think the job statistics don't count merging, but only the incremental backup part itself.
I've had issues with merging performance as well at a client, and worked with support for several weeks. Doing the benchmarks, testing config changes and private fixes.
In the end it improved the merge performance slightly, but I think the clients target storage for backups (a NAS with 4x 5400rpm disks in RAID5) can't handle more.
-
- Veteran
- Posts: 385
- Liked: 39 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: merge time
If your target is a Windows server, open resource monitor and look at disk queue lengths when a merge is taking place, if it's over 2, then your target storage is suffering.
Think yourself lucky, some of our merges take 35 hours... The best way to speed it up is to scrap RAID5 or 6 and use RAID10. If I could go back in time and build the whole system again, I would have made the business pay more and use RAID10.
Think yourself lucky, some of our merges take 35 hours... The best way to speed it up is to scrap RAID5 or 6 and use RAID10. If I could go back in time and build the whole system again, I would have made the business pay more and use RAID10.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: merge time
I'd say, it is all about the server, where target data mover runs (which is, for example, in case of CIFS repository, not the target storage itself, but rather server specified as a gateway for it) and target storage performance.ITP-Stan wrote:Correct if I'm wrong, but (forward incremental) merging I/O is all about the target storage where the backups are stored.The source storage is no longer in play at that point.
That's correct.ITP-Stan wrote:I think the job statistics don't count merging, but only the incremental backup part itself.
-
- Expert
- Posts: 214
- Liked: 61 times
- Joined: Feb 18, 2013 10:45 am
- Full Name: Stan G
- Contact:
Re: merge time
Well since the backup servers has to read the incremental from target storage and then write also to target storage, you have twice the amount of I/O on that target storage than when just creating an incremental. My point was that the source storager where the VM's live, is no longer used when a merge runs.
I'm sure the backupserver itself is also important in terms of RAM and CPU power, but that's always the case if it's also a proxy like it mostly is.
@lando_uk: How can your job run daily if the merge takes 35 hours?
I'm sure the backupserver itself is also important in terms of RAM and CPU power, but that's always the case if it's also a proxy like it mostly is.
@lando_uk: How can your job run daily if the merge takes 35 hours?
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: merge time
That point is correct.ITP-Stan wrote:Well since the backup servers has to read the incremental from target storage and then write also to target storage, you have twice the amount of I/O on that target storage than when just creating an incremental. My point was that the source storager where the VM's live, is no longer used when a merge runs.
Right, you can see the requirements in the corresponding user guide section.ITP-Stan wrote:I'm sure the backupserver itself is also important in terms of RAM and CPU power, but that's always the case if it's also a proxy like it mostly is.
Who is online
Users browsing this forum: Semrush [Bot] and 131 guests