New Caching Feature in Veeam v8 Update 2b

[HJAdams123]

Hello and Good Day Veeam Community....

I have some questions about the new performance related features of Veeam v8 Update 2. Specifically, the job performance improvements and the slow backup storage optimization....

For the slow backup storage optimization, I am assuming more information is being cached in RAM rather than being retrieved from disk on the repository. What parts of a backup job is faster because of this? Considering a forever incremental job that is limited to 30 restore points. Full vbk is 2.7TB and daily increment file is anywhere from 80GB to 220GB. Is it the backup time that's increased? Any idea by how much? (what are other peoples experience?) Also, how about the merge process that occurs in a forever incremental job? Does merge process speed up as well because of the caching feature?

Also, I saw that a repository server needs like 2GB more RAM for this new feature? I have 4 jobs, each with their own Virtual Windows 2008 R2 repository. Each repository has 8GB of RAM. So I need to bump them all to 10GB?

Thanks for any feedback you can provide...

-Harold

[PTide]

Hi Harold,

What parts of a backup job is faster because of this?

In case of increments the proccess of changed blocks tracking gets faster since it heavily relies on meta.

Is it the backup time that's increased?

Did you mean "decreased"?

Any idea by how much? (what are other peoples experience?)

Well, it depends...Please let me explain:

There will be literally no impact of this feature if you have 1-2 single-drived VMs. The more .vmdks you process in your jobs - the more meta you have to deal with - the more performance gain from caching you get.

Does merge process speed up as well because of the caching feature?

Absolutely. The process of merging uses meta quite intensively in order to calculate the differences between RPs.

Also, I saw that a repository server needs like 2GB more RAM for this new feature? I have 4 jobs, each with their own Virtual Windows 2008 R2 repository. Each repository has 8GB of RAM. So I need to bump them all to 10GB?

I don't think so. Based on this article I've attampted to do some calculations...

JOB GIVEN:

.VBK = 2,7 TB
RPs = 30
.VIB (average) = 150 GB

2,7 TB (.VBK) + 150 GB (average backup size) x 30 (RPs) = 7,2 TB.

Let's round it to 7,5.

Next, depending on your blocksize (please refer to this article to get more info about storage settings) you need to multiply 7,5 by corresponding amount of memory (can be found here)

For default blocksize (local target) that would be 7,5 x 175 MB = 1312,5 MB extra per job.

Thank you.

[HJAdams123]

Yeah, I meant decreased, not increased.

Well this helps a lot. Thank you for the detailed response. The biggest performance gain I would like to get is not from the backup time itself (the time to actually backup the vm from snapshot create till the snapshot is removed), but its the merge process. Those virtual machines backup jobs I described, the backup takes 1.5 hours to complete (super fast) but the merge process takes 8-10 hours to complete. So if I can cut down that merge process, even by 30%, this feature would be awesome for me....

-Harold

[PTide]

Caching will do the job for merges, just give it a try. If you do, please give us a feedback about results.

P.S. Btw, how many VMs do you have in that 2,7TB .vbk?

Thank you.

[HJAdams123]

Each job is only backing up one virtual machine, a Exchange 2010 mailbox sever. The servers themselves are about 6.7TB in size. The virtual machines have 1 80GB vmdk, 3 2TB vmdk's, and two 300GB vmdk's.

I am looking to do an update in one of my sites within the next 2 weeks, so I will try to remember and update this post....

-Harold

[HJAdams123]

Well, I went ahead and loaded Update2b this past Sunday....

So far, I am sorta in the same boat, the merge process of the oldest increment is painfully slow. I am watching the jobs, vm processing takes like 1.5 hours. Then the merge process begins. According to Veeam, I get to 37% completion of the merge process in 7 mins. Yet the entire merge process takes 10-14 hours. How is that possible? When I observe active memory for the repository (its a Windows 2008 R2 virtual machines on vsphere 5.5), its actively using amount all 10GB of memory I provisioned for it in the early going. I am assuming that the new caching mechanism kicking in during that 7 minutes....but them active memory after the first 15 minutes falls drastically to ~2GB. I am assuming that's when the merge process comes to a crawl. These repositories used to have 8GB of RAM a piece, and I gave them 2GB more in preparation of Update2b. So the question is, is memory usage is directly tied to the performance of the merge process, why is the datamover only using it in the early going? Why is it I can complete 37% of the merge process in 7 minutes, yet it takes 10-12 hours to complete the rest? Has anyone else notice this sort of behavior during the merge process?

-Harold

[PTide]

Hi Harold,

I'm going to make a number of statements about what has happened in order to reflect how I got it. Please correct me if I got something wrong.

You did all the things mentioned with a single VM (the only VM in a job), 6 drives (6.7 TB in total), backup job went as fast as usual, while it took 7 minutes to complete 37% of merging and 10-12 hours to finish the rest of it. Also during first 10-15 minute RAM consumption was high, but during other 10 hours it was somewhere around 2 Gb out of 10.

Also you did nothing from this guide what could have disabled the cache.

Did I get it right?

UPDATE: Not trying to doubt your actions in any way, but I just want to make sure that I have a proper vision of the situation.

[HJAdams123]

You are exactly right.

[PTide]

Ok, then let's start.

1. It shows 37% completed because during the merge process percents are being counted depending on the amount of VM files processed. All those tiny .vmx, .nvram, .vmxf and others get merged first, displaying that 37% files processed, yet you have at least 3 huge 2Tb .vmdks to go...that's where your 10 hours start.

2. Forever incremental merging requires only the oldest .vib's meta for merging with .vbk. Assume your oldest .vib is 220Gb, then the meta to be kept in RAM would be at most ~140MB (WAN block size). So 2Gb is a normal value for merge process assuming that RAM is used somewhere else aside merging. What caching could do in your case has been already done, I believe.

Let's put it simple - once your oldest .vib's meta is cached, then the only thing your repo has to do is just to merge .vbk and .vib residing on your repo's spindles...

Now the question is - what is your repo type? Please describe it.

Thank you.

P.S. I'm not quite sure of what exactly's been consuming almost all of your RAM during first 10 minutes of merging. Are you sure this did not take place during incremental?

[HJAdams123]

Hey there Pavel....

My repositories are virtual machines on vsphere 5.5 update 2. Guest is Windows 2008 R2 64bit. Two vCPU, 10GB of Ram, 1 vmxnet3 adapter. Virtual machines is composed of two vmdk disks, a 40GB C: volume on a all Flash Pure Storage array. Second vmdk is 15TB volume on a EMC Clarrion AX-4. The D: volume is where backup data is held for the repository. Backend connectivity to the AX-4 is Fibre Channel 4Gbit.

[PTide]

Harold,

I think that for futher investigation you should contact technical support. I can't see anything what could be a reason for such poor performance (assuming .vib 200 Gb and 10 hours merging that would be aroung 5-6 Mb/s), based on the info you've kindly provided.

So please open a case with support and post number here.

Thank you.

[HJAdams123]

I just logged a case. Case # is 00956332.

-Harold

[HJAdams123]

You know, while I wait for support to get with me.... I remember reading in another related thread that this new caching feature can be turned off by setting the HKLM\Software\Veeam\Veeam Backup and Replication "AgentReadOnlyCache" to 0. This suggests to me this reg value should be there by default, but set to "1". I don't have the AgentReadOnlyCache in that location at all. Could this be my problem? Could someone verify on an installation of v8 Update2 there there is no AgentReadOnlyCache setting, or is there supposed to be one, only set to 1.

-Harold

[dellock6]

1 is the default value, so there's no need to have a registry key to define it, that's why the key is not there.
If you want to change the cache behaviour (but I would suggest to talk about this with support first since you already engaged it) you need to create the key first and then configure the desired value.

[HJAdams123]

Oh no, I don't want to turn it off. I just want it to work! I am just grasping at straws... I will wait to see what Support has to say...

-Harold

[mdiver]

Just to clarify:

Code: Select all

AgentReadOnlyCache = 0 -> disable caching
AgentReadOnlyCache = 1 -> enable caching (default value)
AgentReadOnlyCache = 2 -> disable caching for non-inline-dedup Jobs
AgentReadOnlyCache = 3 -> ??? (this was once suggested to us by a veeam support engineer - unfortunately I don't remember the case#)

So what would value "3" actually do?

Thanks, Mike

[Gostev]

Always enable in every scenario (including those where it does not make technical sense and/or dangerous to enable, such as jobs with dedupe disabled, jobs to ExaGrid appliances, etc).