Synthetic full backup fail

[sasilik]

After installing Update 2 I started to see error messages on some jobs at weekend synthetic full backups. At first it sends failed job report and sometime after that job succeeds. Error message is:
Synthetic full backup creation failed Error: Unable to allocate memory for storage metadata bank.

Backup proxies are Windwos 2012 core servers, 4 cores and 4GB RAM and at first look there didn't seem to be with RAM. I can monitor it more at weekend. I also didn't find any similar error messages from forum or elsewhere. I haven't opened support case yet but if error cames again this weekend then I guess I am going to ask from VEEAM support for advice.

[Gostev]

What is your backup repository? Make sure the gateway server meets minimal system requirements for RAM (most likely, it does not).

[sasilik]

Backup repository is CIFS share on Data Domain. Gateway server has 8GB RAM. According to jobs history there was 9 synthetic full jobs running at this time when it failed. I think I am gonna add 1 or 2 GB RAM to gateway server.

[foggy]

According to system requirements, gateway server (backup repository server) should have at least "4 GB RAM plus 2 GB RAM (32-bit OS) or 4 GB RAM (64-bit OS) for each concurrent job", so you seem to kill it by assigning 9 concurrent jobs to it.

[sasilik]

Documentation, http://helpcenter.veeam.com/backup/80/v ... itory.html , also says - CIFS (SMB) share. SMB share cannot host Veeam transport services. For this reason, data to the SMB share is written from a gateway server. By default, this role performs a backup proxy that is used by the job for data transport.
We have backup proxy servers and proxy is disabled on VEEAM server. I am inclined to think that gateway server RAM requirements don't apply here so directly as system requirements say.
But as VEEAM server requirements are - 4 GB RAM plus 500 MB RAM for each concurrent job , then it probably is RAM problem. I added 2 GB RAM, 10GB total now, and I am going to monitor RAM usage at weekend.

[foggy]

If you run several roles on the same server, it should meet the combined system requirements.

[sasilik]

Our configuration is one VEEAM backup server, 64-bit and now with 10GB RAM, backup proxy part is disabled on server. Backup repository is CIFS share on Data Domain and then there are some additional proxy servers. As there is possibility that about 10 jobs run simultaneously is the server yet short about 34GB of RAM?

[foggy]

What server performs the gateway role for this CIFS share?

[sasilik]

"Automatic selection" is selected in Gateway server section.

[foggy]

Anyway, the server performing the role of the gateway should meet the "4 GB RAM plus 2 GB RAM (32-bit OS) or 4 GB RAM (64-bit OS) for each concurrent job" requirement. If the same gateway is used for all 10 concurrent jobs, it should have at least 44GB RAM to be able to execute them.

[sasilik]

If it is used... I laid out our VEEAM setup and what I understood reading documentation I don't see reason why VEEAM Backup server should be Gateway server for all these jobs if backup repository is CIFS share and there are other proxy servers around. I don't want to add that much RAM just because "If same gateway is used ..." for some obscure reason. Server is now about half a year in same setup and same number of jobs and I should have surely seen warnings and errors if there is such memory shortage.

[foggy]

Right, the role of the gateway in your case is performed by proxy servers, not Veeam B&R itself. If different proxies are picked up by different jobs, then the load is spread among them. Most likely at some point one of the proxies was assigned too many tasks.

[sasilik]

After the weekend I saw that one proxy server was short of memory. I can add more memory but is there easy way to see how many concurrent tasks (I guess these are tasks for writing vmdk's?) were running at some point of time?

[foggy]

I guess these are tasks for writing vmdk's?

Yep, task is a VM disk in terms of Veeam B&R. Each disk processing requires one proxy task slot and one on repository/gateway server (both roles are performed by the same server in your case).

I can add more memory but is there easy way to see how many concurrent tasks were running at some point of time?

Not sure the historical data are available. Veeam ONE allows to view the current number of tasks per proxy and track historical memory consumption on proxy servers. Anyway, you could just set the max number of concurrent tasks to a more appropriate for the resources each of the proxy servers has to avoid issues in future.

[sasilik]

CIFS repository has setting "Limit maximum concurrent tasks to: 8". There is 3 proxy servers and setting "Max concurrent tasks" is 4 for every proxy. Proxy servers have 4GB RAM. I am trying to figure out how system requirements apply here, to where and why one proxy gets RAM deficiency.
Job starts. Does backup server assigns job to proxy server (as it supposedly is also gateway server) and proxy server does all gateway server job and vmdks or assigns them elsewhere. Or backup server assigns vmdk writing tasks to proxy servers?
If first then I can imagine how it gets short of RAM, because it needs 2 GB for system and 4GB RAM per job and then some for tasks. Only it should have been already short of RAM from the beginning, not half a year later.
If second then proxy server requirements say 2GB for system and 200MB per concurrent task. If proxy has limited to 4 concurrent task then 4GB total RAM should be enough.
What I am missing here or what is wrong with my speculations above?

[foggy]

Proxy server is assigned both data processing and writing tasks (that are normally applied to the gateway server).

If first then I can imagine how it gets short of RAM, because it needs 2 GB for system and 4GB RAM per job and then some for tasks. Only it should have been already short of RAM from the beginning, not half a year later.

Metadata caching introduced in Update 2 increases RAM consumption on backup repository (or gateway server), which could result in the observed behavior.

[sasilik]

It also mentions that - Using the default "Automatic selection" gateway server setting for Shared folder, EMC DataDomain or HP StoreOnce based backup repositories disables the cache.
And - Disabling built-in deduplication disables the cache

I have shared folder and "Automatic selection" gateway server setting for Shared folder and we don't use inline deduplication as share is on Datadomain which does deduplication itself.

[sasilik]

I am still confused after I took a look at memory usage of veeam server and proxies. Memory usage for one proxy grows significally more when for others it doesn't grow almost at all. Below is Windows Commited bytes graphic for veeam server and three proxies over the weekend. I have not yet added RAM to proxy server but decreased concurrent tasks to 3 for each proxy and there was no errors in this weekend. I guess I am going to add some RAM now because I am going to vacation soon and I want backup jobs to run smoothly. But question stays, why only one proxy gets all load and does backup server decides at some moment to switch at another proxy and its RAM usage starts grow? Do I have to add more RAM to all proxy servers just in case? If situation stays same then maybe there is a case for support.

[foggy]

Indeed, you can involve our technical guys to review your setup more closely. I recommend to configure the max concurrent tasks limit in accordance with available resources, so that even in case the limit is reached, each proxy be still capable of the assigned load.

[sasilik]

Concurrent task limit is only 3 for every proxy. And if task is vmdk handling then during the period there should be about or more than 10 concurrent tasks. And about 5..7 concurrent jobs. What I am read about VEEAM's job and task handling logic in case of CIFS share and automatic gateway selection it just seems weird that one proxy RAM usage is so much different from others.

[sasilik]

Took a look at the time when jobs were ending and every point where there is larger drop of black line one synthetic full job ended. I must conclude from it that all jobs were assigned to first proxy and because of that the RAM usage is so different. And from where I do limit concurrent jobs, not tasks, per proxy?

[foggy]

Keep in mind, that repository tasks are counted separately from the tasks assigned to the proxy, so you need to make sure the total number of tasks that could be assigned to the server meets the requirements. Repository concurrent tasks limit is configured in repository settings.

Seems like the issue is caused by the fact that the gateway server is selected at the job start and used for all the repository tasks within this job, even if some of the data processing job tasks are actually processed by different proxy servers. That's why you're seeing high RAM consumption on one of the proxies - it is selected as gateway for several jobs.

[sasilik]

I figured that out, but why its only one when http://helpcenter.veeam.com/backup/80/v ... erver.html says:

Shared Folder
...
3. In the Gateway server section, specify settings for the gateway server:
If a network connection between the source datastore and the backup repository is fast, choose Automatic selection. In this case, Veeam Backup & Replication will automatically choose a Microsoft Windows server and use it as a gateway server. This can be any Microsoft Windows server added to the backup infrastructure: Microsoft Windows server used for file copying operations, backup proxy, Veeam backup server and so on.
The gateway server is picked at random and per job session: Veeam Backup & Replication may use one gateway server for one job session and another gateway server for another job session.

Yes, there is this "may use" phrase and it seems that it really "may" use but actually uses only one. And as I add RAM to this one proxy server what gives me assurance that VEEAM stays using this server and not chooses at some point to use another and I am yet short of RAM on another proxy server.

[foggy]

If you explicitly specify this server as gateway for the repository, this will assure you from other servers being selected.