Veeam Backup & Replication 8.0.0.917 Patch 1 uses more RAM ?

[albertwt]

People,

Can anyone here please sahre or clarify if Veeam Backup & Replication 8.0.0.917 (Patch 1) requires more RAM resources on the backup server than the previous VBR v 7.0 ?

Because my physical backup server with 24 CPU cores and 32 GB of RAM is running very slowly by itslef to the point that it failes the backup job & the other replication job itself.

See the below error message:

Task failed Error: Exception of type 'System.OutOfMemoryException' was thrown.

Cheers.

[Case # 00813620]

[albertwt]

and here's another one:

Failed to preprocess target Error: Exception of type 'System.OutOfMemoryException' was thrown. Exception of type 'System.OutOfMemoryException' was thrown.

Failed to preprocess target Error: Timed out requesting agent port for client sessions. --tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe], arguments: [ -g"2500-5000" -i"{cd2f0979-9726-4458-8798-cb5b5a80d022}" -l"flush,C:\ProgramData\Veeam\Backup\zz__DR__Application__Servers__1/Agent.zz__DR__Application__Servers__1.Target.log" -f 25 --vddkLogLevel 1 --maxPendingBlocks 10000 -n -x"{292c2dad-cb1b-4072-bae2-f1a24ac384f6}"])

Failed to preprocess target Error: The paging file is too small for this operation to complete. --tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe], arguments: [ -g"2500-5000" -i"{142989ec-06ee-4d16-af5e-c67aa3b9f7f5}" -l"flush,C:\ProgramData\Veeam\Backup\zz__DR__Application__Servers__1/Agent.zz__DR__Application__Servers__1.Target.log" -f 25 --vddkLogLevel 1 --maxPendingBlocks 10000 -n -x"{f3d8449a-115f-48c9-89d1-cb490ee14cc1}"])

Failed to preprocess target Error: Timed out requesting agent port for client sessions. --tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe], arguments: [ -g"2500-5000" -i"{ee06f606-681e-48a3-8faa-07fba6687377}" -l"flush,C:\ProgramData\Veeam\Backup\zz__DR__Application__Servers__1/Agent.zz__DR__Application__Servers__1.Target.log" -f 25 --vddkLogLevel 1 --maxPendingBlocks 10000 -n -x"{1ec36bd6-f85e-4741-9398-0490c75f5903}"])

[emachabert]

You should open a support ticket.

[albertwt]

You should open a support ticket.

Yes, I did that already but no one seems to be able to help me other than suggesting the nonsense like:

1. Double or upgrade the RAM (not possible 32 GB is the maximum memory limit for 2008 R2 Std.)
2. Increase or modify the Backup window to be longer than what it was already set.

so this remains a mystery to me after updating to Patch Number 1.

[foggy]

Those are not acceptable resolutions, please ask for the case escalation.

[albertwt]

Those are not acceptable resolutions, please ask for the case escalation.

Yes, that's the thing. I've loged the call since two days ago but no escalation yet eventhough I've clicked on the Request for the Case update button daily.

[foggy]

You can ask your engineer to escalate the case to a higher tier for further investigation.

[emachabert]

Have you monitored what processes are eating all of your memory when the OOM occurs ? Using windows resources monitor, memory tab, for example.
Are you running a lot of jobs simultaneously ?

[albertwt]

Yes, it is running but mostly in <idle> state.
usually there are just about 3 Backup copy running.

The memory used according to RAMMap software is showing that the top 4 used is by the .VBK and .VIB files.

[mdiver]

We have also observed increased memory consumption on two large backup copy jobs with 8.0.0.917.
These could be isolated down to MFT/NTFS caching of the involved VBK/VIB files using rammap (In task manager this memory will not be visible).

The same jobs ran fine for half a year under V7. In V8 we started with patch#1 right away. So I cannot say it V8 itself or the patch#1 was the source.
We did not find an easy solution for it. Support was involved (case #00727257).

The installation was a monolithic Veeam B&R server with SAN integration. So Veeam DB, proxies and repos all on this single system (2x8 core, 32GB RAM though).
Once we changed to a tiered installation with Veeam DB+service in a VM and now two proxy/repo systems the problems were gone. RAM usage is neglatable now, which is somehow strange as well:

The data being moved is the same. One of the proxy/repo combinations is the old veeam server.

Maybe consider splitting up your installation. A tiered installation runs smoother in my experience. Especially with copy jobs.

[emachabert]

looks like a memory leak. Let us know if you have any feedback.

[albertwt]

We have also observed increased memory consumption on two large backup copy jobs with 8.0.0.917.
These could be isolated down to MFT/NTFS caching of the involved VBK/VIB files using rammap (In task manager this memory will not be visible).

The same jobs ran fine for half a year under V7. In V8 we started with patch#1 right away. So I cannot say it V8 itself or the patch#1 was the source.
We did not find an easy solution for it. Support was involved (case #00727257).

The installation was a monolithic Veeam B&R server with SAN integration. So Veeam DB, proxies and repos all on this single system (2x8 core, 32GB RAM though).
Once we changed to a tiered installation with Veeam DB+service in a VM and now two proxy/repo systems the problems were gone. RAM usage is neglatable now, which is somehow strange as well:

The data being moved is the same. One of the proxy/repo combinations is the old veeam server.

Maybe consider splitting up your installation. A tiered installation runs smoother in my experience. Especially with copy jobs.

Yes, I'm with you mate. (Case # 00813620)

My 24 pCPU core & 32 GB VBR server already assisted with 3x Backup repository server running as VM. But somehow it always causing problem with the Backup Copy job or the backup job.

The work around is to apply the following registry solution which seems to does the trick:

Code: Select all

1.Click Start, click Run, type regedit, and then click OK.
2.Locate and then click the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

3.On the Edit menu, point to New, and then click DWORD Value.
4.In the New Value #1 box, type PoolUsageMaximum, and then press ENTER.
5.Right-click PoolUsageMaximum, and then click Modify.
6.In the Value data box, type 60, click Decimal, and then click OK.
7.If the PagedPoolSize registry entry exists, go to step 8. If the PagedPoolSize registry entry does not exist, create it.

To do this, follow these steps:
•On the Edit menu, point to New, and then click DWORD Value.
•In the New Value #1 box, type PagedPoolSize, and then press ENTER.

8.Right-click PagedPoolSize, and then click Modify.
9.In the Value data box, type ffffffff, and then click OK.
10.Exit Registry Editor, and then restart the computer.

[CaptainFred]

I've got the same problem and similar spec server, 24 pCPU core & 32 GB RAM. Does the above solution work for 2008 R2 SP1 Standard?

[CaptainFred]

Please can someone explain what those settings above do? And should they be different depending on machine spec?

[foggy]

These settings control memory usage on the affected machine. I recommend contacting support to verify your setup and ensure these are the steps to perform in your particular case.

[SyNtAxx]

I have same issue "System.OutOfMemoryException" on my virtual Veeam server. It is just a management server and does no proxy services. 8vCPU and 16GB ram. It fails with this error when I attempt to use my 3par integration. I am starting to lose faith in the product. I have had more cases opened then I wish to admit, several went to tier3 and private patched. One is currently in tier3. Another case I opened the tech support told me "FIPS error means it doesn't work". No kidding ! WHY?

I'm about to request some sort of duty manager. This is ridiculous. I'm sure i'm not the biggest shop out there running veeam, but with an investment in 100+ Enterprise Plus licenses and crashes/bugs/features not working im in a bad spot.

Syntaxx

[Gostev]

Are all of your issues being addressed by our support, or you have some outstanding one that are problematic? I can ask support management to review those. You can also request this from support management directly by clicking "Talk to a Manager" on the Customer Support Portal.

Based on what you are saying, our support has been delivering multiple patches promptly, which is the most important thing. Every environment is special, so of course there will be issues. Just take any other software vendor you are using in your production environment, and look at the size of their periodic patch notes...

I think the most important thing is that we address any issues you may be having promptly. We often hear this is what sets Veeam aside, and we want to make sure we keep it up. So, we do want you to let us know of any stuck or poorly handled support cases.

Thanks!