Comprehensive data protection for all workloads
Post Reply
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Veeam Backup & Replication 8.0.0.917 Patch 1 uses more RAM ?

Post by albertwt »

People,

Can anyone here please sahre or clarify if Veeam Backup & Replication 8.0.0.917 (Patch 1) requires more RAM resources on the backup server than the previous VBR v 7.0 ?

Because my physical backup server with 24 CPU cores and 32 GB of RAM is running very slowly by itslef to the point that it failes the backup job & the other replication job itself.

See the below error message:
Task failed Error: Exception of type 'System.OutOfMemoryException' was thrown.

Cheers.

[Case # 00813620]
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by albertwt »

and here's another one:
Failed to preprocess target Error: Exception of type 'System.OutOfMemoryException' was thrown. Exception of type 'System.OutOfMemoryException' was thrown.

Failed to preprocess target Error: Timed out requesting agent port for client sessions. --tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe], arguments: [ -g"2500-5000" -i"{cd2f0979-9726-4458-8798-cb5b5a80d022}" -l"flush,C:\ProgramData\Veeam\Backup\zz__DR__Application__Servers__1/Agent.zz__DR__Application__Servers__1.Target.log" -f 25 --vddkLogLevel 1 --maxPendingBlocks 10000 -n -x"{292c2dad-cb1b-4072-bae2-f1a24ac384f6}"])

Failed to preprocess target Error: The paging file is too small for this operation to complete. --tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe], arguments: [ -g"2500-5000" -i"{142989ec-06ee-4d16-af5e-c67aa3b9f7f5}" -l"flush,C:\ProgramData\Veeam\Backup\zz__DR__Application__Servers__1/Agent.zz__DR__Application__Servers__1.Target.log" -f 25 --vddkLogLevel 1 --maxPendingBlocks 10000 -n -x"{f3d8449a-115f-48c9-89d1-cb490ee14cc1}"])

Failed to preprocess target Error: Timed out requesting agent port for client sessions. --tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe], arguments: [ -g"2500-5000" -i"{ee06f606-681e-48a3-8faa-07fba6687377}" -l"flush,C:\ProgramData\Veeam\Backup\zz__DR__Application__Servers__1/Agent.zz__DR__Application__Servers__1.Target.log" -f 25 --vddkLogLevel 1 --maxPendingBlocks 10000 -n -x"{1ec36bd6-f85e-4741-9398-0490c75f5903}"])
--
/* Veeam software enthusiast user & supporter ! */
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by emachabert »

You should open a support ticket.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by albertwt »

emachabert wrote:You should open a support ticket.
Yes, I did that already but no one seems to be able to help me other than suggesting the nonsense like:

1. Double or upgrade the RAM (not possible 32 GB is the maximum memory limit for 2008 R2 Std.)
2. Increase or modify the Backup window to be longer than what it was already set.

so this remains a mystery to me after updating to Patch Number 1.
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by foggy »

Those are not acceptable resolutions, please ask for the case escalation.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by albertwt »

foggy wrote:Those are not acceptable resolutions, please ask for the case escalation.
Yes, that's the thing. I've loged the call since two days ago but no escalation yet eventhough I've clicked on the Request for the Case update button daily.
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by foggy » 1 person likes this post

You can ask your engineer to escalate the case to a higher tier for further investigation.
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by emachabert »

Have you monitored what processes are eating all of your memory when the OOM occurs ? Using windows resources monitor, memory tab, for example.
Are you running a lot of jobs simultaneously ?
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by albertwt »

Yes, it is running but mostly in <idle> state.
usually there are just about 3 Backup copy running.

The memory used according to RAMMap software is showing that the top 4 used is by the .VBK and .VIB files.
--
/* Veeam software enthusiast user & supporter ! */
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by mdiver » 1 person likes this post

We have also observed increased memory consumption on two large backup copy jobs with 8.0.0.917.
These could be isolated down to MFT/NTFS caching of the involved VBK/VIB files using rammap (In task manager this memory will not be visible).

The same jobs ran fine for half a year under V7. In V8 we started with patch#1 right away. So I cannot say it V8 itself or the patch#1 was the source.
We did not find an easy solution for it. Support was involved (case #00727257).

The installation was a monolithic Veeam B&R server with SAN integration. So Veeam DB, proxies and repos all on this single system (2x8 core, 32GB RAM though).
Once we changed to a tiered installation with Veeam DB+service in a VM and now two proxy/repo systems the problems were gone. RAM usage is neglatable now, which is somehow strange as well:

The data being moved is the same. One of the proxy/repo combinations is the old veeam server.

Maybe consider splitting up your installation. A tiered installation runs smoother in my experience. Especially with copy jobs.
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by emachabert »

looks like a memory leak. Let us know if you have any feedback.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by albertwt » 1 person likes this post

mdiver wrote:We have also observed increased memory consumption on two large backup copy jobs with 8.0.0.917.
These could be isolated down to MFT/NTFS caching of the involved VBK/VIB files using rammap (In task manager this memory will not be visible).

The same jobs ran fine for half a year under V7. In V8 we started with patch#1 right away. So I cannot say it V8 itself or the patch#1 was the source.
We did not find an easy solution for it. Support was involved (case #00727257).

The installation was a monolithic Veeam B&R server with SAN integration. So Veeam DB, proxies and repos all on this single system (2x8 core, 32GB RAM though).
Once we changed to a tiered installation with Veeam DB+service in a VM and now two proxy/repo systems the problems were gone. RAM usage is neglatable now, which is somehow strange as well:

The data being moved is the same. One of the proxy/repo combinations is the old veeam server.

Maybe consider splitting up your installation. A tiered installation runs smoother in my experience. Especially with copy jobs.
Yes, I'm with you mate. (Case # 00813620)

My 24 pCPU core & 32 GB VBR server already assisted with 3x Backup repository server running as VM. But somehow it always causing problem with the Backup Copy job or the backup job.

The work around is to apply the following registry solution which seems to does the trick:

Code: Select all

1.Click Start, click Run, type regedit, and then click OK.
2.Locate and then click the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

3.On the Edit menu, point to New, and then click DWORD Value.
4.In the New Value #1 box, type PoolUsageMaximum, and then press ENTER.
5.Right-click PoolUsageMaximum, and then click Modify.
6.In the Value data box, type 60, click Decimal, and then click OK.
7.If the PagedPoolSize registry entry exists, go to step 8. If the PagedPoolSize registry entry does not exist, create it.

To do this, follow these steps:
•On the Edit menu, point to New, and then click DWORD Value.
•In the New Value #1 box, type PagedPoolSize, and then press ENTER.

8.Right-click PagedPoolSize, and then click Modify.
9.In the Value data box, type ffffffff, and then click OK.
10.Exit Registry Editor, and then restart the computer.
--
/* Veeam software enthusiast user & supporter ! */
CaptainFred
Enthusiast
Posts: 88
Liked: 2 times
Joined: Jul 31, 2013 12:05 pm
Full Name: Si
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by CaptainFred »

I've got the same problem and similar spec server, 24 pCPU core & 32 GB RAM. Does the above solution work for 2008 R2 SP1 Standard?
CaptainFred
Enthusiast
Posts: 88
Liked: 2 times
Joined: Jul 31, 2013 12:05 pm
Full Name: Si
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by CaptainFred »

Please can someone explain what those settings above do? And should they be different depending on machine spec?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by foggy » 1 person likes this post

These settings control memory usage on the affected machine. I recommend contacting support to verify your setup and ensure these are the steps to perform in your particular case.
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by SyNtAxx »

I have same issue "System.OutOfMemoryException" on my virtual Veeam server. It is just a management server and does no proxy services. 8vCPU and 16GB ram. It fails with this error when I attempt to use my 3par integration. I am starting to lose faith in the product. I have had more cases opened then I wish to admit, several went to tier3 and private patched. One is currently in tier3. Another case I opened the tech support told me "FIPS error means it doesn't work". No kidding ! WHY?

I'm about to request some sort of duty manager. This is ridiculous. I'm sure i'm not the biggest shop out there running veeam, but with an investment in 100+ Enterprise Plus licenses and crashes/bugs/features not working im in a bad spot.

Syntaxx
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Backup & Replication 8.0.0.917 Patch 1 uses more R

Post by Gostev »

Are all of your issues being addressed by our support, or you have some outstanding one that are problematic? I can ask support management to review those. You can also request this from support management directly by clicking "Talk to a Manager" on the Customer Support Portal.

Based on what you are saying, our support has been delivering multiple patches promptly, which is the most important thing. Every environment is special, so of course there will be issues. Just take any other software vendor you are using in your production environment, and look at the size of their periodic patch notes...

I think the most important thing is that we address any issues you may be having promptly. We often hear this is what sets Veeam aside, and we want to make sure we keep it up. So, we do want you to let us know of any stuck or poorly handled support cases.

Thanks!
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 229 guests