CheckStorage failed client error: bad allocation

[Oletho]

Support case ID 5212734

I recently installed Veeam B&R 6.1 patch1 at a customer site.

The initial job of 124 VMs went ok and generated a ~5TB backup file. But every job after that fails immediately with the error message mentioned in subject.

Support suggested that it is a memory problem and proposed to cut the job in smaller portions. But I would prefer a solution, not a workaround, and very much doubt that the system is memory limited. Even if the Veeam servers are freshly rebooted and there are a handful free GBs of memory the job fails instantly.

An equally large replication job runs fine.

Any suggestions?

[dellock6]

What kind of design are you using? Physical servers, hot add with virtual proxies.... It's hard to say what's wrong or what could be improved based on your post.
Only, please remember backup and replication are different operations, so it's not a correct assumption saying that if replication works, so it should do also backups.

Luca.

[Oletho]

It is running appliance mode with a couple of proxies.

Removing 17 small VMs from the job got it running again, so it probably had something to do with numbers or size. Adding memory to Veeam servers made no difference.

We decided to cut the jobs in smaller parts, as that was the only offer from support.

[dellock6]

How many vCPUs has any proxy? Usually proxies are more cpu hungry than memory, after 2-4 gb per VM I never had to increase memory size on proxies.

[Oletho]

4 vCPUs per proxy (4.1 enterprise license). The job failed before CPU activity was rising.

The only reason for increasing memory was that support stated it was a memory limitation problem.

[Gostev]

I know clients who are running very big jobs (in terms of amount of VMs) without any issues. How big is your job?

Some things to check:
- Could it be bad memory on the proxy server?
- Could it be lack of memory on the backup repository server (if you have it separate).

I recommend you keep this case open until you have it resolved, you should not have to change your jobs to accommodate this issue.

[Oletho]

The job had 124 VMs of totally 15 TB. Backup file was around 5TB but not all VMs were backed up when we started seeing this error.

Involved proxy and repository had plenty available memory, as the job failed right after start. All Veeam servers are Windows Server 2008 R2 with latest SP and updates.

They were migrated just to try other hosts.

The case is closed, as the client wanted me to get on with the installation and not spend time arguing with support.

And support scared the shit out of my customer by stating that we had to have a successful job, otherwise we would not be able to restore any VM. I proved that wrong to the customer, but anyway