Large file server backup failing

[dpgator333]

support case 02688961

I am a new Veeam admin (I've used trial and free before) and am just now getting up and running. We have a 5-host VMWare environment, with a HP 3Par SAN for storage. All backups use storage backed snapshots for backup. We have 18 servers monitored/backed up, and the vast majority of them are 100GB or less. 15 of them are in a single backup job that runs like a champ, incrementals well under an hours, no failures, etc. Another job backs up our two SQL servers, which are 5TB and 6TB respectively. These jobs also run without any issue.


Then we have our file server. It's currently across 2 datastores, one for the system disk on SSD disks, the "data" drive on SATA drives. The job keeps failing with a "Insufficient memory to continue the execution of the program" message. I've calculated the amount of memory needed, even if running all jobs at one time, and all components installed on the same server (they are), and I shouldn't need more then 20-25GB at most. I've thrown as much as 200GB of RAM on the backup server, and it just takes longer to fail with more memory. VMWare support has wiped their hands of it, and I believe them. The support guy I spoke with showed and explained their logs to me and it made sense - they receive a request to snapshot the server, and Veeam is supposed to then request a storage snap from the SAN. That does not happen. The SAN doesn't give any indication that it receives a request either. With other jobs, I can clearly see that within a minute or so of a job starting, there is a VMWare snapshot, followed by a 3Par snapshot, then VMWare deletes its snapshot and the storage snapshot remains until job completion.

Veeam support has indicated this seems to be a problem with the storage, but I've not seen any actual or anecdotal evidence that would say why. I don't disagree, I just wish there was something concrete in some documentation that says what this error might mean as far as the storage is concerned.

The Veeam "Integration with Storage Systems" documentation does state that "Backup from Storage Snapshots cannot be used for VMs whose disks are located on VVol datastores" - I can only assume that VVol means "virtual volume", but there is no reference to that in the document. Also, in 3Par parlance, all volumes that are presented to VMWare hosts are "Virtual Volumes", so if that were the case, then none of my backups would work. Or I guess 3Par storage wouldn't work at all. So I'm not sure what that means exactly.

other nuggets...
- the vmdk is provisioned as thin, but I don't think that should matter. Other disks that are thin back up fine.
- the SAN has a max volume size of 16TB, so the datastore in VMWare has been extended repeatedly in order to expand to the size we needed. This is kind of an unknown in terms of potential causes, as no other vmdk/datastore covers more than a single volume on the SAN. But I've no confirmation on that.
- single backup repository is a Qnap NAS/SAN, RAID6 90TB usable volume - connection is 10GB, connected to same switch. the VM virtual switch is actually 10Gbx2 physical ports (teamed) and the SAN is 10Gbx4 teamed. Throughput is pretty darn good, no latency issues.

Anyone else have similar/same issues to this? My understanding is that the sheer size of the server/disk should not matter by itself, so I'm pretty sure this is possible. Just trying to figure out why. In the meantime, we are doing a robocopy of all of the shared folders onto some extra space on our backup disk.

[foggy]

Dan, you should escalate the case to a higher tier for a closer investigation. We need to check memory allocation during the job operation.