The case of the "missing" VBK file...

[ngagne]

Support Case # 01087565
Veeam B&R v8.0.0.2084

I've had this happen a few times now, and really just started to dig into it. The basic issue is that the local SQL instance with the Veeam database crashes occasionally during a backup job. It crashes because of a low virtual memory condition that *may* be caused by 3 backup jobs starting at the exact same time (we let Veeam's resource scheduler handle job concurrency). The Veeam server is a VM with 8GB RAM allocated; the default paging file size of ~2500MB is also untouched.

But that's not why I'm here.

Seemingly (and I do have my jump to conclusions mat out!), when the above occurs, it can cause Veeam to leave things in a state of limbo. These are all reversed incremental jobs, and it looks like one of the first things the job does is rename the VBK file...and then SQL craps out. What's left is a situation where what I suspect happens is that the VBM file doesn't get updated with the new filename. The next time the job runs, we get the following error:

Error: Storage file \\<path_to_last_good (i.e. before rename).vbk is missing from host <proxyIP>

1.) Has anyone seen this before?
2.) I suspect now that the backup chain is worthless (major problem!)
2.) I know that I need to resolve the low virtual memory condition, but I'm more interested in making sure that if this is, in fact a Veeam issue, that it gets the attention that it deserves. If it's something that I'm doing incorrectly, I'm all ears as well - I just want to make sure that whatever it is, is fixed!

[Gostev]

Hi,

Assuming you have "all-in-one" Veeam server (also carrying backup proxy and backup repository roles), with 3 concurrent jobs you are not meeting minimal System Requirements for RAM by a mile.

Backup Repository role alone requires well over 8GB RAM with 3 concurrent jobs, not even considering requirements by other Veeam components and actual SQL Server. So, this will be the first issue for you to fix, as we cannot support a product deployed in an unsupported fashion.

You can either increase RAM allocation for your backup server VM, or - as a quick fix - just reduce concurrent jobs count to 1 (although this will still leave you slightly below minimal amount of RAM required, you should be OK).

Thanks!

[ngagne]

Nope - actually, we have two other proxies and three separate CIFS repos (none directly connected to the Veeam server). Again, I can remedy the low virtual memory thing. I'm more interested in figuring out why Veeam renames a file and then can't find it.

edit: and max concurrent tasks for the "built-in" proxy on the Veeam server is set to 2.

[Gostev]

three separate CIFS repos (none directly connected to the Veeam server)

This means that potentially, Veeam server can still be auto-picked as the gateway server for those, thus running the target data mover, which effectively makes it a Backup Repository server for those CIFS repos.

[Gostev]

Nevertheless, I do recommend you remedy the low virtual memory thing first, as reliable backend database functioning is the key. There's no sense to pursue this particular file renaming issue, because unreliable database will result in hundreds of issues in other places too. In the end, this database is the "brains" of the entire solution...

[ngagne]

Except that's not the case here - the target repo for this job has a different gateway assigned to it. I was hoping to provide you with feedback on a way that the product could potentially be improved and made more resilient. If you don't feel it's worthwhile to pursue, I'm disappointed but understand

[Gostev]

Well, achieving resiliency in a situation when the backend database cannot be trusted data can only be done by implementing our own database engine that CAN be trusted data even through unsupported conditions and crashes (if this is even possible to achieve). But, I am not sure you want us to switch from backup business to database business

Nevertheless, we do want to fully investigate this issue to ensure that it is in fact caused my malfunctioning SQL Server database (as opposed to some bug in our code), so thank you for opening a support case!

[ngagne]

Thanks, that's all I'm asking for!