No warning when backup from storage snapshot fails

[FrancWest]

Hi,

case #03814828, yesterday due to a network issue VBR temporarily lost it's connection to the storage-array during boot-up. After that it appeared that backup from storage snapshots was not working anymore. It failed over to backup using VM snapshots. Only after doing a manual storage rescan backup from storage snapshot functionality worked again.

Two issues here:
- why didn't Veeam automatically enable backup from storage snapshots again after it detected that the storage was available again? Even after a reboot, the backup from storage snapshot functionality didn't work until I did a manual storage rescan.
- When backup from storage snapshot fails and fails over to VM snapshots, there's no warning generated in the jobs. A big problem here is that you don’t notice it right away during backup jobs, since no warning is raised. You only see this one line that a fail over to VMWare snapshot has occurred, but this line even has a green check-mark. To notice this would mean that you have to go through the backup results in detail to see this one line and normally we don’t do that since we only take action if we see ‘warning’ or ‘error’ in the subject of the notification mail and we also have an outlook rule present that separates the 'successful' jobs from the 'failed' or 'warning' jobs.

Frank.

[ejenner]

If it's something you're worried about you have the option to do what I do and set your backups so they will only work with your preferred configuration.

I accept if my chosen configuration does not work then where I've prevented the failover option the whole backup is going to fail.

I've noticed in some cases where a failsafe option does exist it will generate errors or warnings in the log as it does the backup although the job itself will be marked as successfully completed.

I believe my major notification email I receive daily actually counts the errors and warnings... so even if I've had 75 errors / warnings overnight it's fairly likely I'll check the console and find everything has completed successfully. i.e. if a piece of infrastructure was unavailable during the backup schedule and no backups ran on time a lot of errors will be generated in the notification email but none will be reported on the console as the jobs did all successfully complete.

It's one of those areas where personal preference means everyone expects a different response to events which occur during backup. Some people only want to know if it didn't run, so how do you please those guys and the people who want a warning for everything at the same time?

[foggy]

EJ is spot on. Everyone is paranoid in his own way and expects different behavior for notifications. Anyway, the first issue would have been fixed by the periodic automatic infrastructure rescan. As for the warning, then I actually see one in case storage snapshot cannot be created and the job fails over to a VMware snapshot. What message do you see in the job session log? Is it marked as a green check mark as other informational messages?

[FrancWest]

Hi, sure, everyone has different expectations, no problem. But as I understand the periodic scan runs once in 7 days, so that would take a while before it activates backup from storage snapshot again.

Yes, the failover message is marked green. You first have to click on the VM, then you see the line:

'Failed to prepare VM for processing from storage snapshot, failing over to backup from VM snapshot'

This line has a green checkmark in front, in my opinion this should be a yellow warning.

[foggy]

Infrastructure rescan is performed every 4 hours. 7 days is the interval for volumes rescan (unless new snapshots are detected).

Please give me some time for the message investigation, I will get back once I have any news.

[foggy]

Ok, your case is a bit different, failover occurred at the moment of checking available proxy servers and there were none. Nothing has actually failed (unlike my scenario mentioned above, when snapshot creation was unsuccessful) hence the informational status of the message.

[FrancWest]

The issue is still under investigation. Everytime I reboot the backup server, backup from storage snapshots fail. However, I don't quite your explanation: you state nothing has actually failed, but when I enable use storage snapshots first and then fail over to normal snapshots, shouldn't it at least be a warning so that I can take action? For some reason Veeam fails to detect an available proxy, but in fact it should be there (it's the veeam server itself). That surely sounds like there's an issue and it should raise at least a warning

[veremin]

A backup server not finding applicable proxy is an issue indeed, and should be investigated further (which is exactly what our engineers are currently doing). Meanwhile, backup server failing over from one backup method to the other is not a problem, as long as a backup is created, thus, informational status of the message. Thanks!

[foggy]

There's a difference in failover reason here. In your case, when a proxy is not available, the backup from storage snapshot isn't even started, hence it is not a backup attempt failure, just selection of this or that transport mode for future backup. In the case that I've mentioned initially, backup from storage snapshot was attempted and then storage snapshot creation failed so this is a backup attempt failure that is logged as a warning in the job session (not a backup job failure though since subsequent failover to regular backup was successful).

[FrancWest]

Ok, clear, so the only way to be sure that storage snapshots are being used and nothing else would be to change the job settings to not fallback. Else I would never know when here was some sort of issue with the proxy without going through the successful job details.

[foggy]

That's correct.