no vm names in alarm

[bkain1]

We have data stores, in our VBRs, for each job, not vcenters, so you'll have one job that runs over a set of vms (which also come and go daily). The email I get from the VBR lists each VM and either success for the backup, again, by vm, but the information is not passed to V1, it seems. Instead, I get an alarm that looks like this:

Code: Select all

Database log backup job state-PROD_VBR
Job "vbr1 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with error
Collected SQL Server transaction logs do not match any existing database backup: CNPS;model
Job "vbr1 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 84 intervals
Job "vbr1 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 89 intervals

this is not giving me any information on which vm has which problem and when I opened ticket 04817157, I was told this is working as expected and Veeam expects someone, if there is any problem with any vm, to log into the VBR and look at it. While I agree that is what a person would have to do , to solve the issue, the issues of vms not backing up often clear themselves and we are trying to write files, for our ticketing system, by vm and by issue, and we can't, with this output.

Why, if the VBR has the vm name, is it not passed to V1 and is there any way around this? If not, we might have to scrap V1 completely and manually write our own monitoring directly from the VBRs

thanks

becki kain

[wishr]

Hi Becki,

we are trying to write files, for our ticketing system, by vm

Currently, this is not possible with most of the VBR job-related alarms.

Generally peaking, most of our job-related alarms including database log backup job state alarm work on the level of the job, regardless of being assigned to a backup server. This means that if there is an issue with a job you'll be informed on what job is it, where it is running, what happened, etc. Additionally, as a part of the description, we usually show an error message that comes directly from VBR to simplify the investigation and highlight what exactly has happened. You just posted this message above. This way, you are always notified if an issue happens and if an issue has happened, can generate some reports to dig into it.

If you need to be sure that a database has been processed successfully or would like to dig into an issue once an alarm arrived, you should use reporting capabilities, such as Database Protection History if we are speaking about the database log backups, for example.
In the same way, if we are speaking about generic per-workload reporting, you may use VM & Computer Protection History report that, by the way, also supports database log backup jobs.

Thanks

[bkain1]

but it doesn't list what vm's had issues so I can't use the alarm at all. I will look into the other two suggestions

[bkain1]

those are both reports. How do I write the data out, automatically, every 15 minutes to so, to something that can be fed to a ticketing system?

[wishr]

You should still be redirecting alarms to the ticketing system, but if an issue happens it should be investigated and resolved with the help of our reports and VBR UI. VONE alarms let you know what job has en issue, at what VBR server it is running, and information about the issue so you could start the investigation by assigning a ticket to a backup person.

Let's take a look at database log backups. In VBR you get an email with the details stats on the database level only once the task session interval ends. At the end of the interval, you get accumulated information on what databases have been protected and which have not. This information is only available at the end of the interval, thus is not suitable for alarms due to it's nature. That's why we have this data available in VONE in the form of reports and not in a form of an alarm.

Another reason is there are various situations when a job may fail due to side factors, such as, for example, loss of a proxy server or connectivity issue. In this case, there is no indication of what particular workloads are affected. The infrastructure itself, and thus the backup job is affected. Moreover, if you, let's say, add a container to a job (a resource pool or a datastore), the list of affected VMs may change during the presence of the connectivity issue and since there is no connectivity we will not even know what VMs are affected.

In some scenarios, you will get the name of the affected workload in the alarm error text description coming from VBR, but it all depends on what error message VBR writes to the database. If there an indication of the VM name - we will show it in the alarm description in the current VONE version.

I hope it makes sense. Thanks.

[bkain1]

I don't see how I can redirect it. the alarm is "one or more unnamed vms broke in this vbr, this job", which is not what the ticketing system is expecting, it's expecting machine names. How can I automatically write those reports, to a file, from v1, since they have vm names in them?

if not, we're going to have to scrap everything and write our own monitoring.

[wishr]

You may schedule any Veeam ONE reports to be delivered via email or to a specific folder.

[bkain1]

that's great, thanks. as i've said to the local support, i never get a chance to crack the manuals because my schedule is made by other people

[wishr]

I've added a few additional lines to my "large" post above. Hopefully, it clears things up a little bit.

Please note that we also provide a "VM with no backup" and a "Computer with no backup" alarms, that may help you to keep an eye on the RPO. E.g. if a certain VM does not have a backup during the last X days, it will trigger the alarm. I think this is something that may help with your usage scenario.

ticketing system is expecting, it's expecting machine names.

Don't get me wrong, but this sounds strange to me since we are discussing backup and backup infrastructure issues and not the VM issues. VM issues are a part of hypervisor monitoring. E.g. if your VM is experiencing an issue, not related to backup, such as lack of resources and so on, of course, we will raise an alert on this particular VM and highlight its name and what's wrong with it, as a part of vSphere/Hyper-V monitoring, but it's a different story.

By the way, just curious, what was the root cause of the issue described in your initial post? Maybe we should think about it from a different angle.

Thanks

[bkain1]

we can't enumerate vm's in our veeam infrastructure. They come and go all the time and there is no notice given to us. We also cannot add in whole vcenters because we don't have enough licenses for any whole ones, yet and not all of the vcenter may be under veeam control because they are not broken apart by functionality of "only sql" or "only oracle". And whole vcenters come and go, though not as often as vm's, with no notice to us.

The root cause is an alarm of transaction logs not getting backed up, and again, we have jobs by datastores, not by vm's and not by vcenters. We can see, in the vbr, what machines it is, but not in V1. I'm supposed to write the 6 variables (since %7 is not useful) to a directory then suppress writing more, for the remaining rolling 24 hours, depending on the severity of the issue, then the file gets moved to a directory to be picked up by the existing ticketing infrastructure. Without the vm name, i have no way of writing something useful, nor suppressing the files. also, i am not allowed to connect v1 to any vcenter because "it will cause too much overhead". we cannot have vmware tags, for the same reason.

I will look into the canned reports and if you can think of something else, that would be much appreciated. Our local support people agree that the functionality I'm tripping on is as designed, it's not a mistake that I made

[wishr]

I mean, what was the root cause for the logs not being backed up?

Also, could you please provide us with a screenshot from VONE Client UI showcasing the whole alarm information: from description down to the error message that came from VBR?

Thanks

[bkain1]

Code: Select all

12:36:13 PM Unable to release guest. Error: VSSControl: Failed to freeze guest over network, wait timeout
12:36:15 PM Creating VM snapshot 00:00
12:36:15 PM Failed to prepare VM: VSSControl: Failed to freeze guest over network, wait timeout

[bkain1]

v1 looks like this::
Alarm: Database log backup job state-PROD_VBR
Status: Error

Previous status: Error
Time: 5/21/2021 12:40:13 PM

Code: Select all

Details:	Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with error
Collected SQL Server transaction logs do not match any existing database backup: SATEST_CDC;model
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with error
Collected SQL Server transaction logs do not match any existing database backup: EAST;model
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with error
Collected SQL Server transaction logs do not match any existing database backup: model;SATEST_CDC
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with error
Collected SQL Server transaction logs do not match any existing database backup: PBOCV2QA;model
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with error
Collected SQL Server transaction logs do not match any existing database backup: model;CNPS
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 84 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 89 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 44 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 44 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 44 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 76 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 89 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 89 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 76 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 88 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 85 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Transaction logs from some databases were not backed up for the last 88 intervals
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Enumerating SQL Server databases
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Collected SQL Server transaction logs do not match any existing database backup: SATEST_CDC;model
Job "ks1_sh8 SQL Snapshotting Job SQL Server Transaction Log Backup" finished with warning
Collected SQL Server transaction logs do not match any existing database backup: model;SATEST_CDC

[wishr]

Hi Becki,

Thanks for the details.

A few additional questions:

1. What was the root cause for the below issue and how you resolved it?

Code: Select all

12:36:13 PM Unable to release guest. Error: VSSControl: Failed to freeze guest over network, wait timeout
12:36:15 PM Creating VM snapshot 00:00
12:36:15 PM Failed to prepare VM: VSSControl: Failed to freeze guest over network, wait timeout

2. Would it work if we add VM name to the Veeam ONE's alarm description of the particular "Database log backup job state alarm" in the future versions? I.e. if something has failed for a particular VM you will get a separate text for each VM and a dedicated error message that comes from VBR in the alarm description. This way you will be able to pass that data to the script using the %3 variable.

Thanks

[bkain1]

1. No idea what the root cause is. I'm not the vbr person and i just keep retying the job until it finishes. 6 of them failed again and have been just hitting retry all day, every day until the job finally works.
2. Yes, it would help greatly
3. What alarms fall into this catagory so we know which ones we may need to pull directly from each of the vbrs?

[wishr]

Thanks. What category you are talking about?

[bkain1]

you stated "Currently, this is not possible with most of the VBR job-related alarms.". i was curious what that list of alarms are so we know what we might have to pull from the vbrs but I still haven't looked into if we can use the reports (and dump to a NAS drive for ticketing pickup) since I've been dealing with that broken VBR

[wishr]

Yes, most of the job state alarms work at the job level. You can find them in the Alarm Management section of the Client UI by searching for the "job state" keyword.

Keep in mind that usually, these alarms contain an error text in plain text that comes directly from VBR and may contain the name of the processed workload, but again it depends on a concrete situation, job type, and the reason for the failure. A job may fail even before any job task session/job session starts its activities and in such and some other cases (see above) the name of the affected workload cannot be retrieved. Luckily, this is not the case with the Database log backup jobs since there an individual task session is created for each workload so we are able to get this info added to the alarm description.

[bkain1]

okay, thank you

[bkain1]

Wishr, any idea when this will be added into the V1 software? We have to decide how much effort to put into reproducing the functionality, directly from the VBRs

[wishr]

Hi Becki,

What exactly you are talking about?

Thanks

[bkain1]

We cannot use a lot of V1, because of this issue and may need to write, from the VBR's, instead. you said "Would it work if we add VM name to the Veeam ONE's alarm description of the particular "Database log backup job state alarm" in the future versions? I.e. if something has failed for a particular VM you will get a separate text for each VM and a dedicated error message that comes from VBR in the alarm description. This way you will be able to pass that data to the script using the %3 variable."

I was asking when you think this would be in V11 so we know how much effort we should put into creating this functionality, on our own. thanks

[wishr]

For this particular alarm, we will add the VM name to the description in the upcoming versions. If something is missing from any other alarm, this should be discussed separately and individually (please refer to my posts above for more info).

Thanks

[bkain1]

i will, as i trip on issues. thanks