VeeamONE reports storage connection failure after guest file restore

[ChrisGundry]

Hi

After a guest file restore (or others as well) via Veeam B&R finishes, the B&R process then removes the storage from the ESXi host and rescans the host. However, when the storage is removed and before the host is rescanned, the host reports the storage as down and VeeamONE then reports 'storage connection failure'. When our service desk are doing lots of restores or we are automating tasks, this results on a lot of VeeamONE alerts for the same.

Are we doing something wrong, or is this an oversight in Veeam B&R/ONE somewhere?

Thanks

[wishr]

Hi Chris,

If it's happening on the host itself there is not much VONE can do with it as we are pulling the data via vSphere APIs.

Regarding VBR behavior, I can't judge yet if it's normal or not. This should be clarified with our technical support team directly as we do not know any details about your environment and actions leading to the described behavior.

Thanks

[ChrisGundry]

Yes, if it appears on the host itself it will appear on VeeamONE, I agree, but that is the issue...

The behavior of the B&R is going to be the same for all customers, the method by which the restore process happens is not specific to my environment, it is a function of the Veeam software.

I posted here because when I asked in a historical Veeam support case (ages ago so I don't have the # anymore) they said to post it on the forum...

I was hoping that someone (Veeam staff or otherwise) would be able to confirm if they see this behavior. It seems that VeeamONE is still not that widely used!

I guess I will have to raise another support case for this specific issue...

[wishr]

Well, all feature requests should go here to the forums and not to the technical support team. We are always happy to discuss and address them in future versions of VONE. However, the behavior your host is experiencing does not look normal to me and VBR QC team - I just had a talk about this with one of their team members. That's why we should act step-by-step and first of all confirm whether it's a bug in VBR, some specifics of the environment, or get a confirmation that this is alright in your particular case. Based on the result, we can easily define the next steps. This is the primary reason why I'm suggesting opening a case with VBR technical support team and first of all, clarify that point.

As you may know, we already have a setting in the alarms allowing you to suppress an alarm during backup activities. Currently, this setting only covers backup activities and not restores. Based on the outcome of the investigation, if there is still a need to add restores to the coverage of the aforementioned setting (or add a separate setting), we will be able to consider it as a feature request.

Thanks

[ChrisGundry]

I logged this with support Case # 04923278.
They have said this is 'by design'. I absolutely hate that line. So you have designed your product to generate errors when being used normally, that isn't very good!
They have pointed me to https://www.veeam.com/kb2433, which is only for backups, not restores. They have said that the restore logic doesn't integrate with VeeamONE and the workaround is to edit the alarms and use the "Scheduled alarm suppression" feature each time I want to do a restore. Firstly, that is just plain silly! Second, given that I am not the one doing the restores, and the users who do the restores don't have access to VeeamONE settings, this is not viable.

So I have now been asked to submit a feature request to make restores from snapshots work without generating storage failure errors... So?

[wishr]

Hi Chris,

Have you got any comments from VBR support regarding the behavior of the host itself? The point is that it may not be normal, per the VBR QC team, as mentioned above. If this is considered to be expected, we will be able to start planning enhancements on the VONE side.

Update:
I've just spoken to the VBR QC team responsible for storage snapshots. Btw, thanks for mentioning you are using storage snapshots - this completely changes the situation. The datastore VBR attaches to the host during guest file restores is definitely being disconnected after the restore completion and this causes the VONE alarm to trigger, which is expected as support for storage snapshots is not something VONE does currently, so we will threat it as a feature request for future VONE versions.

Thanks

[ChrisGundry]

I sent screenshots of the storage processes on the host when the restore happens, which clearly shows the issue. They asked for Veeam logs, I asked them to confirm which logs because the logs they asked for were for restores from backups, not restores from SS. They then said to not bother because they were transferring the case to VeeamONE team because it was a known issue and by design. The VeeamONE contact then said it was by design.
You should be able to see all that on the case notes, no?

I don't see how this is normal either, but there is no other way to configure it as far as I can see and I have been working with Veeam for a very long time, so I would be surprised if there was another way to do this... Veeam must have lab capabilities that outstrip ours, sure you have access to a Nimble array for SS testing? As I requested in the case, perhaps Veeam can lab this themselves and confirm how the product actually works, as it seems no one is sure how it is supposed to perform (generate errors or not!)...?

[wishr]

Hi Chris,

I've added an update to my post above - please take a look.

Thanks

[ChrisGundry]

Ah sorry, I see I failed to mention SS in the original post/updates. We almost solely do guest restores from SS, so must have slipped my mind when doing the forum post. Yes, this is 100% SS only, no issues with restores from normal backups. It was mentioned in the case of course.

Can you clarify why is this considered a feature request, rather than a bug? Whilst VeeamONE doesn't 'support' restores from SS, if B&R removed the storage from the host cleanly, there would be no error in ESX and therefore no error in VeeamONE. B&R 'supports' restore from SS, it is a product feature, so should work without generating errors on the host. This would happen if VeeamONE wasn't being used at all. So to me this is a bug, not a feature request. If you 'fix' this issue in VeeamONE via a feature request, the only 'fix' in VeeamONE can be saying "ignore the error in ESX if it is due to Veeam", where the fix should actually be in B&R to stop it happening at the source.

[wishr]

I understand and agree with your logic, but there are some factors that should be also considered.

Support for storage snapshots is a feature request because they are not supported currently by VONE. Thus, anything related to storage snapshots falls under this particular entity including any issues related to this functionality.

Speaking about VBR, the approach we use to recover your guest files is the only possible method the technology allows. The datastore attachment procedure is required here since the process utilizes vSphere components and hence we must disconnect it to not cause accumulation. If changes would be possible at some point, we will be happy to apply them.

I hope it helps. Thanks.

[ChrisGundry]

I am not disputing that you need to attach, and of course you need to detach it, it can't stay there forever. I am saying it should be detached cleanly, so there are no errors in ESX. If there are no errors in ESX, then there will be no errors in VeeamONE and no changes or feature requests required in VeeamONE.

[wishr]

I do not think it is technically possible to detach a datastore without generating an event on the vSphere side as this is simply how vSphere works, but I'll discuss it with the VBR team members.