feature request - iSCSI reconnect after interruption

[mcz]

Hi everybody,

we have a qnap nas which provides iSCSI. Our veeam server (Windows 10) connects to the nas and shows a dedicated drive F:\ which is being used within the backup console as a repository. From time to time when we restart the device, the connection will of course be interrupted but after the nas comes up again, windows will automatically do the reconnect. Somehow, veeam backup for office 365 won't try to reconnect and all the backup jobs are failing. I will then have to restart the veeam services to bring the jobs back to work.

It would be nice if veeam would do an automatic reconnect to the iSCSI device.

Please tell me what you think about it - thank you!

[foggy]

Hi Michael, seems that the period during which the device is down exceeds the default job retry periods. You can increase the default values, should help in your case.

[Mike Resseler]

Michael,

As a follow-up: When you do that reconnect, will the jobs run AFTER the reconnect and fail? Or does this happen (the fails) when it is disconnected? What Foggy says is in the case of a backup running. But I think your case is a bit different? Or am I wrong?

Cheers
Mike

[mcz]

Thank you Foggy and Mike for your replies! By the way I would like to mention that this forum is really great fun because you get an answer very very quickly and you really feel that the user voice is being heard - thank you very much for this kind of treatment and please keep doing what you're doing!

Coming back to our original topic: When the iSCSI target isn't available during backup job (or job retry) it will of course fail but if the target gets available again, the next job should be working (because windows does the automatic reconnect and you can browse the directory again) but veeam would fail. As I said, only restarting the veeam services will solve the issue, otherwise every future job (including the retries) would fail.

So in our case we're having one backup job which runs every full hour and it has 3 retries with a delay of 10 minutes. If I'm restarting the nas and let's assume it will take 20 minutes until every service comes up again, at least the next job within the next hour should be successful but as I said, it won't.

Hope it is clear now, please let me know what you're thinking about it - thank you.

[Mike Resseler]

Very clear

I will ask our R&D team to investigate this for the next version and see if we can solve this.

Thanks for letting us know!
Mike

[Mike Resseler]

Hi Michael,

First: My apologies for the late reply (We have been at our internal Sales Kickoff and I still owed you an answer )

We have been trying to reproduce this but failed. In our tests, a reconnect happens automatically.

Could you please create a support call and ask the support engineer to escalate (You can reference this thread to the support engineer)? Then post the case ID to here so I can forward it to our QC department so that they are aware of this specific one.

Thanks
Mike

[mcz]

Hi Mike, meanwhile I've created a ticket: #02717675

[Mike Resseler]

Thanks for this. Let me know how it proceeds!

[coreypenford]

I'll chime in here as well and say the same thing happens in our setup.
Synology NAS with Windows Server 2016 as the Veeam 365 server
We reboot the NAS every couple of weeks for firmware updates, and I've always noticed that Veeam 365 starts reporting issues accessing the repo (disk IO errors) when this happens. Even though as Michael said, the drive is reconnected by Windows automatically and accesssible.

I've just been rebooting the server completely so it's good to know it's just the Veeam service that has to be restarted.

Would love an update to be put into this thread when you have the chance

[mcz]

Hi coreypenford,

currently I'm waiting for a remote session. QA departement is already involved and they'll take a look at our infrastructure. I'll let you know as soon as I have any news!

[mcz]

Okay I just had a remote session with the support engineer and it turns out that the disk from the windows os disappears when the nas (iscsi target) is offline. This behaviour seems to be different when using windows servers as iscsi target. But the real issue is that the proxy service keeps the handles to the database open. When the iscsi disk is online again those handles would be invalid and this breaks the whole process.

The support now is asking the developer why those handles stay open because they are expected to be closed when the job has finished.

[MartijnT]

I think I had a similar problem and also would like this feature.
I created a case: 02733523
The error I got was: JetError -1022, JET_errDiskIO, Disk IO error
I thought the Database was actually damaged. I made sure the backup job was disabled when disconnecting the iSSCI drive but apparently the database was still "connected".
After restarting the Veeam services everything worked again.

[Mike Resseler]

MartijnT,

First: Welcome to the forums! As mcz said, this has gone to the DEV teams for review. At this moment I can't give a status update on it yet but we are going to look into it. Unfortunately I need to tell you that at this moment it is "restart the service" as the option to get it running again.

[mcz]

I was just talking to my support engineer and the dev teams have found an issue which seems to cause our situation. The'll fix it and I will get another information how to apply the fix.

@MartijnT: I have mentioned that your case could also be included and my support engineer Alexander should have taken this case over to him. So I think you will also get the information when the fix is available.

[Mike Resseler]

Michael,

thanks for the update, I hope they indeed have found the issue but knowing them I would actually be surprised if they didn't

Keep us indeed further informed

Thanks
Mike

[coreypenford]

Will this fix be posted publicly or will I have to open a case regarding this as well?

[Mike Resseler]

Hi coreypenford,

At this moment I don't have the information about the fix yet. When I do, I will let you know but the way we work, it is much better if you open a support call also. Then you will get the fix faster. Normally what we do is that we keep this a private fix until the next version and embed it (or the next update) unless it is something that is needed by many customers, then it becomes a KB. In this case I am not sure yet what it is going to be so please open a support case (and let us know the number)

Thank
Mike

[mcz]

The fix isn't available yet, my engineer has also done a further test on friday and collected another dump. That means work is still in progress but I think it will look different in a few days

[mcz]

ok meanwhile I have received the fix but it's not working as expected. So Iet's wait and see what the engineering team is telling us.

[Mike Resseler]

That is not good news. Is it the same case that is still open?

[mcz]

Mike, I'm not sure if I got your question right but yes still the same case # 02717675 which has been opened on the 28th of march

[Mike Resseler]

Yes. Sorry, should have been more clear

If there is still no fix by Friday, please PM me (or use the forum here) so I can talk to some people

Thanks
M/

[mcz]

Thank you Mike, I'll let you know.

[mcz]

OK, yesterday I have received another dll but issue is still there. I've already uploaded the logs and now let's wait for the next run. Seems to be a tricky task.

[Mike Resseler]

Michael,

First, my apologies. I did talk with our teams after your DM and it became indeed clear that it isn't as simple as it looks. I'll spare you the details. I did however forgot to DM you back so mea culpa

[mcz]

Thanks for the reply, Mike, but I would be very interested into the details. So if you like, please let me know what's going on. Thank you!

[Mike Resseler]

In short. It has been discovered that there is a memory leak somewhere in the code. When you restart, the program doesn't realize it needs to do the reconnect and therefore fails. The problem is, it can't be reproduced with all iSCSI connections (if I am not mistaken). So you basically found us a very interesting (but challenging) bug. Which is hard on us now, but actually good looking at it in the long run

[mcz]

Thank you for this explanation Mike, I'm glad that I've found something that maybe would have become obvious in a more annoying way. Support engineer has forgotten to tell me that I should try to do a backup twice because it couldn't work on the first but on the second attempt - which was indeed the case. So finally, the dev's have found a workaround

[Mike Resseler]

Yeah, but a workaround is not what is a final option at Veeam

[mcz]

OK support engineer has replaced some dll's and now I'm testing the behaviour again. What I saw so far is that veeam does the reconnect properly without error message As Mike said, workarounds are not veeam-style