Maintain control of your Microsoft Office 365 email data
mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

feature request - iSCSI reconnect after interruption

Post by mcz » Dec 27, 2017 10:17 am

Hi everybody,

we have a qnap nas which provides iSCSI. Our veeam server (Windows 10) connects to the nas and shows a dedicated drive F:\ which is being used within the backup console as a repository. From time to time when we restart the device, the connection will of course be interrupted but after the nas comes up again, windows will automatically do the reconnect. Somehow, veeam backup for office 365 won't try to reconnect and all the backup jobs are failing. I will then have to restart the veeam services to bring the jobs back to work.

It would be nice if veeam would do an automatic reconnect to the iSCSI device.

Please tell me what you think about it - thank you!

foggy
Veeam Software
Posts: 18390
Liked: 1580 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by foggy » Dec 27, 2017 3:19 pm

Hi Michael, seems that the period during which the device is down exceeds the default job retry periods. You can increase the default values, should help in your case.

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Jan 02, 2018 6:06 am

Michael,

As a follow-up: When you do that reconnect, will the jobs run AFTER the reconnect and fail? Or does this happen (the fails) when it is disconnected? What Foggy says is in the case of a backup running. But I think your case is a bit different? Or am I wrong?

Cheers
Mike

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Jan 03, 2018 9:22 am

Thank you Foggy and Mike for your replies! By the way I would like to mention that this forum is really great fun because you get an answer very very quickly and you really feel that the user voice is being heard - thank you very much for this kind of treatment and please keep doing what you're doing!

Coming back to our original topic: When the iSCSI target isn't available during backup job (or job retry) it will of course fail but if the target gets available again, the next job should be working (because windows does the automatic reconnect and you can browse the directory again) but veeam would fail. As I said, only restarting the veeam services will solve the issue, otherwise every future job (including the retries) would fail.

So in our case we're having one backup job which runs every full hour and it has 3 retries with a delay of 10 minutes. If I'm restarting the nas and let's assume it will take 20 minutes until every service comes up again, at least the next job within the next hour should be successful but as I said, it won't.

Hope it is clear now, please let me know what you're thinking about it - thank you.

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Jan 03, 2018 9:25 am

Very clear :-)

I will ask our R&D team to investigate this for the next version and see if we can solve this.

Thanks for letting us know!
Mike

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Jan 22, 2018 9:02 am

Hi Michael,

First: My apologies for the late reply (We have been at our internal Sales Kickoff and I still owed you an answer :-( )

We have been trying to reproduce this but failed. In our tests, a reconnect happens automatically.

Could you please create a support call and ask the support engineer to escalate (You can reference this thread to the support engineer)? Then post the case ID to here so I can forward it to our QC department so that they are aware of this specific one.

Thanks
Mike

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Mar 30, 2018 8:54 am

Hi Mike, meanwhile I've created a ticket: #02717675

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Mar 30, 2018 9:00 am

Thanks for this. Let me know how it proceeds!

coreypenford
Influencer
Posts: 14
Liked: never
Joined: Mar 28, 2018 6:46 pm
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by coreypenford » Apr 03, 2018 1:27 pm

I'll chime in here as well and say the same thing happens in our setup.
Synology NAS with Windows Server 2016 as the Veeam 365 server
We reboot the NAS every couple of weeks for firmware updates, and I've always noticed that Veeam 365 starts reporting issues accessing the repo (disk IO errors) when this happens. Even though as Michael said, the drive is reconnected by Windows automatically and accesssible.

I've just been rebooting the server completely so it's good to know it's just the Veeam service that has to be restarted.

Would love an update to be put into this thread when you have the chance

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 03, 2018 2:04 pm

Hi coreypenford,

currently I'm waiting for a remote session. QA departement is already involved and they'll take a look at our infrastructure. I'll let you know as soon as I have any news!

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 04, 2018 12:27 pm

Okay I just had a remote session with the support engineer and it turns out that the disk from the windows os disappears when the nas (iscsi target) is offline. This behaviour seems to be different when using windows servers as iscsi target. But the real issue is that the proxy service keeps the handles to the database open. When the iscsi disk is online again those handles would be invalid and this breaks the whole process.

The support now is asking the developer why those handles stay open because they are expected to be closed when the job has finished.

MartijnT
Lurker
Posts: 2
Liked: never
Joined: Apr 06, 2018 8:17 am
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by MartijnT » Apr 06, 2018 8:24 am

I think I had a similar problem and also would like this feature.
I created a case: 02733523
The error I got was: JetError -1022, JET_errDiskIO, Disk IO error
I thought the Database was actually damaged. I made sure the backup job was disabled when disconnecting the iSSCI drive but apparently the database was still "connected".
After restarting the Veeam services everything worked again.

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Apr 10, 2018 7:47 am

MartijnT,

First: Welcome to the forums! As mcz said, this has gone to the DEV teams for review. At this moment I can't give a status update on it yet but we are going to look into it. Unfortunately I need to tell you that at this moment it is "restart the service" as the option to get it running again.

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 10, 2018 9:34 am

I was just talking to my support engineer and the dev teams have found an issue which seems to cause our situation. The'll fix it and I will get another information how to apply the fix.

@MartijnT: I have mentioned that your case could also be included and my support engineer Alexander should have taken this case over to him. So I think you will also get the information when the fix is available.

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Apr 11, 2018 5:12 am

Michael,

thanks for the update, I hope they indeed have found the issue but knowing them I would actually be surprised if they didn't :-)

Keep us indeed further informed

Thanks
Mike

coreypenford
Influencer
Posts: 14
Liked: never
Joined: Mar 28, 2018 6:46 pm
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by coreypenford » Apr 12, 2018 6:26 pm

Will this fix be posted publicly or will I have to open a case regarding this as well?

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Apr 13, 2018 4:59 am

Hi coreypenford,

At this moment I don't have the information about the fix yet. When I do, I will let you know but the way we work, it is much better if you open a support call also. Then you will get the fix faster. Normally what we do is that we keep this a private fix until the next version and embed it (or the next update) unless it is something that is needed by many customers, then it becomes a KB. In this case I am not sure yet what it is going to be so please open a support case (and let us know the number)

Thank
Mike

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 16, 2018 7:46 am

The fix isn't available yet, my engineer has also done a further test on friday and collected another dump. That means work is still in progress but I think it will look different in a few days :)

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 17, 2018 2:26 pm

ok meanwhile I have received the fix but it's not working as expected. So Iet's wait and see what the engineering team is telling us.

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Apr 17, 2018 2:28 pm

That is not good news. Is it the same case that is still open?

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 17, 2018 2:30 pm

Mike, I'm not sure if I got your question right but yes still the same case # 02717675 which has been opened on the 28th of march

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Apr 17, 2018 2:32 pm

Yes. Sorry, should have been more clear ;-)

If there is still no fix by Friday, please PM me (or use the forum here) so I can talk to some people

Thanks
M/

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 17, 2018 3:18 pm

Thank you Mike, I'll let you know.

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 26, 2018 9:56 am

OK, yesterday I have received another dll but issue is still there. I've already uploaded the logs and now let's wait for the next run. Seems to be a tricky task.

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Apr 26, 2018 10:05 am

Michael,

First, my apologies. I did talk with our teams after your DM and it became indeed clear that it isn't as simple as it looks. I'll spare you the details. I did however forgot to DM you back so mea culpa

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 26, 2018 11:24 am

Thanks for the reply, Mike, but I would be very interested into the details. So if you like, please let me know what's going on. Thank you!

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Apr 26, 2018 12:32 pm

In short. It has been discovered that there is a memory leak somewhere in the code. When you restart, the program doesn't realize it needs to do the reconnect and therefore fails. The problem is, it can't be reproduced with all iSCSI connections (if I am not mistaken). So you basically found us a very interesting (but challenging) bug. Which is hard on us now, but actually good looking at it in the long run ;-)

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » Apr 26, 2018 1:01 pm

Thank you for this explanation Mike, I'm glad that I've found something that maybe would have become obvious in a more annoying way. Support engineer has forgotten to tell me that I should try to do a backup twice because it couldn't work on the first but on the second attempt - which was indeed the case. So finally, the dev's have found a workaround :)

Mike Resseler
Product Manager
Posts: 5776
Liked: 621 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by Mike Resseler » Apr 26, 2018 2:08 pm 1 person likes this post

Yeah, but a workaround is not what is a final option at Veeam ;-)

mcz
Expert
Posts: 308
Liked: 57 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: feature request - iSCSI reconnect after interruption

Post by mcz » May 08, 2018 3:27 pm

OK support engineer has replaced some dll's and now I'm testing the behaviour again. What I saw so far is that veeam does the reconnect properly without error message :) As Mike said, workarounds are not veeam-style ;)

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 2 guests