Sessions left open after Veeam (maybe due to VSS?)

[awiesen]

Hi,

We're backing up a VM running Server 2008.
On that VM, in Computer Management > Sessions, there's sessions that were opened to C:\WINDOWS during the time of each backup from Veeam that were accessed very briefly and then left open (and are idle.) It looks like Veeam is opening one of these each time it does a backup. (I assume it's for the VSS support.)

This seems perfectly reasonable that they were opened, but I wonder why they weren't closed? Or maybe Veeam closes old ones after awhile?

Any insight would be appreciated, thanks.

- Alex

[Gostev]

Veeam VSS does close the session immediately after processing completes. I believe this behavior is specific to Windows OS and implementation of this sessions monitor, or due to some OS optimizations. Based on my experience with Windows, sessions stay open for some time after any remote operaions. For example, when I share some file on my computer, other users come and download it in a few seconds, however I can see their session remaining displayed there from quite some time - I actually have to purposely kill them in order to let other users get through (connection limit on desktop OS).

It could be that Windows keeps the session in case the user decides to reconnect... could be some sort of session cache I am guessing.

[Gostev]

I did a little Googling on this, and it looks like Windows indeed keeps idle sessions for some time before closing them completely. The timeout is configurable through the command below, see Idle session time (min) parameter.

Code: Select all

C:\Windows\system32>net config server
Server Name                           \\CENTURION
Server Comment

Software version                      Windows 7 Ultimate
Server is active on
        NetbiosSmb (CENTURION)
        NetBT_Tcpip_{240478B8-B2B2-4CB0-8EC6-4F47FC7B9EB3} (CENTURION)
        NetBT_Tcpip_{EBB25EB4-5BD8-4092-AFA0-3CCD199A286B} (CENTURION)
        NetBT_Tcpip_{A639D327-6002-4474-9C97-8C5FBA59C744} (CENTURION)


Server hidden                         No
Maximum Logged On Users               20
Maximum open files per session        16384

Idle session time (min)               15
The command completed successfully.

[awiesen]

I checked on my end and on our servers we haven't changed the idle timeout; it's also 15 minutes. However, for each Veeam backup job, we have an open session with one open file that is idle for every backup we've done since we last rebooted the servers. I've confirmed this on several other servers.

For example, on one server, we last rebooted it 16 days ago. We do two daily Veeam backup jobs for this server -- one for the C drive, one for another drive (we like to keep the .VBK files separate, even though we lose out on deduplication.) For each backup job, an open session is left open and idle. So we have 32 open sessions, the oldest two of which have been idle for 16 days.

It seems like maybe Veeam is holding a file lock open, which is causing the session to not close properly? The fact that it lists the number of open files for each of these sessions as "1" as opposed to "0" makes me think Windows isn't closing the session due to Veeam holding a file open.

- Alex

[Gostev]

Alex, I will ask QC to try to reproduce this.

[Gostev]

Alex, we have tested this, and in our lab these sessions do close properly, and then correctly disappear as expected. There are no open files either. So this issue looks to be specific to your environment. So please open a support case and our technical staff will dig deeper into this. Thanks!

[awiesen]

More information in case other people run into this problem:

It appears to be specific to Windows Server 2008. (Not Server 2008 R2, although we haven't tested it there.) It happens with all of our Server 2008 VMs, and with none of our XP Pro 32bit VMs. (We no longer have any XP Pro 64 bit VMs as those turned out to be not supported under Veeam with regards to VSS support, although Veeam originally advertised support for them.)

Veeam Support was unable to determine the cause of this problem, so we're stuck trying to jury rig a solution. Veeam support blamed Windows for the problem and told me to write a script, but I have no idea what kind of script I'm supposed to write or what it would contain. I'm hoping for some additional help, but we'll see if they come through.

At any rate, we'll probably have to switch to another backup program if we can't solve this issue. I'm starting to think Veeam isn't a good choice for people backing up Windows VMs -- first we had the problem with Windows XP Professional 64 bit, and now we have this problem with Windows Server 2008. It seems like Windows is not the preferred OS when using Veeam Backup & Replication. I realize the problem might be some configuration issue on our end (on all of our 2008 servers?) but it seems like Veeam's support department ought to be able to solve that, if that was indeed the problem.

Anyways, I hope to be proven wrong, but this is the second serious problem which Veeam has basically said, sorry, you're out of luck with this version of Windows...

- Alex

[Gostev]

Alex, did you consider disabling Veeam VSS completely for your Windows 2008 servers until this is resolved? This will make backup as good as with any other backup tool (as we are the only image-level backup vendor to provide advanced Windows VSS support today).

I hope your issue gets resolved though, can you PM me your support case number - I would like to review what the current state is.

[awiesen]

Hi,

I didn't try disabling VSS, but if that's the solution to this problem, I think Veeam should warn users about this limitation (along with the Windows XP 64 bit one where VSS just doesn't work at all) up front so they don't buy the product expecting to use VSS, only to find it doesn't work properly...

Our case number is ID#520824.

- Alex

[awiesen]

On the positive side, I took a look at the latest Release Notes for Veeam Backup & Replication, and it looks like Veeam did go ahead and note 32 bit only for XP Pro (for VSS support.) (I think we were the first people to try it on 64 bit, and we asked that the documentation reflect this limitation.) It's nice to see that Veeam did update the documentation for that case. If there's no solution to our newer problem, we'd like to at least see Veeam put something in the Release Notes under "Known Issues." Having the documentation updated at least helps us feel that these limitations are being disclosed and are documented.

- Alex

[Gostev]

I didn't try disabling VSS, but if that's the solution to this problem, I think Veeam should warn users about this limitation (along with the Windows XP 64 bit one where VSS just doesn't work at all) up front so they don't buy the product expecting to use VSS, only to find it doesn't work properly...

Our case number is ID#520824.

Well, there are no known issues with Veeam VSS processing of Windows 2008 today, this is basically one of the primary testing platforms for us.

Thanks for the case ID, I recall this issue now, especially after I have re-read the beginning of this topic. We did try to reproduce the issue as soon as you have reported it on forum, but did not observe any issues with connections not closing on Windows 2008 VM - which is what I reported above on Feb 1st. Looks like the case was closed after that, but now it is reopened since yesterday.

I have asked QC to triple check this on clean Windows 2008 install before we go further, but currently this indeed looks like problem specific to your Windows 2008 boxes. I will also consult with devs if they can think of some basic Windows management operation not involving Veeam Backup that could show that the issue is not specific to our solution.

[awiesen]

Just a follow-up in case anyone searches the forums:

After spending additional time with technical support, they tried to have me do things to simulate the same behavior (i.e. without using Veeam B&R, see if sessions were left open) and were unable to do so. After a week they notified me about the presence of a new hot fix for Veeam B&R. We downloaded and installed it, and voila, the problem is solved! All of our sessions are now closed and the problem went away.

Note: Some possible factors related to our problem may be:

(1) VSS
(2) Multiple backup jobs that backup single drives on the same server
(3) Windows Server 2008 (not R2) -- we didn't see this on our XP VMs, for example

It took awhile to get this fixed, but it was nice to see that Veeam support came through in the end with a hot fix that solves our problem. Thanks Veeam!

- Alex

[Gostev]

Thanks for the update! Yes, it was quite an interesting issue to resolve. We were (and still are) unable to reproduce the issue on our lab under the same condition, but after very careful review of the session management code, we have found one possible issue and changed some things, and then testing in your environment showed that this helps (and changes nothing for us). I think what happens is this: in most cases Windows handles the way we used to manage sessions fine, but some specific settings or software affect this.

[larry]

I have Veeam 4.1 and I have this issue on a windows 7. This PC is backed up every hour 24 X 7. Every few days it needs to be rebooted. If I look in net sessions it is full of Veeam connections.

[tsightler]

I'm having this issue with a Windows 7 VM as well, is this "hotfix" included in Veeam 4.1.2. I haven't updated my production environment to 4.1.2 yet but this might be the kick in the pants I need to get it done.

[Gostev]

Yes, documented under Resolved Issues of the Release Notes document for 4.1.2:
http://www.veeam.com/veeam_backup_4_1_2 ... tes_rn.pdf