Comprehensive data protection for all workloads
Post Reply
systeembeheerder
Enthusiast
Posts: 44
Liked: 2 times
Joined: Dec 01, 2009 8:09 pm
Full Name: Addy
Contact:

VSS not working in 6.5 and it did in 6.1

Post by systeembeheerder » Nov 01, 2012 8:33 am

Hi,

I had a good working set of backups when running VBR 6.1. After upgrading to 6.5 this week, backups with Application Awareness on, fail on Windows 2008R2 servers.
Example: backupjob with 3 virtual machines in it, 1 windows 2003, 2 windows 2008R2, finishes with the windows 2003 successful and the other two fail with :

" Failed to prepare guest for hot backup. Error: RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine: [172.18.1.20]. RPC error:The RPC server is unavailable. Code: 1722
Error: RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine: [172.18.1.20]. RPC error:The RPC server is unavailable. Code: 1722 "

clearing the checkbox for application awareness, and rerunning the job again, all succeed.
So, my question/conclusion would be: what change from 6.1 to 6.5? As this worked well, and now it does not.
Is it a know bug?

Also: rebooted all my proxyservers and backup servers, just to be sure, still the same problem.

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Vitaliy S. » Nov 01, 2012 8:51 am

Hi Addy, there were no changes in VSS engine. Based on your error message, I can say that for some reason RPC server on the backed up VM is not available, you can verify that by using other tools accessing the RPC server. In addition to this, please note that you must include support case ID with any issue you are posting on these forums, or your post will be removed - as explained when you click New Topic. Thanks!

systeembeheerder
Enthusiast
Posts: 44
Liked: 2 times
Joined: Dec 01, 2009 8:09 pm
Full Name: Addy
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by systeembeheerder » Nov 01, 2012 9:50 am

Hi Vitaliy,
It worked before, nothing changed but the backup software, so I can't see why I should look into the RPC being the problem, as it worked for months before with 6.1. I will make this a support case.
just my 2 cents: I think a good forum with other users would create a great resource to keep questions away from expensive support personnel. It also gives users a place to see if others are having the same problems. If I put my case with support, I 'm not going to place it on the forum any more, as I have people looking into it, and I don't see the added benefits in posting is here. As I said: my 2 cents. You will have your reasons to do it like this.

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Vitaliy S. » Nov 01, 2012 10:47 am 1 person likes this post

Unfortunately, this doesn't work this way and in most of the cases our support team have to review the logs anyway. However, if you would like to get feedback from the community it's always a good idea to post the support ticket number, so we could update the topic with the resolution (once the case is resolved) for future readers.

And thank you for the feedback!

cristiano.cumer
Influencer
Posts: 17
Liked: 6 times
Joined: Nov 23, 2011 11:18 pm
Full Name: Cristianno Cumer
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by cristiano.cumer » Nov 01, 2012 12:34 pm 1 person likes this post

Hello,

I have the same issue. I will open a case now but as temporary workaround you can follow those steps provided you aren't using local Administrator as backup user

- check that you can't delete the profile of the user you are using for indexing/vss processing on the vm
- reboot the affected vm
- delete the profile of the user you are using for indexing/vss processing
- login as this user on the vm
- logout

Now it should work for some days.

What happens in my environment is that some files in the backup user profile are locked by the system and never released and when this happens RPC calls are failing

Cristiano

cristiano.cumer
Influencer
Posts: 17
Liked: 6 times
Joined: Nov 23, 2011 11:18 pm
Full Name: Cristianno Cumer
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by cristiano.cumer » Nov 01, 2012 1:24 pm

Hello,

my case is #00152306

Cristiano

systeembeheerder
Enthusiast
Posts: 44
Liked: 2 times
Joined: Dec 01, 2009 8:09 pm
Full Name: Addy
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by systeembeheerder » Nov 01, 2012 4:06 pm

Christiano,

By: "Now it should work for some days" , you mean, then is stops working again, and fails because it gets locked files again in the backup user profile?
I only use the backup user for backups. I never login with it on any machine, so how would it get locked files, or do you use this user to login also?

By the way, case: #00152257

cristiano.cumer
Influencer
Posts: 17
Liked: 6 times
Joined: Nov 23, 2011 11:18 pm
Full Name: Cristianno Cumer
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by cristiano.cumer » Nov 01, 2012 5:08 pm

Hello All

I think I did understand what is going on in my backup environment. First of all all the jobs I had in our DMZ went from VIX (vmware tool network-less processing) to RPC processing with the upgrade from 6.1. to 6.5
I did the necessary changes on our firewall for the RPC port range 1025/5000 for 2003 and 49152/65535 for 2008.

This solved most of my problems, but some 2008 VMs still failed with the 1722 error code. I did a check on those machines issuing this command

Code: Select all

netsh int ipv4 show dynamicport tcp
and I got a range from 1025 to 65535 which was not covered by my firewall rules. This is also the reason why sometimes the backup is succeeding and sometimes not, it all depends if the port RPC randomly chooses is the open ports list. Moreover when the backups fails this way it leaves the user profile of the backup user in a inconsistent state, forcing me to delete the user profile.

Now I have restored the ports on those VMs with this command

Code: Select all

netsh int ipv4 set dynamicport tcp start=49152 num=16384
and I will see tonight if the next run will be ok.


Se also

http://www.veeam.com/KB1174
http://support.microsoft.com/kb/929851

Cristiano

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Vitaliy S. » Nov 01, 2012 7:38 pm

cristiano.cumer wrote:First of all all the jobs I had in our DMZ went from VIX (vmware tool network-less processing) to RPC processing with the upgrade from 6.1. to 6.5
Veeam B&R has always been using network connection first and then VIX API to access backed up to perform application aware image processing. There was no change in this logic in v6.5. Most likely there was something else that prevented you from having the successful backup, debug logs will definitely shed some light on this.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by brupnick » Nov 01, 2012 7:43 pm

We intentionally don't allow the RPC ports because they're so expansive and instead rely on VIX. The problem with this is that if both RPC and VIX fail, only the RPC error is reported in the job summary and email. It's not unless you go into the job logs and find the VIX error that you can actually see why VIX failed (which might be different than why RPC failed).

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Gostev » Nov 01, 2012 7:50 pm

In that case, you may want to inverse protocol order as well, so that B&R does not spend time trying to connect over RPC first (see What's New in 6.5 for more information).

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by brupnick » Nov 01, 2012 8:25 pm

Thanks Gostev! And not to hijack this thread, but on 64-bit systems, does this go in the Wow6432Node? Does this KB from 6.1.0.181 Patch 1 still apply for this? I searched the VBR6.5 user's guide and there is no mention of the "InverseVssProtocolOrder" value.
http://www.veeam.com/KB1671

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Gostev » Nov 01, 2012 8:27 pm

This value works for both 6.1 Patch 1 and 6.5, and has to be created into that single and only Veeam registry key where all other existing/default values are already present. Remember that B&R is full blown 64-bit application, so no need to mess with Wow6432 key.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by brupnick » Nov 01, 2012 8:31 pm

I'm running on a 64-bit system and have both of the following on my VBR server:

HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication

Is the first one the default?

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Gostev » Nov 01, 2012 8:38 pm

First one.

cristiano.cumer
Influencer
Posts: 17
Liked: 6 times
Joined: Nov 23, 2011 11:18 pm
Full Name: Cristianno Cumer
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by cristiano.cumer » Nov 01, 2012 8:44 pm

Vitaliy S. wrote:Veeam B&R has always been using network connection first and then VIX API to access backed up to perform application aware image processing. There was no change in this logic in v6.5. Most likely there was something else that prevented you from having the successful backup, debug logs will definitely shed some light on this.
Yes from the logs I can see first an attemp over RPC and then an attempt over VIX failing with the error:

VIX Error: You do not have access rights to this file Code: 13

I think the VIX error is caused by the fact that I'm not using the local or domain Administrator account, but an ad-hoc domain account with local admin rights. With Veeam B&R 6.1 I was able to use this account for application processing but not for indexing over VIX. Now it fails at both tasks.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by brupnick » Nov 01, 2012 8:51 pm

I'm not sure what OS you're using, but I had issues with VSS in Server 2008R2 machines because of UAC and -500 administrator accounts.
http://forums.veeam.com/viewtopic.php?f ... =30#p60841

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Vitaliy S. » Nov 01, 2012 8:57 pm

cristiano.cumer wrote:I think the VIX error is caused by the fact that I'm not using the local or domain Administrator account, but an ad-hoc domain account with local admin rights.
Yes, seems like you've nailed it, pretty sure this is the reason for the failure. If you search these forums, you'll find more topics discussing the same configuration problem.

cristiano.cumer
Influencer
Posts: 17
Liked: 6 times
Joined: Nov 23, 2011 11:18 pm
Full Name: Cristianno Cumer
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by cristiano.cumer » Nov 01, 2012 9:15 pm 1 person likes this post

Vitaliy S. wrote:Yes, seems like you've nailed it, pretty sure this is the reason for the failure. if you search these forums, you'll find more topics with similar problem.
I'm aware of the issue with admin users. But before the update the user I was using was able to perform VSS snapshots over VIX even if indexing was failing:

Anyway it's working beautifully for me now since I limited the dynamic RPC ports to the default Windows 2008 R2 range

tuxracer01
Novice
Posts: 5
Liked: never
Joined: Jun 20, 2012 1:52 pm
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by tuxracer01 » Nov 05, 2012 10:04 am

Hi
We have exact the same problem. After upgrading Veeam to v 6.5 one backup with VSS didn't work anymore. So we analysed that some ports were blocked by the firewall. Other VSS backups with the VM and the backup server in the same network are working without problems. We opened a support case (00152266) to ask which ports have to be open for Veeam 6.5. As a workaround we opened all the ports between backup server and VM. But this can't be the solution, also not to open more than 16000 ports.

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Vitaliy S. » Nov 05, 2012 10:21 am

The required ports didn't change since version 6, please see our User Guide (page 64) for the list of all ports. Also be aware that network connection is not required if you want to use VSS to backup your VMs, as "application-aware image processing" can be performed over VMware Tools (VIX) API.

tuxracer01
Novice
Posts: 5
Liked: never
Joined: Jun 20, 2012 1:52 pm
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by tuxracer01 » Nov 05, 2012 12:35 pm

Sorry, but i wonder why we had blocked ports (which are not mentioned in guide as 49156 or 51722, while veeam was trying to backup and failed. As soon as we open the ports, the backup run succesfully? And why this happened exactly after upgrading?

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Vitaliy S. » Nov 05, 2012 12:45 pm

Not sure, need to investigate this with our technical support team.

tuxracer01
Novice
Posts: 5
Liked: never
Joined: Jun 20, 2012 1:52 pm
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by tuxracer01 » Nov 13, 2012 8:46 am

The problem is the UAC as mentioned in the release notes. As we saw the blocked ports on the firewall, we did not deduce that the problem is the UAC, because we were not aware, that before it worked without network. The "solutions" are turn off UAC or open the firewall.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by brupnick » Nov 13, 2012 4:31 pm 1 person likes this post

Do you have the option to use the built in, -500 SID administrator account? This account has special OS permissions, one of which is to bypass UAC. This allows you to keep UAC enabled, but still be able to perform application-aware processing in VBR.

vnlbeheer
Novice
Posts: 3
Liked: 1 time
Joined: Nov 14, 2012 7:35 am
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by vnlbeheer » Nov 14, 2012 7:50 am 1 person likes this post

We seem to have a similar issue after upgrading from version 6.1 to 6.5. All of our servers in the regular LAN are backupping just fast & fine. However backups of our DMZ servers are very slow. Not the actual backup but the 'pre-processing' fase. In the 'client side' log of the server in the DMZ being backed up we can see the following in the logfiles.

Code: Select all

6-11-2012 4:49:32   4296  INFO    Collecting Exchange information... 
6-11-2012 4:49:32   4296  WARN    Failed to get 'ESE' file path
6-11-2012 4:49:32   4296  WARN        Cannot open 'ESE' registry key
6-11-2012 4:49:32   4296  WARN        Win32 error:The system cannot find the file specified.
6-11-2012 4:49:32   4296  WARN         Code: 2
This continues for around 20 minutes up until 6-11-2012 5:11. And no, this is not an Exchange server. ;-) After that the actual backup starts which is fast. Total job for this server however is around 25-30 minutes whereas normally in 6.1 it would be 4-7 minutes. Nothing has changed in our environment except for Veeam moving from 6.1 to 6.5. Used system accounts & vCenter server are the same, proxy is the same as well.

We have an open case # 00154797 for this.

ffortier
Lurker
Posts: 2
Liked: never
Joined: Dec 08, 2011 8:36 pm
Full Name: Francois Fortier
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by ffortier » Nov 16, 2012 4:14 pm

I have the same problem. I have opened up the ports between my lan and dmz and now everything is working. The ports needed are listed here:

http://www.veeam.com/veeam_backup_6_use ... are_pg.pdf
pages 64-65 show the port listings

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS not working in 6.5 and it did in 6.1

Post by Gostev » Nov 16, 2012 10:50 pm

vnlbeheer wrote:We seem to have a similar issue after upgrading from version 6.1 to 6.5. All of our servers in the regular LAN are backupping just fast & fine. However backups of our DMZ servers are very slow. Not the actual backup but the 'pre-processing' fase. In the 'client side' log of the server in the DMZ being backed up we can see the following in the logfiles.

Code: Select all

6-11-2012 4:49:32   4296  INFO    Collecting Exchange information... 
6-11-2012 4:49:32   4296  WARN    Failed to get 'ESE' file path
6-11-2012 4:49:32   4296  WARN        Cannot open 'ESE' registry key
6-11-2012 4:49:32   4296  WARN        Win32 error:The system cannot find the file specified.
6-11-2012 4:49:32   4296  WARN         Code: 2
This continues for around 20 minutes up until 6-11-2012 5:11. And no, this is not an Exchange server. ;-) After that the actual backup starts which is fast. Total job for this server however is around 25-30 minutes whereas normally in 6.1 it would be 4-7 minutes. Nothing has changed in our environment except for Veeam moving from 6.1 to 6.5. Used system accounts & vCenter server are the same, proxy is the same as well.

We have an open case # 00154797 for this.
This was confirmed to be an issue, hotfix is in the works.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Majestic-12 [Bot] and 24 guests