VSS not working in 6.5 and it did in 6.1

[systeembeheerder]

Hi,

I had a good working set of backups when running VBR 6.1. After upgrading to 6.5 this week, backups with Application Awareness on, fail on Windows 2008R2 servers.
Example: backupjob with 3 virtual machines in it, 1 windows 2003, 2 windows 2008R2, finishes with the windows 2003 successful and the other two fail with :

" Failed to prepare guest for hot backup. Error: RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine: [172.18.1.20]. RPC error:The RPC server is unavailable. Code: 1722
Error: RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine: [172.18.1.20]. RPC error:The RPC server is unavailable. Code: 1722 "

clearing the checkbox for application awareness, and rerunning the job again, all succeed.
So, my question/conclusion would be: what change from 6.1 to 6.5? As this worked well, and now it does not.
Is it a know bug?

Also: rebooted all my proxyservers and backup servers, just to be sure, still the same problem.

[Vitaliy S.]

Hi Addy, there were no changes in VSS engine. Based on your error message, I can say that for some reason RPC server on the backed up VM is not available, you can verify that by using other tools accessing the RPC server. In addition to this, please note that you must include support case ID with any issue you are posting on these forums, or your post will be removed - as explained when you click New Topic. Thanks!

[systeembeheerder]

Hi Vitaliy,
It worked before, nothing changed but the backup software, so I can't see why I should look into the RPC being the problem, as it worked for months before with 6.1. I will make this a support case.
just my 2 cents: I think a good forum with other users would create a great resource to keep questions away from expensive support personnel. It also gives users a place to see if others are having the same problems. If I put my case with support, I 'm not going to place it on the forum any more, as I have people looking into it, and I don't see the added benefits in posting is here. As I said: my 2 cents. You will have your reasons to do it like this.

[Vitaliy S.]

Unfortunately, this doesn't work this way and in most of the cases our support team have to review the logs anyway. However, if you would like to get feedback from the community it's always a good idea to post the support ticket number, so we could update the topic with the resolution (once the case is resolved) for future readers.

And thank you for the feedback!

[cristiano.cumer]

Hello,

I have the same issue. I will open a case now but as temporary workaround you can follow those steps provided you aren't using local Administrator as backup user

- check that you can't delete the profile of the user you are using for indexing/vss processing on the vm
- reboot the affected vm
- delete the profile of the user you are using for indexing/vss processing
- login as this user on the vm
- logout

Now it should work for some days.

What happens in my environment is that some files in the backup user profile are locked by the system and never released and when this happens RPC calls are failing

Cristiano

[cristiano.cumer]

Hello,

my case is #00152306

Cristiano

[systeembeheerder]

Christiano,

By: "Now it should work for some days" , you mean, then is stops working again, and fails because it gets locked files again in the backup user profile?
I only use the backup user for backups. I never login with it on any machine, so how would it get locked files, or do you use this user to login also?

By the way, case: #00152257

[cristiano.cumer]

Hello All

I think I did understand what is going on in my backup environment. First of all all the jobs I had in our DMZ went from VIX (vmware tool network-less processing) to RPC processing with the upgrade from 6.1. to 6.5
I did the necessary changes on our firewall for the RPC port range 1025/5000 for 2003 and 49152/65535 for 2008.

This solved most of my problems, but some 2008 VMs still failed with the 1722 error code. I did a check on those machines issuing this command

Code: Select all

netsh int ipv4 show dynamicport tcp

and I got a range from 1025 to 65535 which was not covered by my firewall rules. This is also the reason why sometimes the backup is succeeding and sometimes not, it all depends if the port RPC randomly chooses is the open ports list. Moreover when the backups fails this way it leaves the user profile of the backup user in a inconsistent state, forcing me to delete the user profile.

Now I have restored the ports on those VMs with this command

Code: Select all

netsh int ipv4 set dynamicport tcp start=49152 num=16384

and I will see tonight if the next run will be ok.


Se also

http://www.veeam.com/KB1174
http://support.microsoft.com/kb/929851

Cristiano

[Vitaliy S.]

First of all all the jobs I had in our DMZ went from VIX (vmware tool network-less processing) to RPC processing with the upgrade from 6.1. to 6.5

Veeam B&R has always been using network connection first and then VIX API to access backed up to perform application aware image processing. There was no change in this logic in v6.5. Most likely there was something else that prevented you from having the successful backup, debug logs will definitely shed some light on this.

[brupnick]

We intentionally don't allow the RPC ports because they're so expansive and instead rely on VIX. The problem with this is that if both RPC and VIX fail, only the RPC error is reported in the job summary and email. It's not unless you go into the job logs and find the VIX error that you can actually see why VIX failed (which might be different than why RPC failed).

[Gostev]

In that case, you may want to inverse protocol order as well, so that B&R does not spend time trying to connect over RPC first (see What's New in 6.5 for more information).

[brupnick]

Thanks Gostev! And not to hijack this thread, but on 64-bit systems, does this go in the Wow6432Node? Does this KB from 6.1.0.181 Patch 1 still apply for this? I searched the VBR6.5 user's guide and there is no mention of the "InverseVssProtocolOrder" value.
http://www.veeam.com/KB1671

[Gostev]

This value works for both 6.1 Patch 1 and 6.5, and has to be created into that single and only Veeam registry key where all other existing/default values are already present. Remember that B&R is full blown 64-bit application, so no need to mess with Wow6432 key.

[brupnick]

I'm running on a 64-bit system and have both of the following on my VBR server:

HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication

Is the first one the default?

[Gostev]

First one.

[cristiano.cumer]

Veeam B&R has always been using network connection first and then VIX API to access backed up to perform application aware image processing. There was no change in this logic in v6.5. Most likely there was something else that prevented you from having the successful backup, debug logs will definitely shed some light on this.

Yes from the logs I can see first an attemp over RPC and then an attempt over VIX failing with the error:

VIX Error: You do not have access rights to this file Code: 13

I think the VIX error is caused by the fact that I'm not using the local or domain Administrator account, but an ad-hoc domain account with local admin rights. With Veeam B&R 6.1 I was able to use this account for application processing but not for indexing over VIX. Now it fails at both tasks.

[brupnick]

I'm not sure what OS you're using, but I had issues with VSS in Server 2008R2 machines because of UAC and -500 administrator accounts.
http://forums.veeam.com/viewtopic.php?f ... =30#p60841

[Vitaliy S.]

I think the VIX error is caused by the fact that I'm not using the local or domain Administrator account, but an ad-hoc domain account with local admin rights.

Yes, seems like you've nailed it, pretty sure this is the reason for the failure. If you search these forums, you'll find more topics discussing the same configuration problem.

[cristiano.cumer]

Yes, seems like you've nailed it, pretty sure this is the reason for the failure. if you search these forums, you'll find more topics with similar problem.

I'm aware of the issue with admin users. But before the update the user I was using was able to perform VSS snapshots over VIX even if indexing was failing:

Anyway it's working beautifully for me now since I limited the dynamic RPC ports to the default Windows 2008 R2 range

[tuxracer01]

Hi
We have exact the same problem. After upgrading Veeam to v 6.5 one backup with VSS didn't work anymore. So we analysed that some ports were blocked by the firewall. Other VSS backups with the VM and the backup server in the same network are working without problems. We opened a support case (00152266) to ask which ports have to be open for Veeam 6.5. As a workaround we opened all the ports between backup server and VM. But this can't be the solution, also not to open more than 16000 ports.

[Vitaliy S.]

The required ports didn't change since version 6, please see our User Guide (page 64) for the list of all ports. Also be aware that network connection is not required if you want to use VSS to backup your VMs, as "application-aware image processing" can be performed over VMware Tools (VIX) API.

[tuxracer01]

Sorry, but i wonder why we had blocked ports (which are not mentioned in guide as 49156 or 51722, while veeam was trying to backup and failed. As soon as we open the ports, the backup run succesfully? And why this happened exactly after upgrading?

[Vitaliy S.]

Not sure, need to investigate this with our technical support team.

[tuxracer01]

The problem is the UAC as mentioned in the release notes. As we saw the blocked ports on the firewall, we did not deduce that the problem is the UAC, because we were not aware, that before it worked without network. The "solutions" are turn off UAC or open the firewall.

[brupnick]

Do you have the option to use the built in, -500 SID administrator account? This account has special OS permissions, one of which is to bypass UAC. This allows you to keep UAC enabled, but still be able to perform application-aware processing in VBR.

[vnlbeheer]

We seem to have a similar issue after upgrading from version 6.1 to 6.5. All of our servers in the regular LAN are backupping just fast & fine. However backups of our DMZ servers are very slow. Not the actual backup but the 'pre-processing' fase. In the 'client side' log of the server in the DMZ being backed up we can see the following in the logfiles.

Code: Select all

6-11-2012 4:49:32   4296  INFO    Collecting Exchange information... 
6-11-2012 4:49:32   4296  WARN    Failed to get 'ESE' file path
6-11-2012 4:49:32   4296  WARN        Cannot open 'ESE' registry key
6-11-2012 4:49:32   4296  WARN        Win32 error:The system cannot find the file specified.
6-11-2012 4:49:32   4296  WARN         Code: 2

This continues for around 20 minutes up until 6-11-2012 5:11. And no, this is not an Exchange server. After that the actual backup starts which is fast. Total job for this server however is around 25-30 minutes whereas normally in 6.1 it would be 4-7 minutes. Nothing has changed in our environment except for Veeam moving from 6.1 to 6.5. Used system accounts & vCenter server are the same, proxy is the same as well.

We have an open case # 00154797 for this.

[ffortier]

I have the same problem. I have opened up the ports between my lan and dmz and now everything is working. The ports needed are listed here:

http://www.veeam.com/veeam_backup_6_use ... are_pg.pdf
pages 64-65 show the port listings

[Gostev]

We seem to have a similar issue after upgrading from version 6.1 to 6.5. All of our servers in the regular LAN are backupping just fast & fine. However backups of our DMZ servers are very slow. Not the actual backup but the 'pre-processing' fase. In the 'client side' log of the server in the DMZ being backed up we can see the following in the logfiles.

Code: Select all

6-11-2012 4:49:32   4296  INFO    Collecting Exchange information... 
6-11-2012 4:49:32   4296  WARN    Failed to get 'ESE' file path
6-11-2012 4:49:32   4296  WARN        Cannot open 'ESE' registry key
6-11-2012 4:49:32   4296  WARN        Win32 error:The system cannot find the file specified.
6-11-2012 4:49:32   4296  WARN         Code: 2

This continues for around 20 minutes up until 6-11-2012 5:11. And no, this is not an Exchange server. After that the actual backup starts which is fast. Total job for this server however is around 25-30 minutes whereas normally in 6.1 it would be 4-7 minutes. Nothing has changed in our environment except for Veeam moving from 6.1 to 6.5. Used system accounts & vCenter server are the same, proxy is the same as well.

We have an open case # 00154797 for this.

This was confirmed to be an issue, hotfix is in the works.