Upgraded to 8.0 u2, Failed to apply retention policy for VM

[wroberts]

Support Case #00907948

After updating to Veeam B&R 8.0 Update 2 I'm having an issue with 2 of 4 replication jobs applying retention policy. This is on an ESXi 5.1 HA cluster. The failing jobs are storing the replicas to iSCSI datastores, the job that succeeds is saving to ESXi local storage. The jobs are replicating the same VMs, from the same iSCSI SAN and from the same vcenter server.

Error/Symptom:
Error: 5/5/2015 10:44:15 AM :: Failed to apply retention policy for VM: VMNAME_HQ_Continuous_Replicas Error: Cannot complete login due to an incorrect user name or password.

The credentials are fine, same credentials used in all 4 jobs, and the actual replication job is 100% successful, including guest processing. It's just the "applying retention policy" that fails.

Eventual Problem:
VMWare has a limit of 32 snapshots it will allow per VM. When that limit is hit all replication jobs will begin to fail stating "Snapshot hierarchy is too deep."

I have tried the following:
1) Reboot the Veeam host server
2) Reboot the vCenter VM
3) Reboot the Veeam proxy servers
4) Restart the ESXi management services on both host servers
5) Try different credentials
6) Create a brand new replication job as follows:

Created a new replication job with a single VM in it, retention policy of 1. First job always completes, second job results are follows:
A) Save replica to a NAS (iSCSI) - fails to apply retention policy
B) Save replica to another NAS (iSCSI) - fails to apply retention policy
C) Save replica to SAN (iSCSI) - fails to apply retention policy
D) Save replica to ESXi Local Storage - success!
Absolutely no changes to the job aside from where the replica should be stored. Credentials never changed.

I'm just guessing here but the only common thread I can find is that I'm using iSCSI, or perhaps non-local storage. Is anyone else finding this behavior? Does anyone have any ideas?

[foggy]

Will, what is the replication job target, is it a standalone host by any chance?

[saxie]

Same Issue here. Support Case # 00910996

[foggy]

In case target is standalone host and there are several standalone hosts added to Veeam B&R console, it is a known issue that support should be able to address.

[saxie]

in my case the destination target is a virtual center managed cluster with two hosts. I am checking DNS now to make sure all my proxies can resolve those hosts and VC. to date only my Veeam B&R server and target proxy server could resolve the VC and hosts at the target site. Do you know if anything changed in how the replication job does the snapshot removal on the replica VMs? I'm thinking it is initiated from the source proxies now instead of from the target proxy in the past? (just guessing here..)

[foggy]

Actually snapshot removal is initiated by VMware itself, Veeam B&R just issues a call to do that. Further investigation should be conducted with the help of support.

[saxie]

of course but veeam still has to log in to the VC to make that API call right? That is what I was referring to. I'll wait for support.

Thanks!

[foggy]

Communication with vCenter is performed through Veeam B&R management server using credentials specified when it was added to the console, so nothing has changed in this regard in Update 2. Keep us posted on how investigation goes.

[saxie]

Thanks

I just heard back from Support that this issue is a known issue in Update 2 and it is with R&D at the moment.

[wroberts]

Thanks Saxie. I just got in touch with tier 2 as well. Hopefully R&D will find and fix this quickly since I expect far more customers will begin having this issue as time goes on.

[wroberts]

Update:

Thanks to Veeam Support I'm back online.

The issue is due to a duplicate entry of the host in the database. Finding and clearing out the osbsolete duplicate host entry(s) in the database solved the problem for me.

If you are having the same problem, before a patch comes out, I recommend contacting Veeam support. They have a query to execute, after stopping all jobs backing up the database, which will remove any references to hosts which Veeam is still trying to connect to but are no longer valid. This can happen after you upgrade a host or vcenter server apparently.

-Will

[ronaldmercieca]

After updating both Office and DR Site proxies to Veeam B&R 8.0 Update 2, I'm having same issue with some jobs when applying retention policy.
I have 3 hosts in office, running ESXi 5.5 Update 2, with VMs residing on 2 SANS connected via iSCSI.
I have 1 host in DR site, also running ESXi 5.5 Update 2 and connected through WAN, with VMs residing on host HDDs.
At some instance, some jobs will complete successfully whilst others will fail with error -
Failed to apply retention policy for VM: XxXxXx_replicas Error: Cannot complete login due to an incorrect user name or password.
My support case number is 00912067