Replication job fails over nvram file

[Jonathan]

Veeam B&R 4 has been running fine for months at our shop, but as of yesterday, I have one replication job failing.

It's a freshly spawned Windows Server 2003 X64 VM. Quick details: 2 vCPU, 2GB, 2 x 20GB VMDK (thin), vmfs on fc array, using vSphere.

The replication job contains six similar VMs, which still replicate just fine. Most of them reside on the same source VMFS and the target is obviously the same as well. So we can rule out any authentication issues, insufficient disk space, network or fibre channel hiccups.

It seems to get stuck in the early stages at the point where it tries to copy the nvram file.
Here's the job log:

----------------------

1 of 7 files processed

Total VM size: 40,00 GB
Processed size: 3,23 KB
Processing rate: 0 KB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 3-6-2010 08:34:40
End time: 3-6-2010 08:35:33
Duration: 0:00:52

Replicating file "[EVA_VMFS06] Rincewind/Rincewind.nvram"
BackupFile failed
Client error: External FIB refers to non-existing file. FIB: [Rincewind.nvram]. File: [RelativePath://0x0000000c=Rincewind.nvram;].
Cannot backup file in the service console mode. File: [nfc://conn:[FQDN_of_virtual_center_host],nfchost:host-8,stg:datastore-3387@Rincewind/Rincewind.nvram]. VBK: [veeamfs:*:/Rincewind.nvram@/vmfs/volumes/4ad48728-3ffa8970-feea-0019bbca9f82/VeeamBackup/Rincewind(vm-17754)/replica.vbk]. RBK: [veeamfs:6:/Rincewind.nvram@/vmfs/volumes/4ad48728-3ffa8970-feea-0019bbca9f82/VeeamBackup/Rincewind(vm-17754)/2010-06-02T040056.vrb].

Server error: End of file
Backup request cannot be processed. Source file path: [nfc://conn:[FQDN_of_virtual_center_host],nfchost:host-8,stg:datastore-3387@Rincewind/Rincewind.nvram].

---------------------

I should have known. Naming a VM after a wizard who's like a magnet to utter mayhem was asking for trouble.

Any ideas?

[Vitaliy S.]

Hello Jonathan,

It seems like this file is missing on your destination target for some reason, try copying Rincewind.nvram to datastore-3387@Rincewind/ folder and re-run the job. If this doesn't help please feel free to open a support ticket with our technical team directly.

Thanks!

[Jonathan]

I can't just copy the file. Nvram files are locked when the VM is powered on, so vSphere client won't let me take a copy.
I've tried this with several VMs and it just won't work.

Any suggestions?

[Vitaliy S.]

Jonathan, have you tried doing it using Datastore Browser, cause it works for me if i try to download that file from a running VM.

[Jonathan]

Yep. Using the datastore browser, it says "Expected put message. Got ERROR."
The user account has the Administrator role on the datacenter, so it's got every right to copy this file.

Tried a vSphere client session to one of the hosts directly using the root account: doesn't work either.
Logging on to one of the vSphere hosts using ssh, dropping into root and trying to transfer the nvram file using ftp, it also fails.

This behaviour is across the board, for all of my VMs, so you would think this is by design.

[Vitaliy S.]

It seems like I've found out why you're not able to download it, please have a look at VMware's KB article:
http://kb.vmware.com/kb/1019286

If copying nvram file to your destination target doesn't help, please refer to our support team directly for assistance.

Thanks!

[Jonathan]

Thanks Vitaliy! That seems to work.

Strangely enough, the job continued to fail several times over the weekend, stating:
Replica VM file '/vmfs/volumes/4ad48728-3ffa8970-feea-0019bbca9f82/VeeamBackup/Rincewind(vm-17754)/replica.vbk' is missing from destination host '[fqdn_of_destination_host]'.

But the last automatic retry this morning actually succeeded without my intervention!
If the next scheduled backup runs without errors, I think we can call this one solved.

[Vitaliy S.]

Jonathan, good to hear that. Keep an eye on that replication job, and contact our support team in case you are still having the issue. Thanks!