Virtual disks still attached to backup server

[jyarborough]

We are running Veeam Backup and Replication 4.1.1 and I am having a problem with some of our servers being backed up. There are some phantom snapshots that caused errors during the last backup job and now my backup server has 15 virtual hard disks attached to it. Can I safely dissassociate these or is there a recommended way of cleaning this up? I have the backup server offline right now. I'm a little concerned because of the servers that gave the snapshot errors had some major problems after I did a cold storage migration (all through the vSphere Client). It would not power on because it said that the parent disk had changed. Luckily I found a set of virtual disks that were left in the original location that seemed to have the most recent data and correct file sizes so I created a new virtual machine, added those disks and was back in business (after nearly driving my car off a cliff ). I'm afraid that the other servers are going to be the same way if I power them off but I will be sure to get an image backup using a third party tool like ghost or Acronis before I actually power off the machine.

Any ideas why this would have happened? Also note that it is only happening on our Windows 2000 virtual machines, none of the other machines are like this. Strange coincidence?

Thanks!

John Yarborough

[Gostev]

Hi John, do you have automount disabled on your Veeam Backup server?

[jyarborough]

No I do not. In fact, I noticed on one of the servers it would come up and say the disk was not formatted and to use it I needed to format it. This always made me a little uneasy but clicking cancel seemed to satisfy the request and the jobs completed successfully. I assume this should be disabled? And I also assume that is in the install guide?

[Gostev]

Run these commands in command prompt of Veeam Backup server:

Code: Select all

automount disable
automount scrub
exit

and then reboot.

[jyarborough]

I ran diskpart then ran those commands based on http://www.yellow-bricks.com/2008/02/11 ... -what-now/. However, the virtual disks are still attached to the Veeam Backup virtual machine. Should I just remove those from the virtual machine? I am little worried that I may have caused more damage than I am aware of based on that article. Sounds like this could have effected the VMFS volumes even. How can I tell if there is anything else that might need fixed because I didn't have automount disabled?

Also, is there a reference for this in your VBR documents? I need to go back and review the install guide I guess.

[McClane]

Yes, you can remove them from the backup machine. In fact, you should remove them, because the ESX can't consolidate snapshots any more as long as the disks are attached to another maschine. After the removal look for vm's with lots of snapshots in the filesystem. vSphere Client might not show them all.

[Gostev]

John, with automount disabled the damage to disk is impossible. This is well proven by years of VCB usage, where you need to connect VMFS volumes directly to backup server for VCB to work.

You can remove this disks, but overall it is normal to see these disks added to Veeam Backup server, because this is how Virtual Appliance mode is designed to work: backed up VM disks are hot-added to the backup server VM for direct access to disks data through ESX I/O stack.

[jyarborough]

Thanks for the help. I figured that was normal to see them while backups are running but not after all the jobs are completed. Love the product and can't wait to see SureBackup!

[jyarborough]

I am actually still seeing problems with this. I went through and cleaned up all the virtual disks that were attached to the Veeam server, then went in and took new snapshots so that the consolidation helpers would show up again, and then I deleted all the snapshots manually. I went in and confirmed that the datastores looks clean. Then I went in and tried to run a backup but it errors out with:

Removing snapshot RemoveSnapshot failed, snapshotRef "snapshot-2245", timeout "3600000" Unable to access file <unspecified filename> since it is locked Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually. Possible causes for snapshot removal failure: - Network connectivity issue, or vCenter server is too busy to serve the request - ESX host was unable to process snapshot removal request in a timely manner - Snapshot was already removed by another application BackupDisk failed Client error: The specified transports are not available. Available transports: [nbd]. Specified transports: [hotadd]. VMDK path: [[san-vol2] clientsrv1/clientsrv1.vmdk]. Server error: End of file

The only thing that has changed is that we are now backing up to a UNC path. Do I need to change from Virtual Appliance mode to Network?

[Vitaliy S.]

John,

Actually, no, because snapshot removal procedure doesn't depend on the backup mode you choose. Do you see this error at the end of the job or in the beginning?

If you see it at the end, please re-run the job, this will automatically remove the snapshot that was left due to connectivity issue.

Thank you!

[Gostev]

Do you see this on one VM, or on all VMs?

[jyarborough]

Changing from Virtual Appliance mode to vStorage API Network mode resolved the issue and now is allowing me to perform backups. The thing that made me look at this is the statement in the error:

Client error: The specified transports are not available. Available transports: [nbd]. Specified transports: [hotadd].

So does this mean that hotadd does not work when backing up a virtual machine sitting on a SAN through a virtual Veeam BR server sitting on the SAN to a CIFS share accessed through the Veeam server's guest OS?

So far I am only seeing this on the machines mentioned before (2 jobs each backing up 2 Windows 2000 servers) because I have not re-enabled any of the other jobs yet. I will re-enable one of the jobs that never had any problems before and report back.

[Gostev]

So does this mean that hotadd does not work when backing up a virtual machine sitting on a SAN through a virtual Veeam BR server sitting on the SAN to a CIFS share accessed through the Veeam server's guest OS?

This is probably most typical scenario, and it definitely works. But there are a few things that can prevent this from working, like IDE disks on backed up VM for instance.

Also, please check what version are you using from Help | About.

[jyarborough]

The VM's do not have any IDE hard disks, all SCSI. The CD-ROM drives are set to "Client Device" and in "Passthrough IDE (recommended)" mode however. I don't think this should effect anything though. I'm not sure what you mean by checking the version? We are on ESX 4.0 Update 1 (no additional patches), all of our VM's are at hardware level 7, all VM tools are up to date, Veeam Backup is at 4.1.1, anything else?

Also, I just ran one of the jobs that had been working fine when going to a local drive (VMDK sitting on the SAN) on the backup server. The only change I made is pointing it to the UNC path and moving the previous backup files there. I see it go through and:

- create a snapshot
- then it gets the .vmx file and a couple of others
- then it reconfigured the backup server to add the -flat.vmdk file
- then it sits there for a few minutes with a "Processing rate" of 0KB/s
- then it tries to remove the snapshot and fails with:

RemoveSnapshot failed, snapshotRef "snapshot-2256", timeout "3600000"
Unable to access file <unspecified filename> since it is locked
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually.
Possible causes for snapshot removal failure:
- Network connectivity issue, or vCenter server is too busy to serve the request
- ESX host was unable to process snapshot removal request in a timely manner
- Snapshot was already removed by another application

BackupDisk failed
Client error: The specified transports are not available. Available transports: [nbd]. Specified transports: [hotadd]. VMDK path: [[san-vol1] TS01/TS01.vmdk].

Server error: End of file

Now I have to go and manually do the snapshot dance to clean up the hidden consolidation helpers. It's like it can't access the -flat.vmdk files when it is in Virtual Appliance mode but it works fine in Network Mode. Can I send any logs that would help?

A little more clarification on our setup is:

4 ESX hosts
2 FC SAN's where all of our VM's are located including the Veeam Backup server
1 NAS where store backups using CIFS/UNC paths

[Gostev]

Yes, it would be best if you open support case and provide all logs from Help | Support information for investigation.

[TrevorBell]

Anton -:

automount disable
automount scrub
exit

not sure what automount scrub does ? i just automount disable then exit

[Gostev]

This removes existing mount points which are no longer in the system. This is needed if you do the procedure AFTER running backup jobs. If you disable automount before deploying Veeam Backup, then you don't need this command. But it won't hurt in all cases

[jyarborough]

Just wanted to post back with an update. I opened a case with Veeam support and we decided to run updates on ESX before doing anything else. I installed vCenter Update Manager and ran all patches against my ESX hosts and now Virtual Appliance mode works again. Not sure if this is actually what resolved the problem but I am now able to use it again so I'm not asking anymore questions

I did notice though that even with diskpart/automount disable/automount scrub I am still seeing a prompt asking me to format some of the disks for one particular Windows 2000 server. Not sure why this happens but would be interested in any thoughts.

Thanks again for all the help!

[Gostev]

John, thanks very much for posting an update/resolution for the future readers.

[icebun]

Hi jyarborough

Just to let you know that I have exactly the same issue and workaround.

In my case, some of the smaller VMs backup up OK with the the Virtual Appliance mode but not the larger ones with multiple VMDKs.

Sticking to Network mode on via Storage API works a treat.

[rgs3]

I have the exact same issue..Veeam support told me to contact VMware

[tfleener]

Did you ever get this resolved.

I am experiencing the same symptons, and have contacted tech support and they do not seem to have any idea why this is happening.

The scenario is the same
create a snapshot
- then it gets the .vmx file and a couple of others
- then it reconfigured the backup server to add the -flat.vmdk file
- then it sits there for a few minutes with a "Processing rate" of 0KB/s
- then it tries to remove the snapshot and fails with:

We are fiber cards, Vsphere 4.0 Upd 1, running VMWare in appliance mode. Network mode works fine, however applilcance or SAN mode does not work.

We have not IDE drives, we are pretty generic.

Tom

[icebun]

Hey Tom,

My current setup is a VM running VBR 64 bit.

For all VMs that have VMDKs under 1TB, I use Virtual Appliance mode, for those over I use Network Mode (vStorage API). As Gostev pointed out, this is a limitation of the API from VMware and not Veeam.

Incidentally I logged a seperate call to VMware just to find out about the next API release date and possible fixes but the call is still open.

Before I abided by the rules of the API, I was forever getting disks stuck on the VBR with the invisible snapshot situation on the guest VM. Once the disks have been removed, I would have to take another snapshot on the VM to reveal the first one used by Veeam before deleting them all.

I have also dedicated a large LUN just for the VBR server itself so I don't fall foul of running out of "swap space" while the disks get mounted during a backup session.

So far as a result, I have not had any issues regarding attached disks from other VMs.

I do plan to install VBR on a physical server attached to a tape library so that I can get archives for Monthly and Yearly offsite. Unless of course someone can advise me otherwise.