Hot add didn't work for one server

[withanh]

Status window shows:

Code: Select all

Backing up file "[Golf_VMFS-iSCSI01] vanc-graphics01/vanc-graphics01-flat.vmdk"
BackupDisk failed
Client error: The specified transports are not available. Available transports: [nbd]. Specified transports: [hotadd]. VMDK path: [[Golf_VMFS-iSCSI01] vanc-graphics01/vanc-graphics01.vmdk].

Server error: End of file

But SAN/NBD with Changed Block Tracking works fine. The other servers I backup work fine in VA/HOTADD mode...

Ideas?

[Gostev]

Ahh, I can't believe you did not search for the error first there is couple of existing threads about this error with my answers.

Most common reasons: either this VM has IDE disks, or [Golf_VMFS-iSCSI01] storage is not connected to ESX server running Veeam Backup VM (or ESX does not have read permissions on this LUN). If this is not the case, please contact our support for full logs analysis.

[withanh]

You're right, shame on me

This was a P2V, but I think the only IDE is the CDRom, the hard drives are all SCSI, the C:\ is LSI Logic Parallel and the D:\ is Paravirtual.

I did notice that the CDRom is set for Emulate IDE instead of Passthrough IDE, I will change that to Passthrough and see if that helps me.

Thanks Anton

[withanh]

So in my defense, I did search for "HOTADD not working" and found this thread http://www.veeam.com/forums/viewtopic.p ... 200&p=9328 but that talks about the hard drives being IDE not the CDRom, all of my HD's are SCSI...

[Gostev]

I *think* I heard something about P2V and hot add failures, but I am not sure what was the resolution there. Our support should know. I would definitely appreciate if you update this thread with the resolution once you make this working.

Not sure if CD-ROM can be causing this, but who knows.

[withanh]

Of course I will

[withanh]

Case # 518834

[withanh]

Support is struggling with this one a bit. The first machine that has this issue they say is a corrupted VM and shows multiple .VMDK files with the same name and size. They also say that the HOTADD only supports drives smaller than 1TiB. This VM has a 1TiB data drive, so that's fine. I'll also work with VMware support on the possible corrupted disk.

But then another VM failed, all SCSI disks, largest drive is 675GiB so we're nowhere near the 1TiB limit. She said "Most likely the issue you are experiencing with this machine is not the same", even though nearly everything is the same with it. It was originally a P2V and has never successfully backed up. I have not tried other backup methods yet (SAN/NBD with CBT) because I felt it was the same issue and thought that when one was solved, the other would be as well. Guess not.

Then for the coup de gras, about 50% of my VMs did not back up last night, all configured for HOTADD and all with SCSI only disks, and they have been working all along, and now they didn't.

As much as I love this product and think it does a great job, the Level 1 support is a little lacking. I praised the support earlier on another thread, even defending Veeam support, and thinking back, the great experience was after I complained and got it escalated and someone else involved, the issue then was with Level 1. It's like a roller coaster, support is great, support is crappy, support is great, support is crappy. Ugh!

[Gostev]

Makes sense... I pulled up the release notes for vStorage API 1.1.1 and found this in the Known Issues section:
• HotAdd advanced transport cannot mount large (1 TB) VMDK files.
• With HotAdd transport, full backup of a virtual machine fails when multiple disks have the same name, as they could if on different datastores.

Also, for that other VM you do not have resolution for so far - check if some virtual disks are possibly set to "independent"?

By level 1, you mean those folks who take the phone call?

[withanh]

I don't know entirely how your support structure works, but the way I generally see support is similar to this:

Customer care - receive phone call, triage case, determine severity, direct call to an appropriate technician
Level 1 - Support technician
Level 2 - Senior support technician
Level 3 - Supervising support technician
Level 4 - Development

No, none of my disks are set to independent. The first two machines have never backed up, they were both P2V conversion last weekend. The other machines are a mixture of native VM and P2V, but have always backed up with VA/HOTADD prior to last night.

[withanh]

I just re-read your post,

With HotAdd transport, full backup of a virtual machine fails when multiple disks have the same name, as they could if on different datastores.

The C:\ and the D:\ are on separate datastores, they probably both have the same .VMDK name. So that could be part of it as well, although we are still up against the 1TiB limit on that machine. I'm going to re-do that to a 750TiB drive this weekend and see if that helps. The other machine, though, is all on the same datastore, so it definitely has different .VMDK names, and my Exchange is on 3 separate data stores (C:\, D:\Mailstores, E:\Logs) for performance. That has backed up with VA/HOTADD all week, it only failed last night.

It's strange for sure! But now @benmilligan is following the case, assisting the tech that is assigned to it, so hopefully with his coaching her, things will move along...

[Vitaliy S.]

Darhl,

Probably I can explain you why your jobs were successfull for some time and then some of them failed, the reason for that is that you have this kind of error in the job logs:

Code: Select all

Failed to allocate SCSI target ID

So I wonder how many concurrent jobs do you run using Virtual Applicance mode? I'm asking this, cause there was a known issue for VCB Hot-Add mode with lack of SCSI IDs, and the resolution for this was to add more SCSI controllers to the VM running backup server, I believe it should also help in your situation.

Thank you!

[withanh]

Hi Vitaliy,

At most I run 3 concurrent backup jobs. How would I add additional SCSI controllers without adding virtual disks?

I think you may have it on the issue, when I look at the settings for the Veeam server, I see 14 attached hard disks, and looking at them, they are duplicate hard drives for nas01 & graphics01 over and over. Why is the Veeam VM not releasing those? They are also marked as independent and non-persistent. See screenshot at http://www.flickr.com/photos/45779723@N04/4276943546/

Thanks,

Darhl

[Gostev]

Darhl, did you disable automount on Veeam Backup proxy (just like for VCB)?
These disks are managed by vStorage API, not Veeam code. But it is normal to see them, this is how Hot Add backups work.

If you are indeed running concurrent jobs, then I think we have just nailed down the issue. I think there are instructions on adding additional SCSI controllers, see the article that Vitaly had referenced above - just add new small disk to VM, and place it on a separate storage controller.

[withanh]

Sure did, will double check, yes it is disabled:

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\administrator>diskpart

Microsoft DiskPart version 6.1.7600
Copyright (C) 1999-2008 Microsoft Corporation.
On computer: VANC-VEEAM01

DISKPART> automount

Automatic mounting of new volumes disabled.

DISKPART>

[Vitaliy S.]

Darhl,

Do you see anything in common between those disks? Are you able to reproduce the issue by removing all those disks, then running jobs again and checking whether you see those disks appearing once again or not, anyway please keep up working with our support team, I'm sure together we will be able to find the root cause for your issue.

Thank you!

[withanh]

The disks are not the same .VMDK name, they are numbered but not sequentially, it doesn't make any sense. I will forward the info to support.

I will also do as you suggest and remove the duplicate drives. One thing that is weird is HD1 is the boot drive, but was moved to SCSI 0:3. It's just weird. I'll probably move that back to 0:0 where it belongs.

[ceebee]

I have the same exact issue and i am being told to get in touch with VMWare as this is not a Veeam issue! Fantastic!
I am working with Ben M. on this issue but this email was from Stanislav.
This is getting bloody ridiculous!!!
Not only the product is buggy and does not work as touted but the the icing on the cake is the support. If someone wants to write a book on how to setup a complete cluster F***** of a support then they need to call Veeam.
I am about done with my patience.

[Gostev]

I have the same exact issue

Hello Chet, what issue are you talking about specifically? The original issue discussed in this thread were indeed caused by some known vStorage API limitations (I've posted a quote from VMware documentation on the previous page).

[withanh]

Chet,

Were your backup issues caused by some corruption in a P2V conversion? All details point to that as the problem here. That is VMware's job to resolve, not Veeam's job to resolve.

Anton, I worked with Stanislav over the weekend and we've got most of this straightened out. The last piece that is missing is the part I just mentioned above. When I am finished with VMware today, I will post back what their resolution was for the P2V conversions.

h

[Gostev]

Darhl, do you have any updates? I am travelling today, and it is hard for me to check with our support directly.

[withanh]

The reason the HOTADD/Virtual Appliance mode was not working for some VMs is because I have the disks on different data stores. This results in the .VMDKs having the same name. When Veeam then tries to HOTADD the disks, it fails because VMware does not allow disks with the same name.

The filename of disk 1 is \\datastore1\servername\diskname.vmdk
The filename of disk 2 is \\datastore2\servername\diskname.vmdk

It seems that it does not see the datastore name as part of the path for the file, so it's looking at the \servername\diskname.vmdk and they certainly are the same at that point.

h

[sbrumlwy]

I had this same issue and none of the above suggestions worked. Everything checked out ok. I solved the problem by migrating the Veeam host vm to a data store with a 4mb block size. The VM that was failing to backup had a 500GB .vmdk disk. The datastore that housed the Veeam VM had a 1 MB block size.

Why? I have no idea. But maybe it helps someone else. And for the record, VMware Data recovery choked on the same VM, in the same way.

Steven

[Vitaliy S.]

Hello Steve,

The issue seems to be related to VMFS block size formatting and maximum file size for each type:

• 1MB block size – 256GB maximum file size
• 2MB block size – 512GB maximum file size
• 4MB block size – 1024GB maximum file size
• 8MB block size – 2048GB maximum file size

That explains why you were successfull after moving Veeam Backup and Replication VM to another datastore with 4MB block size of VMFS, because the size of the Hot-Added virtual disk was bigger than maximum file size supported for 1MB block formatted datastore.

[sbrumlwy]

Yes Vitaliy, thanks for your response. I guess my confusion was that I assumed the hot added disk never actually lives on the data store with the smaller block size, so I didn't think that would be a problem. But evidently something happens during hot add that makes the disk think it on the smaller data store (which actually doesn't have enough space to hold it, even if it had the right block size). The data store with the Veeam VM is only a 200GB data store, so I never considered a larger block size for it. I guess it would be a good practice to go ahead and use larger block sizes for data stores no matter what. I am new to this and learning.

[Gostev]

Now when you mention it is 200GB datastore and 500GB VMDK, I recognize one known issue we have documented in the release notes :

• If ESX datastore hosting Veeam Backup VM has less free space than biggest virtual disk of backed up VM, Virtual Appliance mode fails with the insufficient disk space error

So looks like this issue has nothing to do with VMFS block size...

[withanh]

I have noticed a different issue with snapshots as well. I had a VM that had a 25GiB OS drive and a 250GiB Data drive. It was running on a VMFS partition with 1MB blocks (256GiB max file size). It kept failing during the snapshot operation which didn't make sense to me because I stayed under the limits of the max file size. After I created a new partition with 2MB blocks (512GiB max file size) the snapshots (and subsequent backup operation) started working properly. The only explanation I could find is because the total machine size was 275GiB which was over the 256GiB max.

h

[Gostev]

Darhl, yep this issue with snapshots was reported before by some customers after they upgraded to vSphere, see this thread for example.

[sbrumlwy]

That sounds reasonable Gostev. But the error msg and logs did not reference "insufficient disk space" in any way. It just said it was unable to power on the scsi disk, and that the hotadd transport was not available. An "insufficent disk space" error may have saved me some time!

Now when you mention it is 200GB datastore and 500GB VMDK, I recognize one known issue we have documented in the release notes : So looks like this issue has nothing to do with VMFS block size...

[withanh]

Yeah, I found that out way early on when we first installed vSphere and Veeam, it hasn't been an issue for me