Comprehensive data protection for all workloads
withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Hot add didn't work for one server

Post by withanh » Jan 12, 2010 3:12 pm

Status window shows:

Code: Select all

Backing up file "[Golf_VMFS-iSCSI01] vanc-graphics01/vanc-graphics01-flat.vmdk"
BackupDisk failed
Client error: The specified transports are not available. Available transports: [nbd]. Specified transports: [hotadd]. VMDK path: [[Golf_VMFS-iSCSI01] vanc-graphics01/vanc-graphics01.vmdk].

Server error: End of file
But SAN/NBD with Changed Block Tracking works fine. The other servers I backup work fine in VA/HOTADD mode...

Ideas?
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 12, 2010 3:19 pm

Ahh, I can't believe you did not search for the error first ;) there is couple of existing threads about this error with my answers.

Most common reasons: either this VM has IDE disks, or [Golf_VMFS-iSCSI01] storage is not connected to ESX server running Veeam Backup VM (or ESX does not have read permissions on this LUN). If this is not the case, please contact our support for full logs analysis.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 12, 2010 3:26 pm

You're right, shame on me :oops:

This was a P2V, but I think the only IDE is the CDRom, the hard drives are all SCSI, the C:\ is LSI Logic Parallel and the D:\ is Paravirtual.

I did notice that the CDRom is set for Emulate IDE instead of Passthrough IDE, I will change that to Passthrough and see if that helps me.

Thanks Anton
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 12, 2010 3:32 pm

So in my defense, I did search for "HOTADD not working" and found this thread http://www.veeam.com/forums/viewtopic.p ... 200&p=9328 but that talks about the hard drives being IDE not the CDRom, all of my HD's are SCSI...
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 12, 2010 3:32 pm

I *think* I heard something about P2V and hot add failures, but I am not sure what was the resolution there. Our support should know. I would definitely appreciate if you update this thread with the resolution once you make this working.

Not sure if CD-ROM can be causing this, but who knows.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 12, 2010 3:33 pm

Of course I will :D
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 12, 2010 5:30 pm

Case # 518834
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 1:49 pm

Support is struggling with this one a bit. The first machine that has this issue they say is a corrupted VM and shows multiple .VMDK files with the same name and size. They also say that the HOTADD only supports drives smaller than 1TiB. This VM has a 1TiB data drive, so that's fine. I'll also work with VMware support on the possible corrupted disk.

But then another VM failed, all SCSI disks, largest drive is 675GiB so we're nowhere near the 1TiB limit. She said "Most likely the issue you are experiencing with this machine is not the same", even though nearly everything is the same with it. It was originally a P2V and has never successfully backed up. I have not tried other backup methods yet (SAN/NBD with CBT) because I felt it was the same issue and thought that when one was solved, the other would be as well. Guess not.

Then for the coup de gras, about 50% of my VMs did not back up last night, all configured for HOTADD and all with SCSI only disks, and they have been working all along, and now they didn't.

As much as I love this product and think it does a great job, the Level 1 support is a little lacking. I praised the support earlier on another thread, even defending Veeam support, and thinking back, the great experience was after I complained and got it escalated and someone else involved, the issue then was with Level 1. It's like a roller coaster, support is great, support is crappy, support is great, support is crappy. Ugh!
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 15, 2010 3:13 pm

Makes sense... I pulled up the release notes for vStorage API 1.1.1 and found this in the Known Issues section:
• HotAdd advanced transport cannot mount large (1 TB) VMDK files.
• With HotAdd transport, full backup of a virtual machine fails when multiple disks have the same name, as they could if on different datastores.

Also, for that other VM you do not have resolution for so far - check if some virtual disks are possibly set to "independent"?

By level 1, you mean those folks who take the phone call?

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 3:23 pm

I don't know entirely how your support structure works, but the way I generally see support is similar to this:

Customer care - receive phone call, triage case, determine severity, direct call to an appropriate technician
Level 1 - Support technician
Level 2 - Senior support technician
Level 3 - Supervising support technician
Level 4 - Development

No, none of my disks are set to independent. The first two machines have never backed up, they were both P2V conversion last weekend. The other machines are a mixture of native VM and P2V, but have always backed up with VA/HOTADD prior to last night.
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 3:54 pm

I just re-read your post,
Gostev wrote:With HotAdd transport, full backup of a virtual machine fails when multiple disks have the same name, as they could if on different datastores.
The C:\ and the D:\ are on separate datastores, they probably both have the same .VMDK name. So that could be part of it as well, although we are still up against the 1TiB limit on that machine. I'm going to re-do that to a 750TiB drive this weekend and see if that helps. The other machine, though, is all on the same datastore, so it definitely has different .VMDK names, and my Exchange is on 3 separate data stores (C:\, D:\Mailstores, E:\Logs) for performance. That has backed up with VA/HOTADD all week, it only failed last night.

It's strange for sure! But now @benmilligan is following the case, assisting the tech that is assigned to it, so hopefully with his coaching her, things will move along...
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Vitaliy S.
Product Manager
Posts: 22965
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hot add didn't work for one server

Post by Vitaliy S. » Jan 15, 2010 4:06 pm

Darhl,

Probably I can explain you why your jobs were successfull for some time and then some of them failed, the reason for that is that you have this kind of error in the job logs:

Code: Select all

Failed to allocate SCSI target ID
So I wonder how many concurrent jobs do you run using Virtual Applicance mode? I'm asking this, cause there was a known issue for VCB Hot-Add mode with lack of SCSI IDs, and the resolution for this was to add more SCSI controllers to the VM running backup server, I believe it should also help in your situation.

Thank you!

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 4:32 pm

Hi Vitaliy,

At most I run 3 concurrent backup jobs. How would I add additional SCSI controllers without adding virtual disks?

I think you may have it on the issue, when I look at the settings for the Veeam server, I see 14 attached hard disks, and looking at them, they are duplicate hard drives for nas01 & graphics01 over and over. Why is the Veeam VM not releasing those? They are also marked as independent and non-persistent. See screenshot at http://www.flickr.com/photos/45779723@N04/4276943546/

Thanks,

Darhl
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 15, 2010 4:52 pm

Darhl, did you disable automount on Veeam Backup proxy (just like for VCB)?
These disks are managed by vStorage API, not Veeam code. But it is normal to see them, this is how Hot Add backups work.

If you are indeed running concurrent jobs, then I think we have just nailed down the issue. I think there are instructions on adding additional SCSI controllers, see the article that Vitaly had referenced above - just add new small disk to VM, and place it on a separate storage controller.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 4:54 pm

Sure did, will double check, yes it is disabled:
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\administrator>diskpart

Microsoft DiskPart version 6.1.7600
Copyright (C) 1999-2008 Microsoft Corporation.
On computer: VANC-VEEAM01

DISKPART> automount

Automatic mounting of new volumes disabled.

DISKPART>
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Vitaliy S.
Product Manager
Posts: 22965
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hot add didn't work for one server

Post by Vitaliy S. » Jan 15, 2010 5:08 pm

Darhl,

Do you see anything in common between those disks? Are you able to reproduce the issue by removing all those disks, then running jobs again and checking whether you see those disks appearing once again or not, anyway please keep up working with our support team, I'm sure together we will be able to find the root cause for your issue.

Thank you!

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 6:22 pm

The disks are not the same .VMDK name, they are numbered but not sequentially, it doesn't make any sense. I will forward the info to support.

I will also do as you suggest and remove the duplicate drives. One thing that is weird is HD1 is the boot drive, but was moved to SCSI 0:3. It's just weird. I'll probably move that back to 0:0 where it belongs.
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

ceebee
Influencer
Posts: 17
Liked: never
Joined: Dec 24, 2009 2:39 am
Full Name: Chet
Contact:

Re: Hot add didn't work for one server

Post by ceebee » Jan 18, 2010 4:29 am

I have the same exact issue and i am being told to get in touch with VMWare as this is not a Veeam issue! Fantastic!
I am working with Ben M. on this issue but this email was from Stanislav.
This is getting bloody ridiculous!!! :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops:
Not only the product is buggy and does not work as touted but the the icing on the cake is the support. If someone wants to write a book on how to setup a complete cluster F***** of a support then they need to call Veeam.
I am about done with my patience.

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 18, 2010 11:31 am

ceebee wrote:I have the same exact issue
Hello Chet, what issue are you talking about specifically? The original issue discussed in this thread were indeed caused by some known vStorage API limitations (I've posted a quote from VMware documentation on the previous page).

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 18, 2010 1:54 pm

Chet,

Were your backup issues caused by some corruption in a P2V conversion? All details point to that as the problem here. That is VMware's job to resolve, not Veeam's job to resolve.

Anton, I worked with Stanislav over the weekend and we've got most of this straightened out. The last piece that is missing is the part I just mentioned above. When I am finished with VMware today, I will post back what their resolution was for the P2V conversions.

h
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 19, 2010 8:32 am

Darhl, do you have any updates? I am travelling today, and it is hard for me to check with our support directly.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 27, 2010 2:56 pm

The reason the HOTADD/Virtual Appliance mode was not working for some VMs is because I have the disks on different data stores. This results in the .VMDKs having the same name. When Veeam then tries to HOTADD the disks, it fails because VMware does not allow disks with the same name.

The filename of disk 1 is \\datastore1\servername\diskname.vmdk
The filename of disk 2 is \\datastore2\servername\diskname.vmdk

It seems that it does not see the datastore name as part of the path for the file, so it's looking at the \servername\diskname.vmdk and they certainly are the same at that point.

h
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

sbrumlwy
Novice
Posts: 5
Liked: never
Joined: Feb 02, 2010 10:55 am
Full Name: Steve Brumley
Contact:

Re: Hot add didn't work for one server

Post by sbrumlwy » Feb 04, 2010 3:40 am

I had this same issue and none of the above suggestions worked. Everything checked out ok. I solved the problem by migrating the Veeam host vm to a data store with a 4mb block size. The VM that was failing to backup had a 500GB .vmdk disk. The datastore that housed the Veeam VM had a 1 MB block size.

Why? I have no idea. But maybe it helps someone else. And for the record, VMware Data recovery choked on the same VM, in the same way.

Steven

Vitaliy S.
Product Manager
Posts: 22965
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hot add didn't work for one server

Post by Vitaliy S. » Feb 04, 2010 10:51 am

Hello Steve,

The issue seems to be related to VMFS block size formatting and maximum file size for each type:

• 1MB block size – 256GB maximum file size
• 2MB block size – 512GB maximum file size
• 4MB block size – 1024GB maximum file size
• 8MB block size – 2048GB maximum file size

That explains why you were successfull after moving Veeam Backup and Replication VM to another datastore with 4MB block size of VMFS, because the size of the Hot-Added virtual disk was bigger than maximum file size supported for 1MB block formatted datastore.

sbrumlwy
Novice
Posts: 5
Liked: never
Joined: Feb 02, 2010 10:55 am
Full Name: Steve Brumley
Contact:

Re: Hot add didn't work for one server

Post by sbrumlwy » Feb 04, 2010 2:47 pm

Yes Vitaliy, thanks for your response. I guess my confusion was that I assumed the hot added disk never actually lives on the data store with the smaller block size, so I didn't think that would be a problem. But evidently something happens during hot add that makes the disk think it on the smaller data store (which actually doesn't have enough space to hold it, even if it had the right block size). The data store with the Veeam VM is only a 200GB data store, so I never considered a larger block size for it. I guess it would be a good practice to go ahead and use larger block sizes for data stores no matter what. I am new to this and learning.

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Feb 04, 2010 3:09 pm

Now when you mention it is 200GB datastore and 500GB VMDK, I recognize one known issue we have documented in the release notes :
• If ESX datastore hosting Veeam Backup VM has less free space than biggest virtual disk of backed up VM, Virtual Appliance mode fails with the insufficient disk space error
So looks like this issue has nothing to do with VMFS block size... :D

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Feb 04, 2010 3:27 pm

I have noticed a different issue with snapshots as well. I had a VM that had a 25GiB OS drive and a 250GiB Data drive. It was running on a VMFS partition with 1MB blocks (256GiB max file size). It kept failing during the snapshot operation which didn't make sense to me because I stayed under the limits of the max file size. After I created a new partition with 2MB blocks (512GiB max file size) the snapshots (and subsequent backup operation) started working properly. The only explanation I could find is because the total machine size was 275GiB which was over the 256GiB max.

h
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 24784
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Feb 04, 2010 3:38 pm

Darhl, yep this issue with snapshots was reported before by some customers after they upgraded to vSphere, see this thread for example.

sbrumlwy
Novice
Posts: 5
Liked: never
Joined: Feb 02, 2010 10:55 am
Full Name: Steve Brumley
Contact:

Re: Hot add didn't work for one server

Post by sbrumlwy » Feb 04, 2010 3:44 pm

That sounds reasonable Gostev. But the error msg and logs did not reference "insufficient disk space" in any way. It just said it was unable to power on the scsi disk, and that the hotadd transport was not available. An "insufficent disk space" error may have saved me some time! :lol:
Gostev wrote:Now when you mention it is 200GB datastore and 500GB VMDK, I recognize one known issue we have documented in the release notes : So looks like this issue has nothing to do with VMFS block size... :D

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Feb 04, 2010 3:44 pm

Yeah, I found that out way early on when we first installed vSphere and Veeam, it hasn't been an issue for me :)
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Post Reply

Who is online

Users browsing this forum: john_wood and 18 guests