Comprehensive data protection for all workloads
withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Hot add didn't work for one server

Post by withanh » Jan 12, 2010 3:12 pm

Status window shows:

Code: Select all

Backing up file "[Golf_VMFS-iSCSI01] vanc-graphics01/vanc-graphics01-flat.vmdk"
BackupDisk failed
Client error: The specified transports are not available. Available transports: [nbd]. Specified transports: [hotadd]. VMDK path: [[Golf_VMFS-iSCSI01] vanc-graphics01/vanc-graphics01.vmdk].

Server error: End of file
But SAN/NBD with Changed Block Tracking works fine. The other servers I backup work fine in VA/HOTADD mode...

Ideas?
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 23608
Liked: 3115 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 12, 2010 3:19 pm

Ahh, I can't believe you did not search for the error first ;) there is couple of existing threads about this error with my answers.

Most common reasons: either this VM has IDE disks, or [Golf_VMFS-iSCSI01] storage is not connected to ESX server running Veeam Backup VM (or ESX does not have read permissions on this LUN). If this is not the case, please contact our support for full logs analysis.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 12, 2010 3:26 pm

You're right, shame on me :oops:

This was a P2V, but I think the only IDE is the CDRom, the hard drives are all SCSI, the C:\ is LSI Logic Parallel and the D:\ is Paravirtual.

I did notice that the CDRom is set for Emulate IDE instead of Passthrough IDE, I will change that to Passthrough and see if that helps me.

Thanks Anton
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 12, 2010 3:32 pm

So in my defense, I did search for "HOTADD not working" and found this thread http://www.veeam.com/forums/viewtopic.p ... 200&p=9328 but that talks about the hard drives being IDE not the CDRom, all of my HD's are SCSI...
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 23608
Liked: 3115 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 12, 2010 3:32 pm

I *think* I heard something about P2V and hot add failures, but I am not sure what was the resolution there. Our support should know. I would definitely appreciate if you update this thread with the resolution once you make this working.

Not sure if CD-ROM can be causing this, but who knows.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 12, 2010 3:33 pm

Of course I will :D
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 12, 2010 5:30 pm

Case # 518834
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 1:49 pm

Support is struggling with this one a bit. The first machine that has this issue they say is a corrupted VM and shows multiple .VMDK files with the same name and size. They also say that the HOTADD only supports drives smaller than 1TiB. This VM has a 1TiB data drive, so that's fine. I'll also work with VMware support on the possible corrupted disk.

But then another VM failed, all SCSI disks, largest drive is 675GiB so we're nowhere near the 1TiB limit. She said "Most likely the issue you are experiencing with this machine is not the same", even though nearly everything is the same with it. It was originally a P2V and has never successfully backed up. I have not tried other backup methods yet (SAN/NBD with CBT) because I felt it was the same issue and thought that when one was solved, the other would be as well. Guess not.

Then for the coup de gras, about 50% of my VMs did not back up last night, all configured for HOTADD and all with SCSI only disks, and they have been working all along, and now they didn't.

As much as I love this product and think it does a great job, the Level 1 support is a little lacking. I praised the support earlier on another thread, even defending Veeam support, and thinking back, the great experience was after I complained and got it escalated and someone else involved, the issue then was with Level 1. It's like a roller coaster, support is great, support is crappy, support is great, support is crappy. Ugh!
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 23608
Liked: 3115 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 15, 2010 3:13 pm

Makes sense... I pulled up the release notes for vStorage API 1.1.1 and found this in the Known Issues section:
• HotAdd advanced transport cannot mount large (1 TB) VMDK files.
• With HotAdd transport, full backup of a virtual machine fails when multiple disks have the same name, as they could if on different datastores.

Also, for that other VM you do not have resolution for so far - check if some virtual disks are possibly set to "independent"?

By level 1, you mean those folks who take the phone call?

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 3:23 pm

I don't know entirely how your support structure works, but the way I generally see support is similar to this:

Customer care - receive phone call, triage case, determine severity, direct call to an appropriate technician
Level 1 - Support technician
Level 2 - Senior support technician
Level 3 - Supervising support technician
Level 4 - Development

No, none of my disks are set to independent. The first two machines have never backed up, they were both P2V conversion last weekend. The other machines are a mixture of native VM and P2V, but have always backed up with VA/HOTADD prior to last night.
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 3:54 pm

I just re-read your post,
Gostev wrote:With HotAdd transport, full backup of a virtual machine fails when multiple disks have the same name, as they could if on different datastores.
The C:\ and the D:\ are on separate datastores, they probably both have the same .VMDK name. So that could be part of it as well, although we are still up against the 1TiB limit on that machine. I'm going to re-do that to a 750TiB drive this weekend and see if that helps. The other machine, though, is all on the same datastore, so it definitely has different .VMDK names, and my Exchange is on 3 separate data stores (C:\, D:\Mailstores, E:\Logs) for performance. That has backed up with VA/HOTADD all week, it only failed last night.

It's strange for sure! But now @benmilligan is following the case, assisting the tech that is assigned to it, so hopefully with his coaching her, things will move along...
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Vitaliy S.
Product Manager
Posts: 22004
Liked: 1354 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hot add didn't work for one server

Post by Vitaliy S. » Jan 15, 2010 4:06 pm

Darhl,

Probably I can explain you why your jobs were successfull for some time and then some of them failed, the reason for that is that you have this kind of error in the job logs:

Code: Select all

Failed to allocate SCSI target ID
So I wonder how many concurrent jobs do you run using Virtual Applicance mode? I'm asking this, cause there was a known issue for VCB Hot-Add mode with lack of SCSI IDs, and the resolution for this was to add more SCSI controllers to the VM running backup server, I believe it should also help in your situation.

Thank you!

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 4:32 pm

Hi Vitaliy,

At most I run 3 concurrent backup jobs. How would I add additional SCSI controllers without adding virtual disks?

I think you may have it on the issue, when I look at the settings for the Veeam server, I see 14 attached hard disks, and looking at them, they are duplicate hard drives for nas01 & graphics01 over and over. Why is the Veeam VM not releasing those? They are also marked as independent and non-persistent. See screenshot at http://www.flickr.com/photos/45779723@N04/4276943546/

Thanks,

Darhl
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 23608
Liked: 3115 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hot add didn't work for one server

Post by Gostev » Jan 15, 2010 4:52 pm

Darhl, did you disable automount on Veeam Backup proxy (just like for VCB)?
These disks are managed by vStorage API, not Veeam code. But it is normal to see them, this is how Hot Add backups work.

If you are indeed running concurrent jobs, then I think we have just nailed down the issue. I think there are instructions on adding additional SCSI controllers, see the article that Vitaly had referenced above - just add new small disk to VM, and place it on a separate storage controller.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Hot add didn't work for one server

Post by withanh » Jan 15, 2010 4:54 pm

Sure did, will double check, yes it is disabled:
Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\administrator>diskpart

Microsoft DiskPart version 6.1.7600
Copyright (C) 1999-2008 Microsoft Corporation.
On computer: VANC-VEEAM01

DISKPART> automount

Automatic mounting of new volumes disabled.

DISKPART>
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Post Reply

Who is online

Users browsing this forum: Exabot [Bot], Google [Bot], GTFbnelson, ta_yoshizumi, Toshi123, touch.thongjurai and 26 guests