Veeam replicated VM will not power on with CLI

[Danielf]

I am in the process of testing out Veeam's replication functionality, and I've come across a problem:

Currently, I have Veeam create a replicated VM from Host A to Host B, over the network. That works fine. If I go into Vcenter and power up the replicated VM, that also works fine (although I was queried initially about the status of the VM - copy/moved/etc.). However, if I attempt to power up the VM using the Vsphere CLI, I get the response "no virtual machine found". I can use the CLI to identify the precise location of the .vmx file, copy and paste that into the command line, and it still doesn't work. My question is - why is the CLI failing to see the virtual machine during the power up command?

[Vitaliy S.]

Daniel, can you power up any other VM with the same vSphere CLI command? By the way, what is the command you're using?

[Danielf]

Daniel, can you power up any other VM with the same vSphere CLI command? By the way, what is the command you're using?

Absolutely. The command I am using is vmware-cmd.pl (usernamepassword) (host) /vmfs/volumes/vol1/Veeambackup/vm(vm-56)/vm.vmx start

[Danielf]

Any ideas here, gents? Does Veeam somehow re-name the .vmx?

[Gostev]

No, definitely not. We backup and restore VMX file intact.

[Danielf]

No, definitely not. We backup and restore VMX file intact.

What about the configuration file? Why would a CLI command that works for normal VMclones not work for a Veeam clone?

[Gostev]

I was talking about configuration file above... VMX is VM configuration file.

[Danielf]

I *believe* I have foudn the problem. Veeam, when making a replication, adds a parenthesis to the storage path: (vm-56) in this case. Apparently the CLI doesn't like that, and I've tried quotes, etc.

Question: can I reconfigure Veeam to not add parenthesis to the file path when it creates a clone?

[tsightler]

Have you tried preceding the ( with a \ ? That's typically the escape character in most shells to tell it to interpret the next character as a literal.

[Danielf]

Have you tried preceding the ( with a \? That's typically the escape character in most shells to tell it to interpret the next character as a literal.

Both parenthesis? Like so? \(vm-56\)

[tsightler]

Exactly like that, I just tried it on my ESX 4.1 host and that worked fine.

[Danielf]

Just tried, no joy.

Here is the exact CLI command I am using and the result reported (please note, that file location is a copy paste from the -l output):

C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -U ****** -P ******* -H 10.0.0.20 /vmfs/volumes/4daa0249-2482052d-0074-0026b949729a/VeeamBackup/Mail\(vm-228\)/Mail3.vmx start

No virtual machine found.

[tsightler]

Are you ESX or ESXi? Is this the remote console?

Like I said I just did exactly this:

vmware-cmd /vmfs/volumes/VMFS_REPL/VeeamBackup/zso-testvm\(vm-29075\)/zso-testvm.vmx start

And it worked fine.

Also, you do know that the "normal" way to fire up a Veeam replica is to use the Veeam GUI, right? Doing it directly through vCenter or the command line breaks the ability to continue replication without starting a full replica again. I'm assuming your testing for the case where Veeam is not available?

[tsightler]

Oh, so it is remote CLI. That means you need to use "windows" syntax for escape. Give me a minute, I think I can come up with it.

[Danielf]

Are you ESX or ESXi? Is this the remote console?

Like I said I just did exactly this:

vmware-cmd /vmfs/volumes/VMFS_REPL/VeeamBackup/zso-testvm\(vm-29075\)/zso-testvm.vmx start

And it worked fine.

Also, you do know that the "normal" way to fire up a Veeam replica is to use the Veeam GUI, right? Doing it directly through vCenter or the command line breaks the ability to continue replication without starting a full replica again. I'm assuming your testing for the case where Veeam is not available?

Vsphere Essentials Enterprise Plus for now - ESX. This command is being run from the Vcenter that hosts Veeam (windows box) against the server VCenter resides on.

I do know this is normally run via Veeam, however, I am trying to write a script that will power on a clone automatically when a VM on another machine goes offline.

[tsightler]

OK, understood. I think that Windows typically uses the "^" character for escape, so you might try the command as:

C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -U ****** -P ******* -H 10.0.0.20 /vmfs/volumes/4daa0249-2482052d-0074-0026b949729a/VeeamBackup/Mail^(vm-228^)/Mail3.vmx start

I don't happen to have a Windows machine handy with VMware remote CLI installed, but I think the above might work.

[Danielf]

No joy, damn!

[tsightler]

Yep, this problem looks like a problem with the vmware-cmd.pl script of the Remote CLI, but at this point I don't think it has anything to do with the parentheses characters. It appears that the routine which parses the $vm_cfg_path variable assumes that the path will always be "<machine directory>/<machine.vmx>", so basically just one path deep. For a Veeam replicated VM there's an extra directory "VeeamBackup/<machine directory>/<machine.vmx>", and the parsing routing is truncating the <machine.vmx> part.

But, in the meantime, I do have a workaround for you. If you pass the "vm_cfg_path" in the format that the "filter" already needs, then the vmware-cmd.pl script doesn't bother to parse it, it just accepts the path exactly as you type it. The "vm_cfg_path" is in the format of:

[VMFS-VOL-NAME] Filepath/machine.vmx

So for your example above the command would be as follows:

vmware-cmd.pl -U ****** -P ******* -H 10.0.0.20 "[VMFS-VOL] VeeamBackup/Mail(vm-228)/Mail3.vmx" start

Note that you should use the actual name of your specific VMFS volume and, with this format, it won't take the UUID, only the volume friendly name (but that should really be easier), and you'll need to use quotes because there's a space between the volume name and the path.

I can probably come up with a patch for vmware-cmd.pl to actually fix this, but the workaround should be good. Give it a try and let me know.

[tsightler]

More research and I've found the problem code in the "generate_cfg_path" subroutine in vmware-cmd.pl. The code that returns the path looks like this:

if($ds_flag == 1) {
return ("[" . $dsname . "] " . $patharr[4] . "\/" . $patharr[5]);
}

So it's hard coded to return the datastore name in brackets, and then only two path elements with a "/" in between. Since the replica's have a third path element (the "VeeamBackup" folder) then it doesn't return the full path to the vmx.

The most ghetto fix is probably to just copy vmware-cmd.pl to vmware-cmd-veeam.pl, and change the line above to:

if($ds_flag == 1) {
return ("[" . $dsname . "] " . $patharr[4] . "\/" . $patharr[5] . "\/" . $patharr[6]);
}

Then use this custom vmware-cmd for your Veeam replica scripts. I tested this and it works. It wouldn't be that difficult to hack up a "correct" fix for this, but I'm not sure it's worth it since the workaround I gave you earlier should work fine.

[Danielf]

The volume is named Standy Data Store. As such, I am using this command:

C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -U XXX -P XXX -H 10.0.0.20 "[STANDBY DATA STORE] VeeamBackup/Mail(vm-228)/Mail3.vmx" start

Unfortunately, no joy. And let me add I very much apprecaite your help here!

[Danielf]

More research and I've found the problem code in the "generate_cfg_path" subroutine in vmware-cmd.pl. The code that returns the path looks like this:

if($ds_flag == 1) {
return ("[" . $dsname . "] " . $patharr[4] . "\/" . $patharr[5]);
}

So it's hard coded to return the datastore name in brackets, and then only two path elements with a "/" in between. Since the replica's have a third path element (the "VeeamBackup" folder) then it doesn't return the full path to the vmx.

The most ghetto fix is probably to just copy vmware-cmd.pl to vmware-cmd-veeam.pl, and change the line above to:

if($ds_flag == 1) {
return ("[" . $dsname . "] " . $patharr[4] . "\/" . $patharr[5] . "\/" . $patharr[6]);
}

Then use this custom vmware-cmd for your Veeam replica scripts. I tested this and it works. It wouldn't be that difficult to hack up a "correct" fix for this, but I'm not sure it's worth it since the workaround I gave you earlier should work fine.

I am afraid I don't know how to do what you are suggesting. Specifically "copy vmware-cmd.pl to vmware-cmd-veeam.pl, and change the line above to:".

[Danielf]

Never mind, I got it, and you are a GOD AMONGST MEN! Thanks mate.

[Gostev]

you are a GOD AMONGST MEN!

QFE

[tsightler]

So I'm curious, did you get the patched script to work, or the workaround by using the different path format. I'm guessing the patched script because I think another bug would keep you from being able to use the initial workaround I provided as it looks like the script would fail to detect your volume name inside brackets because it has spaces. We actually use dashes in our VMFS volume names instead of spaces because of some similar issue in early versions of VMware's tools back in the ESX 2.x days.

[tsightler]

Also, since my first "patch" for vmware-cfg.pl was just a ghetto hack to work specifically with Veeam I thought I'd post a more correct patch that won't break the existing functionality and would work no matter the depth of the path. In the original vmware-cmd.pl replace the following line:

Code: Select all

if($ds_flag == 1) {
return ("[" . $dsname . "] " . $patharr[4] . "\/" . $patharr[5]);
}

with the following:

Code: Select all

if($ds_flag == 1) {
   my $vm_real_path = ("[" . $dsname . "] " . $patharr[4] );
   my @path_elements = @patharr[5 .. $#patharr];
   foreach my $element (@path_elements) {
      $vm_real_path = $vm_real_path . "\/" . $element;
   }
   return $vm_real_path;
}

This code will work with VM's in the VeeamBackup folder, and will not break existing functionality.

BTW, it turns out this problem has been going on a long time. When I just did a search to see if there was any chance of submitting my patch to VMware to get them to incorporate the fix, I found the following thread in the VMware Communities:

http://communities.vmware.com/thread/197897

A couple of people reported the same issue (not with Veeam, but with VM's in subfolders) and there's even a patch posted there as well (slightly different than my patch above, but pretty much the exact same concept). This thread is from 2009 so it doesn't look like it's been fixed yet, but I may still try.