[BETA] [HPE] Backup fails, no logs available for export

[dcolpitts]

I have two separate VMs that are failing to backup. One is a Citrix Netscaler instance, and the job immediately fails with "VM-NAME: Value cannot be null. (Parameter 's')". The other is the VME Manager VM, and it fails with "VM-NAME: Sequence contains no matching element".

When I go to "Help --> Support Information --> Export logs for this job", none of the VME Plugin jobs are listed.

[EvgenyBaev]

Hello,

Plugin jobs are not included in the Export logs for this job - not only for the HPE VME plugin, but for other plugins as well.

You can find the backup job logs here:
C:\ProgramData\Veeam\Backup\Plugins\HPEMORPHEUSVME\Backup

It would be very helpful if you could share these logs with us.

[dcolpitts]

@EvgenyBaev - I'm emailing them to Cody Ault shortly.

[EvgenyBaev]

Hello, dcolpitts

Thank you for logs.

That what we have after some investigation.

1) VM-NAME: Value cannot be null. (Parameter 's')
This error may happen when we've got
"unitNumber": null,
for a volume in VM specification from HPE.

We parse this parameter to get deviceIndex for our common VM metadata. Here is an example of disk metadata.
{
"diskName": "cdrom",
"busType": "Scsi",
"diskUid": "10e51f9d-f756-45da-a4fe-53c9b4573f2c",
"storageContainerUid": "5",
"storageContainerName": "DS01",
"deviceIndex": 1,
"isCdrom": true,
"size": 990216192,
"isBootDisk": false
}

For my VMs I see that it suits the number for controllerMountPoint:
"controllerMountPoint": "359:0:9:4",
"unitNumber": "4",

In your case it's null:
"controllerMountPoint": "138:0:9:0",
"unitNumber": null,

We will ask HPE in which cases it can be that "unitNumber" is null for volume. And maybe from our side the better approach would be to get device number from "controllerMountPoint"

I'll try to find a way of reproducing it and return to you with a workaround today.

2) VM-NAME: Sequence contains no matching element
This one can appear when serial number for volume in VM XML doesn't match to the volume UUID in HPE response for a server.
var correspondingDisk = virtDisksSpec.First(disk => disk.Serial == volume.Uuid);

Here is an example for on of my VMs.
The data from VM xml:
<disk type='block' device='disk'>
<boot order='1' />
<serial>f5de0ead-2e6d-41c2-bc94-39217c1270d0</serial>
<target dev='vda' bus='virtio' />
<driver name='qemu' type='raw' cache='none' io='io_uring' discard='unmap' />
<source dev='/dev/mapper/360002ac000000000000000f20002e0b3' />
</disk>

That's what I got from HPE:
"volumes": [
{
"id": 924,
"name": "root",
"description": null,
"controller": {
"id": 379,
"name": "VIRTIO 0"
},
"controllerId": 379,
"controllerMountPoint": "379:0:9:0",
"unitNumber": "0",
"deviceName": "/dev/vda",
"deviceDisplayName": "vda",
...
"uuid": "f5de0ead-2e6d-41c2-bc94-39217c1270d0",

In my case "uuid" matches serial number from XML.

As I see in logs serial number is not presented at all for VM volumes for the affected VM - GMS-VMEMGR01:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='io_uring' discard='ignore'/>
<source file='/vme/GMS-VMEMGR01/hvm_29-disk-0' index='3'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu'/>
<target dev='sda' bus='sata'/>
<readonly/>
<boot order='2'/>
<alias name='sata0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>

But we have uuid presented for volumes in VM specification in REST API.
Seems like it was not populated for some reason when creating a VM.

I'll try to check how it may happen. And we will also discuss with HPE team.
I think there can be a temporary workaround with adding serial number to VM XML manually and updating VM from config with virsh. But I'll do more tests and return to you with results and more detailed workaround.

[dcolpitts]

So that particular VM (the one with the Parameter 's' error) was a vendor provided KVM appliance that I could not use VME's provisioning for. Instead, the qcow2 and the .xml provided by the vendor were scp'd to the datastore, and "virsh define vm-name.xml" was used to register it. I gave Cody the link for that appliance yesterday and he has a copy of it downloaded. The vendor provided .xml needs a few tweaks however to run out of the box. He also has a copy of the working one so you guys can play with it on your side. I'm sure you'll run into this often (virsh defined VMs).

And the VM GMS-VMEMGR01 is the VME Manager appliance, so if it's messed up, that one is on HPE, not me - I haven't messed with it other than to reconfigure once to increase RAM from 12GB to 16GB.

dcc

[EvgenyBaev]

Hello, Dean

1) VM-NAME: Value cannot be null. (Parameter 's')
I still cannot reproduce this one on our lab. We're already met this issue but quite recently and the fix wasn't pushed to beta.
Could you please try to call reconfigure for that VM through Moprheus UI without changing anything and check if "unitNumber" changes to an integer value in server specification (if you can check through REST API).
If it doesn't help we can provide a fix for you if needed (but most likely on Monday).
I asked HPE team how it can happen but they didn't answer yet.

2) VM-NAME: Sequence contains no matching element
Morpheus manager should be normally skipped on rescan and it shouldn't be possible to select it as a backup target.
Seems like you encountered a bug in our rescan procedure.

I see your HPE manager name in our REST API is
"name": "vmemgr01"

We get in from Response "Get" "https://<your manager domain name>/api/appliance-settings
"applianceId": "vmemgr01"

But the externalID we get from HPE is different:
"externalId": "GMS-VMEMGR01",
"name": "GMS-VMEMGR01"

Did you rename the manager? Or do you know how did it happen?

In our rescan procedure we get the list of VMs from HPE manager and then we filter our manager VM and worker VMs.
we try to get manager VM by externalId
if (vm.ExternalId == _morpheus.Name)

If the condition is not true (which happened in your case) then we consider that the VM is not a manager and we show it among backup targets.

Seems like we cannot rely on name here and we should check VmID instead. I will create a bug.

Also verified for my manager that <serial> is empty indeed for disks:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='io_uring' discard='ignore'/>
<source file='/var/morpheus/kvm/local/hpe-vm-essentials-8.0.10-1.qcow2' index='2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu'/>
<target dev='sda' bus='sata'/>
<readonly/>
<boot order='2'/>
<alias name='sata0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>

3) You've also got issues with a worker stuck in TurningOff state.
It's a known bug which was unfortunately fixed later than beta release.
It can happen when the VeeamHpeMorpheusVmeSvc is restarted when worker is in TurningOff state. It will be fixed in 1.0 HPE release.
If having that worker in UI bothers you we can provide a way to remove it.

4) As for issues on restore - I will take a look tomorrow.

[dcolpitts]

Hi Evgeny.

1.) I'm not sure where to look or how to look for the unitNumber. I don't see the field "unitNumber" in the dumpxml output. Monday is also fine - I've excluded it from the backup for now, and have a manual backup of the config file and the license file - that's all I really need for now to rebuild if it were to go down.
2.) No I did not rename the manager. I did reconfigure the manager to move it the boot volume of the host to the data volume (mounted on /vme)
3.) You or Cody can email the fix when you have time.
4.) My installation is your installation. Just ping me if you need access to take a further look. As I told Cody, we can do pretty much anything to it other than delete the running VMs

I had expected there to be bumps in this road, so it's all good!

dcc

[Davidp1]

I am having similar issues with my VM Essentials environment with multiple VMs reporting "12/5/2025 2:57:35 AM Failed : Vaidio Data : Value cannot be null. (Parameter 's') (0:00:02)" or "12/5/2025 2:06:19 AM Failed : vme : Value cannot be null. (Parameter 's') (0:00:02)". I was able to get it to start preparing the workload by manually launching a Active Full one vme and this kicked it off and got to "preparing for backup" and after an hour or so it was still sitting there so I stopped it.

I tried to create a fresh 24.04 ubuntu instance and running a backup job on that as well. That also was able to make it "preparing for backup" but this seems be running indefinitely it has been running over night and still there now actually "12/5/2025 3:49:52 PM In progress : Test 22 : Preparing for backup (12:16:56)"

I would love to pull logs but am not quite sure to do that on the Backup and Replication Sever Linux Appliance.

[EvgenyBaev]

Hi, Davidp1

If you are working with Linux VBR via WebConsole you can just collect log bundle from Configuration -> About -> Support information -> Collect and download logs.

We will try to prepare a testing fix on Monday. Looks like the issue is reproduced more often than we expected. We periodically got it on our lab but mostly after cloning a VM.

Though if you have time and desire we can try a workaround.
1) Connect to HPE manager linux console via SSH.

2) Grab the credentials for morpheus DB from /opt/morpheus/conf/application.yml. They should be at the top of yaml config:
username: 'morpheus'
password: '<password>'

3) Establish the connection with morpheus DB (they have MySQL) using any DB client (PGAdmin, DBeaver)
In SSH tab specify you HPE manager IP/domain name and credentials for ssh connection
In DB tab use - localhost:3306 and database credentials

4) Then we can execute a query to get the list of affected VMs:
select cs.name, cs.id, sv.id, sv.unit_number, sc.unit_number, sc.bus_number from compute_server cs
left join compute_server_storage_volume cssv on cssv.compute_server_volumes_id=cs.id
left join storage_volume sv on sv.id=cssv.storage_volume_id
left join storage_controller sc on sv.controller_id =sc.id
where sv.unit_number is NULL and sc.id is not null

So we will be able to find out how many such VMs do we have in general.

5) After that we can update unit_number which are null
UPDATE storage_volume sv
JOIN compute_server_storage_volume cssv ON sv.id = cssv.storage_volume_id
JOIN compute_server cs ON cssv.compute_server_volumes_id = cs.id
JOIN storage_controller sc ON sv.controller_id = sc.id
SET sv.unit_number = N
WHERE sv.unit_number IS NULL
AND cs.name = '<affected VM name>'

It would be better to grab a correct N from "controllerMountPoint" - should be that last number (e.g. 4 for "controllerMountPoint": "359:0:9:4").
Unfortunately I haven't found how to get it alternatively in morpheus DB. But I can check it in logs and let you know.