[BETA] [HPE] Backup fails, no logs available for export

[dcolpitts]

I have two separate VMs that are failing to backup. One is a Citrix Netscaler instance, and the job immediately fails with "VM-NAME: Value cannot be null. (Parameter 's')". The other is the VME Manager VM, and it fails with "VM-NAME: Sequence contains no matching element".

When I go to "Help --> Support Information --> Export logs for this job", none of the VME Plugin jobs are listed.

[EvgenyBaev]

Hello,

Plugin jobs are not included in the Export logs for this job - not only for the HPE VME plugin, but for other plugins as well.

You can find the backup job logs here:
C:\ProgramData\Veeam\Backup\Plugins\HPEMORPHEUSVME\Backup

It would be very helpful if you could share these logs with us.

[dcolpitts]

@EvgenyBaev - I'm emailing them to Cody Ault shortly.

[EvgenyBaev]

Hello, dcolpitts

Thank you for logs.

That what we have after some investigation.

1) VM-NAME: Value cannot be null. (Parameter 's')
This error may happen when we've got
"unitNumber": null,
for a volume in VM specification from HPE.

We parse this parameter to get deviceIndex for our common VM metadata. Here is an example of disk metadata.
{
"diskName": "cdrom",
"busType": "Scsi",
"diskUid": "10e51f9d-f756-45da-a4fe-53c9b4573f2c",
"storageContainerUid": "5",
"storageContainerName": "DS01",
"deviceIndex": 1,
"isCdrom": true,
"size": 990216192,
"isBootDisk": false
}

For my VMs I see that it suits the number for controllerMountPoint:
"controllerMountPoint": "359:0:9:4",
"unitNumber": "4",

In your case it's null:
"controllerMountPoint": "138:0:9:0",
"unitNumber": null,

We will ask HPE in which cases it can be that "unitNumber" is null for volume. And maybe from our side the better approach would be to get device number from "controllerMountPoint"

I'll try to find a way of reproducing it and return to you with a workaround today.

2) VM-NAME: Sequence contains no matching element
This one can appear when serial number for volume in VM XML doesn't match to the volume UUID in HPE response for a server.
var correspondingDisk = virtDisksSpec.First(disk => disk.Serial == volume.Uuid);

Here is an example for on of my VMs.
The data from VM xml:
<disk type='block' device='disk'>
<boot order='1' />
<serial>f5de0ead-2e6d-41c2-bc94-39217c1270d0</serial>
<target dev='vda' bus='virtio' />
<driver name='qemu' type='raw' cache='none' io='io_uring' discard='unmap' />
<source dev='/dev/mapper/360002ac000000000000000f20002e0b3' />
</disk>

That's what I got from HPE:
"volumes": [
{
"id": 924,
"name": "root",
"description": null,
"controller": {
"id": 379,
"name": "VIRTIO 0"
},
"controllerId": 379,
"controllerMountPoint": "379:0:9:0",
"unitNumber": "0",
"deviceName": "/dev/vda",
"deviceDisplayName": "vda",
...
"uuid": "f5de0ead-2e6d-41c2-bc94-39217c1270d0",

In my case "uuid" matches serial number from XML.

As I see in logs serial number is not presented at all for VM volumes for the affected VM - GMS-VMEMGR01:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='io_uring' discard='ignore'/>
<source file='/vme/GMS-VMEMGR01/hvm_29-disk-0' index='3'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu'/>
<target dev='sda' bus='sata'/>
<readonly/>
<boot order='2'/>
<alias name='sata0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>

But we have uuid presented for volumes in VM specification in REST API.
Seems like it was not populated for some reason when creating a VM.

I'll try to check how it may happen. And we will also discuss with HPE team.
I think there can be a temporary workaround with adding serial number to VM XML manually and updating VM from config with virsh. But I'll do more tests and return to you with results and more detailed workaround.

[dcolpitts]

So that particular VM (the one with the Parameter 's' error) was a vendor provided KVM appliance that I could not use VME's provisioning for. Instead, the qcow2 and the .xml provided by the vendor were scp'd to the datastore, and "virsh define vm-name.xml" was used to register it. I gave Cody the link for that appliance yesterday and he has a copy of it downloaded. The vendor provided .xml needs a few tweaks however to run out of the box. He also has a copy of the working one so you guys can play with it on your side. I'm sure you'll run into this often (virsh defined VMs).

And the VM GMS-VMEMGR01 is the VME Manager appliance, so if it's messed up, that one is on HPE, not me - I haven't messed with it other than to reconfigure once to increase RAM from 12GB to 16GB.

dcc

[EvgenyBaev]

Hello, Dean

1) VM-NAME: Value cannot be null. (Parameter 's')
I still cannot reproduce this one on our lab. We're already met this issue but quite recently and the fix wasn't pushed to beta.
Could you please try to call reconfigure for that VM through Moprheus UI without changing anything and check if "unitNumber" changes to an integer value in server specification (if you can check through REST API).
If it doesn't help we can provide a fix for you if needed (but most likely on Monday).
I asked HPE team how it can happen but they didn't answer yet.

2) VM-NAME: Sequence contains no matching element
Morpheus manager should be normally skipped on rescan and it shouldn't be possible to select it as a backup target.
Seems like you encountered a bug in our rescan procedure.

I see your HPE manager name in our REST API is
"name": "vmemgr01"

We get in from Response "Get" "https://<your manager domain name>/api/appliance-settings
"applianceId": "vmemgr01"

But the externalID we get from HPE is different:
"externalId": "GMS-VMEMGR01",
"name": "GMS-VMEMGR01"

Did you rename the manager? Or do you know how did it happen?

In our rescan procedure we get the list of VMs from HPE manager and then we filter our manager VM and worker VMs.
we try to get manager VM by externalId
if (vm.ExternalId == _morpheus.Name)

If the condition is not true (which happened in your case) then we consider that the VM is not a manager and we show it among backup targets.

Seems like we cannot rely on name here and we should check VmID instead. I will create a bug.

Also verified for my manager that <serial> is empty indeed for disks:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='io_uring' discard='ignore'/>
<source file='/var/morpheus/kvm/local/hpe-vm-essentials-8.0.10-1.qcow2' index='2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu'/>
<target dev='sda' bus='sata'/>
<readonly/>
<boot order='2'/>
<alias name='sata0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>

3) You've also got issues with a worker stuck in TurningOff state.
It's a known bug which was unfortunately fixed later than beta release.
It can happen when the VeeamHpeMorpheusVmeSvc is restarted when worker is in TurningOff state. It will be fixed in 1.0 HPE release.
If having that worker in UI bothers you we can provide a way to remove it.

4) As for issues on restore - I will take a look tomorrow.

[dcolpitts]

Hi Evgeny.

1.) I'm not sure where to look or how to look for the unitNumber. I don't see the field "unitNumber" in the dumpxml output. Monday is also fine - I've excluded it from the backup for now, and have a manual backup of the config file and the license file - that's all I really need for now to rebuild if it were to go down.
2.) No I did not rename the manager. I did reconfigure the manager to move it the boot volume of the host to the data volume (mounted on /vme)
3.) You or Cody can email the fix when you have time.
4.) My installation is your installation. Just ping me if you need access to take a further look. As I told Cody, we can do pretty much anything to it other than delete the running VMs

I had expected there to be bumps in this road, so it's all good!

dcc

[Davidp1]

I am having similar issues with my VM Essentials environment with multiple VMs reporting "12/5/2025 2:57:35 AM Failed : Vaidio Data : Value cannot be null. (Parameter 's') (0:00:02)" or "12/5/2025 2:06:19 AM Failed : vme : Value cannot be null. (Parameter 's') (0:00:02)". I was able to get it to start preparing the workload by manually launching a Active Full one vme and this kicked it off and got to "preparing for backup" and after an hour or so it was still sitting there so I stopped it.

I tried to create a fresh 24.04 ubuntu instance and running a backup job on that as well. That also was able to make it "preparing for backup" but this seems be running indefinitely it has been running over night and still there now actually "12/5/2025 3:49:52 PM In progress : Test 22 : Preparing for backup (12:16:56)"

I would love to pull logs but am not quite sure to do that on the Backup and Replication Sever Linux Appliance.

[EvgenyBaev]

Hi, Davidp1

If you are working with Linux VBR via WebConsole you can just collect log bundle from Configuration -> About -> Support information -> Collect and download logs.

We will try to prepare a testing fix on Monday. Looks like the issue is reproduced more often than we expected. We periodically got it on our lab but mostly after cloning a VM.

Though if you have time and desire we can try a workaround.
1) Connect to HPE manager linux console via SSH.

2) Grab the credentials for morpheus DB from /opt/morpheus/conf/application.yml. They should be at the top of yaml config:
username: 'morpheus'
password: '<password>'

3) Establish the connection with morpheus DB (they have MySQL) using any DB client (PGAdmin, DBeaver)
In SSH tab specify you HPE manager IP/domain name and credentials for ssh connection
In DB tab use - localhost:3306 and database credentials

4) Then we can execute a query to get the list of affected VMs:
select cs.name, cs.id, sv.id, sv.unit_number, sc.unit_number, sc.bus_number from compute_server cs
left join compute_server_storage_volume cssv on cssv.compute_server_volumes_id=cs.id
left join storage_volume sv on sv.id=cssv.storage_volume_id
left join storage_controller sc on sv.controller_id =sc.id
where sv.unit_number is NULL and sc.id is not null

So we will be able to find out how many such VMs do we have in general.

5) After that we can update unit_number which are null
UPDATE storage_volume sv
JOIN compute_server_storage_volume cssv ON sv.id = cssv.storage_volume_id
JOIN compute_server cs ON cssv.compute_server_volumes_id = cs.id
JOIN storage_controller sc ON sv.controller_id = sc.id
SET sv.unit_number = N
WHERE sv.unit_number IS NULL
AND cs.name = '<affected VM name>'

It would be better to grab a correct N from "controllerMountPoint" - should be that last number (e.g. 4 for "controllerMountPoint": "359:0:9:4").
Unfortunately I haven't found how to get it alternatively in morpheus DB. But I can check it in logs and let you know.

[EvgenyBaev]

Hello, Dean and David

We prepared a fix for the issue
VM-NAME: Value cannot be null. (Parameter 's')

Cody will share it with you.

It's a new plugin exe file for Windows VBR. If you need a plugin for Linux VBR we can share it as well.

To check the fix:
1) Uninstall VeeamPluginHpeMorpheusVme_13.1.0.206.
2) Install VeeamPluginHpeMorpheusVme_13.1.0.213.
3) Try to run backup job again.

If you have any troubles, let me know.

[Davidp1]

Good Afternoon,

Thank you for this update I am running this on the Linux VBR Appliance and would love to get access to this update. One interesting thing I found was a failure to backup once a restoration has been completed. I created a VM specifically for testing and ran it for a few days and backs up were always successful. After the weekend of backups I deleted a single file and performed a VM Restoration to test functionality and the VM was restored with the file i removed no issues there, but after performing that restoration all backup attempts I have tried have failed for the same reason "12/8/2025 9:34:14 PM Failed : Veaam Backup Demo : Value cannot be null. (Parameter 's') (0:00:02)"

[vitaliy.yugay]

Hello David,
As per the issue you are describing, we would appreciate more details about your particular case. Please open a new forum thread for your issue. The fact that the it appeared with a restored VM is not expected. And we would like to understand what caused the issue. Do you mind enabling Debug logging for the plugin and reproduce restore and backup as you described (also share the logs bundle with us)?

Also, is there anything special with the VM itself? We found similar issues after cloning of VMs with disks on Alletra datastores

• what storage type is used for your VM (Alletra/local/GFS2,etc)?

• did you perform any VM Essentials actions with the VM after restoration: cloning, native VME backup, etc?

Here are instructions for enabling Debug logs on Windows:

1. Open the appsettings.json file as administrator.
   Default location: C:\Program Files\Veeam\Plugins\HPE Morpheus VM Essentials\Service

• In the Logging section, set the "LogLevel" to "Debug".

• Restart the VeeamHpeMorpheusVmeSvc service.

• Reproduce the issue and collect the log file.

• Afterwards, you may set the log level back to Info and restart the service again.

For VSA the procedure is a bit different:

1. Login to the Veeam Host Manager Interface of your VSA with Veeam admin user credentials

• Navigate to Logs and Services > Host Configuration tab

• Locate and download a configuration file with the following path: /etc/veeam/platform-service-hpemorpheusvme/appsettings.json

• Edit the "LogLevel" property to "Trace", save and import the edited file to the VSA in Host Management Interface
  {...,"Logging":{"LogLevel":"Trace",...},...}
  See more details in VBR 13 User Guide: https://helpcenter.veeam.com/docs/vbr/u ... tion-files

• Restart the plugin service veeam-platform-service-hpemorpheusvme, see https://helpcenter.veeam.com/docs/vbr/u ... m-services

• Download and share logs, see https://helpcenter.veeam.com/docs/vbr/u ... tml?ver=13

[MrWinsonWolve]

I got this error messages when existing HVM VMs were discovered on a new VME. Couldn't fix it. As I created new VMs on the cluster, there issue was gone.

[cody.ault]

I got this error messages when existing HVM VMs were discovered on a new VME. Couldn't fix it. As I created new VMs on the cluster, there issue was gone.

So the issue has resolved itself for you or you also need the fix?

[Davidp1]

Hello David,
As per the issue you are describing, we would appreciate more details about your particular case. Please open a new forum thread for your issue. The fact that the it appeared with a restored VM is not expected. And we would like to understand what caused the issue. Do you mind enabling Debug logging for the plugin and reproduce restore and backup as you described (also share the logs bundle with us)?

Also, is there anything special with the VM itself? We found similar issues after cloning of VMs with disks on Alletra datastores

• what storage type is used for your VM (Alletra/local/GFS2,etc)?

• did you perform any VM Essentials actions with the VM after restoration: cloning, native VME backup, etc?

Here are instructions for enabling Debug logs on Windows:

1. Open the appsettings.json file as administrator.
   Default location: C:\Program Files\Veeam\Plugins\HPE Morpheus VM Essentials\Service

• In the Logging section, set the "LogLevel" to "Debug".

• Restart the VeeamHpeMorpheusVmeSvc service.

• Reproduce the issue and collect the log file.

• Afterwards, you may set the log level back to Info and restart the service again.

For VSA the procedure is a bit different:

1. Login to the Veeam Host Manager Interface of your VSA with Veeam admin user credentials

• Navigate to Logs and Services > Host Configuration tab

• Locate and download a configuration file with the following path: /etc/veeam/platform-service-hpemorpheusvme/appsettings.json

• Edit the "LogLevel" property to "Trace", save and import the edited file to the VSA in Host Management Interface
  {...,"Logging":{"LogLevel":"Trace",...},...}
  See more details in VBR 13 User Guide: https://helpcenter.veeam.com/docs/vbr/u ... tion-files

• Restart the plugin service veeam-platform-service-hpemorpheusvme, see https://helpcenter.veeam.com/docs/vbr/u ... m-services

• Download and share logs, see https://helpcenter.veeam.com/docs/vbr/u ... tml?ver=13

Thank you for these instructions but unfortunately the environment would not start up properly after changing the json as mentioned. It was giving bad gateway errors and only accessible via port :10443. Even after re-importing the original unchanged json it was stuck in that state.

I ended up doing a fresh VM install of the Linux VBR with the updated plugin that was released on Monday and these issues appear to be fixed. I will continue to test and report any issue I find.