Host-based backup of KVM-based VMs (Red Hat Virtualization, Oracle Linux Virtualization Manager and Proxmox VE)
Post Reply
mdws
Influencer
Posts: 17
Liked: never
Joined: Nov 03, 2011 6:14 pm
Contact:

Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by mdws »

Hello,

since the update from 12.0 to 12.1 we have failed and stucked Backups on some VMs, the Error on the Engine is:
command RemoveImageTicketVDS failed: Image daemon request failed: 'status=500, reason=Internal Server Error, error=Server failed to perform the request, check logs\n'

In Veeam we get:

Unable to finalize image transfer 18e98d64-ba3d-4eb9-bebf-4f7386bb5267: Finalize ImageTransfer timeout
Unable to finalize backup 5530fcaa-7411-41fd-9fbb-569272727927: Cannot stop VM backup. There is an active image transfer for VM backup
Failed to perform backup: Cannot stop VM backup. There is an active image transfer for VM backup

We are on oVirt Engine 4.5.3.2-1.el8, thats the same as "Red Hat Virtualization 4.4 SP 1 Batch Update 3"
https://access.redhat.com/documentation ... virt_4_5_3

As we had this Error never before and it started exactly with the first Backup after the Upgrade from 12.0 to 12.1, we think it's related to the Update...

But as we are not on official RHV, Veeam refuses to support us. (Case #03004921 + #07070307)
RHV does not sell RHV Subscriptions anymore, no Trial available anymore. We have no change to switch to official RHV or to check if this Error also happens on official RHV.

So we have to downgrade to 12.0 or kick Veeam out of our Business, with both options we would not be happy.
Does anyone else have this Problems after updating to 12.1? (And perhaps have a solution ;-))

Thanks!
mdws
Influencer
Posts: 17
Liked: never
Joined: Nov 03, 2011 6:14 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by mdws »

Nobody?

Does anyone run Veeam for RHV and no Errors on the Engine?
If yes, it would be nice to know the Engine and Node Versions...

Thanks!
PTide
Product Manager
Posts: 6560
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by PTide »

Hello,

So far neither our QA or support teams have never seen anything like that on RHV.
If will update this thread if there are any reports.

Please note that if there will be issued a fix for RHV, it won't mean that such fix will be guaranteed to work with plain oVirt too.

Thanks!
Yury.Panchenko
Veeam Software
Posts: 11
Liked: 5 times
Joined: Jun 26, 2014 8:10 am
Full Name: Yury Panchenko
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by Yury.Panchenko »

Hello,
> There is an active image transfer for VM backup
Can you check that disks of these vms aren't locked by other backup/image transfer
Ovirt could have some stuck image transfers and than we can't backup this vm.

What type of disks have these vms?
jsattelb
Lurker
Posts: 2
Liked: never
Joined: Jan 12, 2024 6:07 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by jsattelb »

Do we know what version of ovirt-imageio was tested? Please see: https://github.com/oVirt/ovirt-imageio/ ... 1889731851

You may also reference #03094001. I closed it recently after devising a workaround.

These are some of the issues I have encountered after upgrading to VBR 12.1 and RHEV Plugin 4:

1. The VM backup becomes stuck in a finalizing state. Manual intervention is necessary to cancel a backup request using sample scripts from the oVirt SDK. It might be related to either #2 or #3.
2. state=500 error messages from RemoveImageTicketVDS. I resolved by modifying a file in ovirt-image to produce the intended response code. It now produces a state=409 conflict message instead of a 500-internal server error message. There are still a lot of 409 messages in the event log, but thing seem to keep going.
3. We're unable to take a backup of a VM that contains any addition (or lingering) snapshots. Appears resolved by removing all snapshots. This certainly was not the case before. I previously had snapshots dating back to August.

After addressing #2 and #3, the backups seem to complete successfully.
mdws
Influencer
Posts: 17
Liked: never
Joined: Nov 03, 2011 6:14 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by mdws »

I compiled and downgraded a Node to:
https://ftp.redhat.com/redhat/linux/ent ... ev.src.rpm
followed by a reboot.
I get the same error as before:
command RemoveImageTicketVDS failed: Image daemon request failed: 'status=500, reason=Internal Server Error, error=Server failed to perform the request, check logs\n'
The stucking backups with error:
Failed to perform backup: Cannot stop VM backup. There is an active image transfer for VM backup
and:
Finalize ImageTransfer timeout
are gone after some backups.
I can't believe that the error:
command RemoveImageTicketVDS failed: Image daemon request failed: 'status=500, reason=Internal Server Error, error=Server failed to perform the request, check logs\n'
does not happen on RHV, can you please check this in your QA environment? As far as I could test until now, this error does not prevent the backup running successfully, but it slows down the backups compared to v3 / 12.0 and produces a lot of useless error messages on the Engine

And please check the github link jsattelb posted, Nir Soffer analyzed the process and also came to the result that there is something wrong in the clients sequence (which was fine with v3 / 12.0).
mdws
Influencer
Posts: 17
Liked: never
Joined: Nov 03, 2011 6:14 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by mdws »

One failed Backup (of ~ 100) today, Veeam UI says:

Unable to create incremental backup: Cannot backup VM: Disk is locked. Please try again later.
Failed to perform backup: Cannot backup VM: Disk is locked. Please try again later.

On Engine the Backup Task is still: "Executing"

Don't know if it matters but the VM is powered off and Backup size is 0 B, for other powered off VMs the Backup works.
mdws
Influencer
Posts: 17
Liked: never
Joined: Nov 03, 2011 6:14 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by mdws »

@Yury.Panchenko all disks are qcow2 disks
Yury.Panchenko
Veeam Software
Posts: 11
Liked: 5 times
Joined: Jun 26, 2014 8:10 am
Full Name: Yury Panchenko
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by Yury.Panchenko »

Hello,
I've already asked our support team to contact you again. We need more details about the setup.
mdws
Influencer
Posts: 17
Liked: never
Joined: Nov 03, 2011 6:14 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by mdws »

Hello,

until now I have no questions to my setup in the tickets, only the message that they won't work on this tickets because it's no RHV.
I tested with 3 environments until now, one I did build from scratch only for debugging this problem.
All behave exactly the same, from oVirt 4.5.3 - 4.5.6, alos tried with latest OLVM, exactly the same behaviour.
I also tested switching back to 12.0 in the Test Environment, no errors anymore....
bbenard
Novice
Posts: 5
Liked: never
Joined: Mar 30, 2021 8:36 pm
Full Name: Brock Benard
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by bbenard »

We are experiencing this exact issue after upgrading the RHV plugin to 4.0. The RHV 3.0 plugin was working fine with VBR 12.1. I have an active open case #01948917, uploaded multiple log bundles this week, and have yet to hear back from any Veeam support staff. I'm losing my patience and about ready to remove all of it and go back to RHV 3.0!

RedHat Virtualization Manager version - 4.5.4-1.0.27.el8
Veeam Backup and Replication version - 12.1.0.2131
Veeam RHV plugin version - 4.0 (12.4.0.385)
mdws
Influencer
Posts: 17
Liked: never
Joined: Nov 03, 2011 6:14 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by mdws »

Hello Brock,

pls check your PMs.
PTide
Product Manager
Posts: 6560
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by PTide »

@bbenard,

Would you double-check your case number please? It does not look like a valid case ID.

Thanks!
bbenard
Novice
Posts: 5
Liked: never
Joined: Mar 30, 2021 8:36 pm
Full Name: Brock Benard
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by bbenard »

Sorry case number is Case #07111999
bbenard
Novice
Posts: 5
Liked: never
Joined: Mar 30, 2021 8:36 pm
Full Name: Brock Benard
Contact:

Re: Veeam RHV 12.1 command RemoveImageTicketVDS failed

Post by bbenard »

In the meantime we have rolled back to the RHV 3.0 plug-in as Veeam informed us that they will not support Oracle KVM even though it's the same thing under the covers.....
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests