-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Virtual lab and Citrix Virtual apps/desktops
Hi,
this is probably not a bug in Veeam, but it might be useful information for people who are using Citrix XenApp in their environment. Here it goes:
I noticed probably a bug in the Citrix delivery controller. We use Veeam backup and replication for backing up our environment. When using Sure Backup to verify the backups this causes havoc with the Citrix Machine catalogs.
The Citrix Delivery Controller seems to have an issue with this functionality. When the VM is added to the vSphere environment by the replication job or sure backup job, the UUID of the VM is the same, since this is the original VM from the backup. The same thing happens when you clone a VM. This clone also gets the same UUID. When this situation exists, one of the services running on the Citrix delivery controller replaces the original VM in the machine catalog with the temporary VM from sure backup The end-result of this is that you end-up with a non-working machine catalog, since the temporary VM is being deleted when verification is completed. Also, the delivery controller shouldn't touch the catalog at all.
Further investigation shows that this issue is caused by the fact that the Citrix Broker uses the extensiondata.config.uuid as the unique identified. It stores this value in the HostedMachineID of the broker database. Since the temporary VM is a clone of the original VM, it has the same extensiondata.config.uuid. Apparently there's some re-scanning process in the background running on the broker that for some reason replaces the VM in the catalog without any message or notification.
I opened a case with citrix already.
Franc.
this is probably not a bug in Veeam, but it might be useful information for people who are using Citrix XenApp in their environment. Here it goes:
I noticed probably a bug in the Citrix delivery controller. We use Veeam backup and replication for backing up our environment. When using Sure Backup to verify the backups this causes havoc with the Citrix Machine catalogs.
The Citrix Delivery Controller seems to have an issue with this functionality. When the VM is added to the vSphere environment by the replication job or sure backup job, the UUID of the VM is the same, since this is the original VM from the backup. The same thing happens when you clone a VM. This clone also gets the same UUID. When this situation exists, one of the services running on the Citrix delivery controller replaces the original VM in the machine catalog with the temporary VM from sure backup The end-result of this is that you end-up with a non-working machine catalog, since the temporary VM is being deleted when verification is completed. Also, the delivery controller shouldn't touch the catalog at all.
Further investigation shows that this issue is caused by the fact that the Citrix Broker uses the extensiondata.config.uuid as the unique identified. It stores this value in the HostedMachineID of the broker database. Since the temporary VM is a clone of the original VM, it has the same extensiondata.config.uuid. Apparently there's some re-scanning process in the background running on the broker that for some reason replaces the VM in the catalog without any message or notification.
I opened a case with citrix already.
Franc.
-
- Product Manager
- Posts: 14840
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Hi Frank,
interesting observation. Thanks for sharing.
What I do not understand: how does the production delivery controller know about the VM running in a virtual lab? I mean, there should be no access from production to Virtual Labs.
Best regards,
Hannes
interesting observation. Thanks for sharing.
What I do not understand: how does the production delivery controller know about the VM running in a virtual lab? I mean, there should be no access from production to Virtual Labs.
Best regards,
Hannes
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Hi Hannes,
Because the delivery controller has a connection to the vcenter server. That’s were it gets it information from.
Franc.
Because the delivery controller has a connection to the vcenter server. That’s were it gets it information from.
Franc.
-
- Veeam Software
- Posts: 540
- Liked: 193 times
- Joined: Mar 07, 2016 3:55 pm
- Full Name: Ronn Martin
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Hey Franc, I was at Citrix for many years and I need to brush up a bit on my MCS/PVS operation but yes I would think this would be the expected behavior since those machine creation facilities communicate directly to the hypervisor. Is that what you're hearing from Citrix as well?
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Citrix is still investigating, but they tent to believe it’s indeed the issue. They will propose a code change to development to use the MorefID instead of the UUID as the unique identifier in the MCS machine catalog.
The problem I now have is that I can’t exclude VMs from the surebackup job, so I can’t prevent Veeam from starting up the XenApp VMs and causing my machine catalogs to break. The XenApp VMs are in a backup job with many other VMs. I’ll see if I can work around this by putting the VMs in a separate backup job and not including that backup job in the sure backup job.
The problem I now have is that I can’t exclude VMs from the surebackup job, so I can’t prevent Veeam from starting up the XenApp VMs and causing my machine catalogs to break. The XenApp VMs are in a backup job with many other VMs. I’ll see if I can work around this by putting the VMs in a separate backup job and not including that backup job in the sure backup job.
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Thanks Franc, you are my hero.
Have the same problem in a customers infrastructure.
I will test this after my vacation next week. Create a backup job „without citrix vms“ as a Source for the SureBackup Job.
Have the same problem in a customers infrastructure.
I will test this after my vacation next week. Create a backup job „without citrix vms“ as a Source for the SureBackup Job.
Product Management Analyst @ Veeam Software
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
You're welcome! It would be nice though if you can exclude specific VMs within Veaam for a sure backup job. You can exclude VMs in all other job types but not in sure backup.
There's another thread about this, it's many, many years old but still not implemented:
vmware-vsphere-f24/is-it-possible-to-ex ... t6668.html
There's another thread about this, it's many, many years old but still not implemented:
vmware-vsphere-f24/is-it-possible-to-ex ... t6668.html
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Yes, this would be a nice feature. I have already added me to the feature request there
Product Management Analyst @ Veeam Software
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
We have tested surebackup Job with excluded Citrix Server this morning successfully.
Everything is running fine, as long we don‘t boot up The Citrix Servers in the virtual Lab.
Thanks again Franc, can you post an update as soon you have a solution from Citrix Support?
Everything is running fine, as long we don‘t boot up The Citrix Servers in the virtual Lab.
Thanks again Franc, can you post an update as soon you have a solution from Citrix Support?
Product Management Analyst @ Veeam Software
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Hi,
I will. Citrix support closed the case, but our account manager will inform me when the fix or hotfix is being implemented. They didn’t gave a timeline though.
I will. Citrix support closed the case, but our account manager will inform me when the fix or hotfix is being implemented. They didn’t gave a timeline though.
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Ok, thanks
Product Management Analyst @ Veeam Software
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
I reached out to Citrix again, since I haven't heard from them. They say nothing has changed and asked me to contact Veeam to see if they could do a change on their side so that the UUID is changed after creating the replica. I pointed out that it's not a Veeam issue, but a Citrix issue, since I can reproduce it also by simply cloning the VM from vCenter itself. They also asked me to create a new case to research the issue again, where as they first said an enhancement request was made already.... So I guess this is leading to nowhere and we only have the workaround currently in place by using the powershell script to change the UUID's after cloning, or wait for the feature request from Veeam to exclude VMs from replicating.
-
- Veeam ProPartner
- Posts: 300
- Liked: 44 times
- Joined: Dec 03, 2015 3:41 pm
- Location: UK
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
I'm hoping to use a Veeam Lab/Sure Backup job, to test an upcoming Citrix Virtual Apps upgrade.
Do you know if this affects just the Delivery Agent VMs running under MCS, or the Infrastructure servers as well?
I was just going to test the upgrade of the Delivery Controller, Database, License Server, Studio, Director and StoreFront. No Delivery Agent VMs in the job.
I won't start the Sure Backup job if it'll affect the live system - but none of the components above are on MCS deployed servers.
Do you know if this affects just the Delivery Agent VMs running under MCS, or the Infrastructure servers as well?
I was just going to test the upgrade of the Delivery Controller, Database, License Server, Studio, Director and StoreFront. No Delivery Agent VMs in the job.
I won't start the Sure Backup job if it'll affect the live system - but none of the components above are on MCS deployed servers.
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
For us it's only affecting the VDA's since the Citrix Delivery Controller is not using the moRefID,but the bios UUID as the unique identifier to identify the VMs it needs to powercontrol. With the other server I have no issues, since the delivery controller doesn't control them.
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Was there ever a solution from Citrix?
I need to test 20-30 citrix vdas with surebackup.
But if they disconnect from the director, i cannot do it.
I need to test 20-30 citrix vdas with surebackup.
But if they disconnect from the director, i cannot do it.
Product Management Analyst @ Veeam Software
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Unfortunately, no.
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Ok, thanks Franc.
Now I need to rebuild the backup jobs to exclude the Citrix Server. :/
Now I need to rebuild the backup jobs to exclude the Citrix Server. :/
Product Management Analyst @ Veeam Software
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
I haven’t tested this with sure backup (I used it for a replication jobs), but there is a way to change the UUID for a vm using Powershell.
You need to find a way to execute this script before the vm is powered on by surebackup. I don’t know if that’s possible or not.
Code: Select all
$VMs = get-vm -location "<vm folder>"
foreach ($vm in $VMs){
$newUuid = new-guid
echo "VM: " $VM.name "New UUID: " $newuuid
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.uuid = $newUuid
$vm.Extensiondata.ReconfigVM_Task($spec)
start-sleep -s 2
}
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Thanks for the script. I will have a look at it.
We have backup jobs per customer, i will remove the citrix vms from the jobs and create a dedicated Backup job for the citrix vms, if there is no other option (missing surebackup vm exclusion here )
We don‘t use tags at the moment. We are looking in other automation methods. But tags could be an option someday.
We have backup jobs per customer, i will remove the citrix vms from the jobs and create a dedicated Backup job for the citrix vms, if there is no other option (missing surebackup vm exclusion here )
We don‘t use tags at the moment. We are looking in other automation methods. But tags could be an option someday.
Product Management Analyst @ Veeam Software
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
We have found a solution for us.
We removed one old ESXI Host from the vcenter and it is now a standalone host.
We are doing the surebackups on this host, the Citrix Director can not see the BIOS UUID.
We removed one old ESXI Host from the vcenter and it is now a standalone host.
We are doing the surebackups on this host, the Citrix Director can not see the BIOS UUID.
Product Management Analyst @ Veeam Software
-
- Veteran
- Posts: 528
- Liked: 104 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
Great! That’s indeed also a solution! Although not everyone has a spare esxi host laying around
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Virtual lab and Citrix Virtual apps/desktops
It's a really oooooooooooooold esxi host I think, its from 2012 or 2013, but it works
256 GB RAM and 1Gbit Interface is enough at the moment to test 5-10 vms booting up at the same time with heartbeat test.
256 GB RAM and 1Gbit Interface is enough at the moment to test 5-10 vms booting up at the same time with heartbeat test.
Product Management Analyst @ Veeam Software
Who is online
Users browsing this forum: AdsBot [Google], mattskalecki, restore-helper, ronnmartin61 and 93 guests