Virtual lab and Citrix Virtual apps/desktops

[FrancWest]

Hi,

this is probably not a bug in Veeam, but it might be useful information for people who are using Citrix XenApp in their environment. Here it goes:

I noticed probably a bug in the Citrix delivery controller. We use Veeam backup and replication for backing up our environment. When using Sure Backup to verify the backups this causes havoc with the Citrix Machine catalogs.

The Citrix Delivery Controller seems to have an issue with this functionality. When the VM is added to the vSphere environment by the replication job or sure backup job, the UUID of the VM is the same, since this is the original VM from the backup. The same thing happens when you clone a VM. This clone also gets the same UUID. When this situation exists, one of the services running on the Citrix delivery controller replaces the original VM in the machine catalog with the temporary VM from sure backup The end-result of this is that you end-up with a non-working machine catalog, since the temporary VM is being deleted when verification is completed. Also, the delivery controller shouldn't touch the catalog at all.

Further investigation shows that this issue is caused by the fact that the Citrix Broker uses the extensiondata.config.uuid as the unique identified. It stores this value in the HostedMachineID of the broker database. Since the temporary VM is a clone of the original VM, it has the same extensiondata.config.uuid. Apparently there's some re-scanning process in the background running on the broker that for some reason replaces the VM in the catalog without any message or notification.

I opened a case with citrix already.

Franc.

[HannesK]

Hi Frank,
interesting observation. Thanks for sharing.

What I do not understand: how does the production delivery controller know about the VM running in a virtual lab? I mean, there should be no access from production to Virtual Labs.

Best regards,
Hannes

[FrancWest]

Hi Hannes,

Because the delivery controller has a connection to the vcenter server. That’s were it gets it information from.

Franc.

[ronnmartin61]

Hey Franc, I was at Citrix for many years and I need to brush up a bit on my MCS/PVS operation but yes I would think this would be the expected behavior since those machine creation facilities communicate directly to the hypervisor. Is that what you're hearing from Citrix as well?

[FrancWest]

Citrix is still investigating, but they tent to believe it’s indeed the issue. They will propose a code change to development to use the MorefID instead of the UUID as the unique identifier in the MCS machine catalog.

The problem I now have is that I can’t exclude VMs from the surebackup job, so I can’t prevent Veeam from starting up the XenApp VMs and causing my machine catalogs to break. The XenApp VMs are in a backup job with many other VMs. I’ll see if I can work around this by putting the VMs in a separate backup job and not including that backup job in the sure backup job.

[Mildur]

Thanks Franc, you are my hero.
Have the same problem in a customers infrastructure.

I will test this after my vacation next week. Create a backup job „without citrix vms“ as a Source for the SureBackup Job.

[FrancWest]

You're welcome! It would be nice though if you can exclude specific VMs within Veaam for a sure backup job. You can exclude VMs in all other job types but not in sure backup.

There's another thread about this, it's many, many years old but still not implemented:

vmware-vsphere-f24/is-it-possible-to-ex ... t6668.html

[Mildur]

Yes, this would be a nice feature. I have already added me to the feature request there

[Mildur]

We have tested surebackup Job with excluded Citrix Server this morning successfully.
Everything is running fine, as long we don‘t boot up The Citrix Servers in the virtual Lab.

Thanks again Franc, can you post an update as soon you have a solution from Citrix Support?

[FrancWest]

Hi,

I will. Citrix support closed the case, but our account manager will inform me when the fix or hotfix is being implemented. They didn’t gave a timeline though.

[Mildur]

Ok, thanks

[FrancWest]

I reached out to Citrix again, since I haven't heard from them. They say nothing has changed and asked me to contact Veeam to see if they could do a change on their side so that the UUID is changed after creating the replica. I pointed out that it's not a Veeam issue, but a Citrix issue, since I can reproduce it also by simply cloning the VM from vCenter itself. They also asked me to create a new case to research the issue again, where as they first said an enhancement request was made already.... So I guess this is leading to nowhere and we only have the workaround currently in place by using the powershell script to change the UUID's after cloning, or wait for the feature request from Veeam to exclude VMs from replicating.

[ferrus]

I'm hoping to use a Veeam Lab/Sure Backup job, to test an upcoming Citrix Virtual Apps upgrade.
Do you know if this affects just the Delivery Agent VMs running under MCS, or the Infrastructure servers as well?

I was just going to test the upgrade of the Delivery Controller, Database, License Server, Studio, Director and StoreFront. No Delivery Agent VMs in the job.
I won't start the Sure Backup job if it'll affect the live system - but none of the components above are on MCS deployed servers.

[FrancWest]

For us it's only affecting the VDA's since the Citrix Delivery Controller is not using the moRefID,but the bios UUID as the unique identifier to identify the VMs it needs to powercontrol. With the other server I have no issues, since the delivery controller doesn't control them.

[Mildur]

Was there ever a solution from Citrix?
I need to test 20-30 citrix vdas with surebackup.
But if they disconnect from the director, i cannot do it.

[FrancWest]

Unfortunately, no.

[Mildur]

Ok, thanks Franc.

Now I need to rebuild the backup jobs to exclude the Citrix Server. :/

[FrancWest]

I haven’t tested this with sure backup (I used it for a replication jobs), but there is a way to change the UUID for a vm using Powershell.

Code: Select all

 $VMs = get-vm -location "<vm folder>"
foreach ($vm in $VMs){
	$newUuid = new-guid
	echo "VM: " $VM.name "New UUID: " $newuuid
	$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
	$spec.uuid = $newUuid
	$vm.Extensiondata.ReconfigVM_Task($spec)
	start-sleep -s 2
}

You need to find a way to execute this script before the vm is powered on by surebackup. I don’t know if that’s possible or not.

[FrancWest]

Ok, thanks Franc.

Now I need to rebuild the backup jobs to exclude the Citrix Server. :/

Using tags that should be doable.

[Mildur]

Thanks for the script. I will have a look at it.

We have backup jobs per customer, i will remove the citrix vms from the jobs and create a dedicated Backup job for the citrix vms, if there is no other option (missing surebackup vm exclusion here )
We don‘t use tags at the moment. We are looking in other automation methods. But tags could be an option someday.

[Mildur]

We have found a solution for us.
We removed one old ESXI Host from the vcenter and it is now a standalone host.
We are doing the surebackups on this host, the Citrix Director can not see the BIOS UUID.

[FrancWest]

Great! That’s indeed also a solution! Although not everyone has a spare esxi host laying around

[Mildur]

It's a really oooooooooooooold esxi host I think, its from 2012 or 2013, but it works
256 GB RAM and 1Gbit Interface is enough at the moment to test 5-10 vms booting up at the same time with heartbeat test.