No go. So far the tweaks that we have enabled to get certain processes in VeeamONE to work 100% properly are not causing any improvement in performance. I sent the following to support, to address the other outstanding issues I have seen since Update 4, as well.
Mamadou -
In your previous support response, you asked me to think through what I was looking for in the VeeamONE category/group/tag sync process. Hopefully I can articulate it a bit more below. Thank you for your consideration to this and for reading through. I am also cc'ing my regional SE (Vinh Pham) and Michael White (Field Product Manager, Veeam R&D). Vinh is new to my environment and I would like to keep him in the loop on any major issues we have. Michael asked me to file this VeeamONE performance issues support case when he was on site a couple weeks ago to review our VAO installation and DR processes, and I want to keep him in the loop as well.
My ultimate goal is that I want to be able to manually control all my existing tags from VeeamONE while still giving VeeamONE some ability to automate the re-assignment of tags when an condition changes. VeeamONE as our 'tag administration portal', if you will, allows us to have multiple vCenters, yet keep a single, central repository of Tags that can be applied to all our data centers equally and without having to go to each vCenter and manually create the category and tags needed. Both, the manual AND the automating portion of this are key because I need to be able to manually manipulate a group/tag on a server at any time, but the automation of group/tag changes is important, so that processes are orchestrated when we change a specific property (generally the name) of an object. We understand the power of tagging and how we can orchestrate everything from backups, to replicas and failover plans with our grouping/tagging policies. We also generate reports based on these groups/tags, so it is paramount that they work properly. It was working just fine before the most recent Update 4, through combination of Business View rules for automation and static/dynamic categories and groups. What has messed everything up is the new, DEFAULT, ability to allow multiple tags/groups per object.
My understanding is that when setting up a new Category, if I select the 'Single Parameter' Categorization Method, the child groups/tags are all automated based on a single property of a VM. This method does not work for us, because there is no single VM property that will correlate to a backup policy, or a retention policy, etc. Subsequently, the 'Grouping Expression' Categorization Method allows me to code in a specific set of variables that will auto assign groups/tags to an object, but again...none of the parameters in this method are directly tied to that things I need to categorize in regards to my data protection processes.
This leaves the 'Multiple Conditions' Categorization Method as the only method allowing me to MANUALLY change groups/tag AS WELL AS have some automated rules where necessary. The dilemma is this:
'Multiple Conditions' allows for multiple groups/tags to be applied, by default. This is the main issue because a majority of our vSphere Tags are set up as ONE TAG PER OBJECT. (In vCenter, this is set at the 'Add Category' stage with 'Tags Per Object: One tag'). VeeamONE DOES NOT SEEM TO ACCOUNT FOR THIS SETTING and allows multiple groups to be applied to an object, regardless of the setting in vCenter. In my humble opinion, THIS NEEDS TO BE FIXED. If vCenter says 'one tag per object' VeeamONE should be aware of this and any change to an object with an existing group should overwrite the existing group, NOT just add another group to the object. Adding more than one group when vCenter only allows one tag, breaks all kinds of processes and skews reports. (more on this, below)
The biggest use case for this in our environment is in server offboarding/decommissioning. Onboarding is simple; we add the VM and within a short period of time, we are able to apply categorization in VeeamONE, which then syncs to vCenter and drives all our data protection services. Our process for decommissioning a server involves renaming the VM object in vCenter with the text 'DECOMMISSIONED' as a prefix to the VM name. Prior to Update 4, we would rename the VM object in vCenter and VeeamONE's rules would see this change. Based on the new name, a variety of tags would be reassigned as 'Decommissioned' - thereby AUTOMATICALLY removing the VM from nightly backups, retention policies, replication processes and from fail over plans in VAO. Additionally, the VM would be removed from reports that are generated in VeeamONE Reporter. When a server is decommissioned, it no longer needs to be monitored or reported on, but it needs to remain in the vCenter infrastructure for a period of time in case we find an issue with powering off the machine.
This was working flawlessly until Update 4. Update 4 Business View simple ADDS the 'Decommissioned' group to the VeeamONE object, but the vCenter object retains its old tag in all the different categories. Because of this, it causes all kinds of Veeam B&R errors; because now the VM is powered off and cannot be reached for app aware processing, backup copy jobs continue looking for new backup data, but aren’t finding any and therefore report errors or warnings; replication processes error out because there is no backup data to replicate; and VAO Plans still think the VM is part of the plan and error because the replica is out of date, when it is no longer needed. In a best case scenario, where a backup still happens, we still end up with decommissioned servers permeating unnecessarily through our data protection services grid.
EXAMPLE: VM object 'SERVER-123' has the following Business View Category>Group assignments:
Backup Policy > 'Business Critical'
Replication Policy > 'Business Critical'
Data Retention Policy > 'MTR - Mid Term Retention'
VAO Fail Over Plan > 'VAO Test Plan 1'
and many more...
Each of the above Business View Categories is tied to an individual process in our data protection environment. Each of the Categories also has a Group/Tag named 'Decommissioned' with an automated Grouping Condition that reclassifies an object based on its name in vCenter. E.g. "[NAME] > [CONTAINS] > [DECOM]" With this Grouping Condition, when the name on VM object 'SERVER-123' is changed to 'DECOMMISSIONED_Server-123' VeeamONE (Update 4) simply ADDS the corresponding 'Decommissioned' Group to each Category. Previous versions used to REPLACE the existing Group in each of the affected Categories. In this case, this new 'Decommissioned' Group addition does not get synced to vCenter, because vCenter only allows one tag per object and the object VM already contained a tag. VeeamONE, on the other hand, shows both Group boxes as checked, thereby skewing reports and causing havoc with the backup/replication/etc. processes that are dependent on these tags.
Further Examples:
VeeamONE (Update 4) now looks like this for object "DECOMMISSIONED_SERVER-123":
Backup Policy > 'Business Critical', 'Decommissioned'
Replication Policy > 'Business Critical', 'Decommissioned'
Data Retention Policy > 'MTR - Mid Term Retention', 'Decommissioned'
VAO Fail Over Plan > 'VAO Test Plan 1', 'Decommissioned'
Etc...
vCenter still looks like this for object "DECOMMISSIONED_SERVER-123":
Backup Policy > 'Business Critical'
Replication Policy > 'Business Critical'
Data Retention Policy > 'MTR - Mid Term Retention'
VAO Fail Over Plan > 'VAO Test Plan 1'
and many more...
VeeamONE (Update 3a and Older) would have looked like for object "DECOMMISSIONED_SERVER-123":
Backup Policy > 'Decommissioned'
Replication Policy > 'Decommissioned'
Data Retention Policy > 'Decommissioned'
VAO Fail Over Plan > 'Decommissioned'
Etc...
As I have been thinking all of this through, I have thought of a temproarty workaround. When we decommission a machine, we can 'Reset categorization' on the VeeamONE object before changing the name. This *should* cause all tags to be wiped clean and then when VeeamONE sees the new object name, it will apply the individual 'Decommissioned' Group/Tag to each Category. Ultimately, though, I think VeeamONE should take into account the vCenter setting for the Category and determine whether vCenter allows for 'One tag' or 'Many tags' per VM. If only one is allowed, then all existing tags should be overwritten. If many tags are allowed, then all tags should get written to vCenter. In our case, we will likely never use a multiple tags per VM setup; at least not for the processes that I am concerned with here, which are the main ones that are being affected by Update 4's new method of sync.
I understand, I think, more fully where VeeamONE's processing stands at this point in time. I really hope that changes can be made to accommodate the vCenter setup of Categories and Tags, as that seems to be a more logical arrangement. But, all this said, it does nothing to really help me in my environment today. It is looking more and more like I just need to set up a new server, because at this point, so many things are wonky in VeeamONE that I need a fresh start and, I think, will make the change to a dedicated on-board SQL server dedicated to VeeamONE for performance reasons. With that in mind, I still am not finding in the VeeamONE documentation in a way that speaks to me, an answer to this question:
How do I connect vCenter and import all the EXISTING tag infrastructure (and associated tags-per-object) into VeeamONE, THEN allow VeeamONE to manage (add, remove, modify) it all?
Connecting the synchronization to vCenter only allows for changes to be made in vCenter. The other options don't seem to 'map' anything in VeeamONE to vCenter the way the old VeeamONE Business View allowed me to. If I created a new tag in vCenter, I was able to map it to VeeamONE and VeeamONE would create the Category and Group that I could then assign to objects. OR, I could create a new Category/Group in VeeamONE and it would export to vCenter. There doesn’t seem to be any way to do both of these things, now and I don't want to have to recreate 15+ categories and 100+ tags, all with custom properties in VeeamONE, then apply all these new values to 1000 objects and change all my backup/replication/VAO jobs to tie into them. How do VMware users who are new to VeeamONE add their systems with all their existing tagging infrastructure to a new VeeamONE instance and then allow ONE to manage it all? I am either completely blind, or it is not called out in the documentation as to how this is best done.
Thank you for your time and consideration in reading this. I am sure it will take some time to really digest and I would appreciate it if you can escalate this on to the development team for any answers that might be out there.
The final line of that also applies to anyone else who reads this and cares to weigh in. Thanks so much for looking!