Business categorization for your virtual environment
Post Reply
jtupeck
Enthusiast
Posts: 49
Liked: 9 times
Joined: Aug 27, 2013 3:44 pm
Full Name: Jason Tupeck
Contact:

VeeamONE Update 4 Performance Issues with Business View

Post by jtupeck » Apr 03, 2019 8:58 pm 1 person likes this post

Support Case ID #03496505

I am having massive slowness issues with VeeamONE Business View (and ONLY Business View) since upgrading to Update 4 about a month and a half ago, shortly after release. Rather than retyping everything, here are my case details that I logged with Support:

Ever since upgrading to Update 4 of VeeamONE, Business View categories and groups have been severely messed up and performance is dog slow. I have been working around it, because I have not had time to open a case with Veeam Support, but finally am getting around to it.

Our VeeamONE instance is tied into multiple vCenter instances and has been the central process for tagging VMs for about 2 years, using the Business View Web client. The web client was slow, but was manageable and did its function well - certainly MUCH more efficient than individually editing tags in vCenter. Tags drive ALL our backup, replication, VAO fail over plan, data retention policies, and much, much more. With Update 4, managing tags has come to a screeching halt. It takes MINUTES, and sometimes TENS of MINUTES for a manual categorization process to complete and display changes in Business View. Just searching a group for a subset of VMs based on name can take abnormally long.

In addition to the slowness issues that I am experiencing, I also see issues with the tag sync with vCenter, now that VeeamONE allows for multiple groups selections, while vCenter tags are only allowed one per VM. I notice that if more than one box is checked in Business View, it will not sync with vCenter correctly. This is an issue, because we had rules in Business View.OLD that would re-group VMs based on VM name. For instance...if a server was decommissioned, we would rename it in vCenter and then Business View would automate the retagging of the VM with 'decommission' polices, removing it from backups, replication process, VM counts, fail over plans, etc. It really streamlined our server onboarding and offboarding processes. This does not work, now...though I have made some grouping rules changes that seem to help, its not perfect like it was before.

I should probably note that Business View categorization and grouping seems to be the only thing affected since applying Update 4. Other areas of VeeamONE Monitor Client seem to be responsive. We use minimal [VeeamONE] alerting features, but use the Infrastructure View all the time, as well as manage/view reports that are driven by the categories and groups. These reports are often inaccurate, due to the grouping/tagging issues.
Has anyone else seen this, or similar performance issues since updating to U4? It is seriously, PAINFULLY slow and makes me wish to have the web console again. It was slightly slow when trying to do many things at once, but it was at least accurate and did it's job to perfection. I am hoping we can get a resolution, now that I have time to work with Support on this, but I wanted to post in the forums to see if anyone else had noticed anything similar.

Shestakov
Veeam Software
Posts: 6836
Liked: 697 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: VeeamONE Update 4 Performance Issues with Business View

Post by Shestakov » Apr 04, 2019 3:05 pm 1 person likes this post

Hi Jason,
I remember one similar case from the time of release, but issue is not that popular.
What you are describing is not a normal behavior so it worth working with the support team.

jtupeck
Enthusiast
Posts: 49
Liked: 9 times
Joined: Aug 27, 2013 3:44 pm
Full Name: Jason Tupeck
Contact:

Re: VeeamONE Update 4 Performance Issues with Business View

Post by jtupeck » Apr 04, 2019 3:07 pm 1 person likes this post

We have a call in about 20 minutes to dig into it, so hopefully it will prove to be a fruitful conversation.

Shestakov
Veeam Software
Posts: 6836
Liked: 697 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: VeeamONE Update 4 Performance Issues with Business View

Post by Shestakov » Apr 04, 2019 4:11 pm 1 person likes this post

I hope so. Let us know the results.

jtupeck
Enthusiast
Posts: 49
Liked: 9 times
Joined: Aug 27, 2013 3:44 pm
Full Name: Jason Tupeck
Contact:

Re: VeeamONE Update 4 Performance Issues with Business View

Post by jtupeck » Apr 05, 2019 9:02 pm 1 person likes this post

No go. So far the tweaks that we have enabled to get certain processes in VeeamONE to work 100% properly are not causing any improvement in performance. I sent the following to support, to address the other outstanding issues I have seen since Update 4, as well.
Mamadou -

In your previous support response, you asked me to think through what I was looking for in the VeeamONE category/group/tag sync process. Hopefully I can articulate it a bit more below. Thank you for your consideration to this and for reading through. I am also cc'ing my regional SE (Vinh Pham) and Michael White (Field Product Manager, Veeam R&D). Vinh is new to my environment and I would like to keep him in the loop on any major issues we have. Michael asked me to file this VeeamONE performance issues support case when he was on site a couple weeks ago to review our VAO installation and DR processes, and I want to keep him in the loop as well.

My ultimate goal is that I want to be able to manually control all my existing tags from VeeamONE while still giving VeeamONE some ability to automate the re-assignment of tags when an condition changes. VeeamONE as our 'tag administration portal', if you will, allows us to have multiple vCenters, yet keep a single, central repository of Tags that can be applied to all our data centers equally and without having to go to each vCenter and manually create the category and tags needed. Both, the manual AND the automating portion of this are key because I need to be able to manually manipulate a group/tag on a server at any time, but the automation of group/tag changes is important, so that processes are orchestrated when we change a specific property (generally the name) of an object. We understand the power of tagging and how we can orchestrate everything from backups, to replicas and failover plans with our grouping/tagging policies. We also generate reports based on these groups/tags, so it is paramount that they work properly. It was working just fine before the most recent Update 4, through combination of Business View rules for automation and static/dynamic categories and groups. What has messed everything up is the new, DEFAULT, ability to allow multiple tags/groups per object.

My understanding is that when setting up a new Category, if I select the 'Single Parameter' Categorization Method, the child groups/tags are all automated based on a single property of a VM. This method does not work for us, because there is no single VM property that will correlate to a backup policy, or a retention policy, etc. Subsequently, the 'Grouping Expression' Categorization Method allows me to code in a specific set of variables that will auto assign groups/tags to an object, but again...none of the parameters in this method are directly tied to that things I need to categorize in regards to my data protection processes.

This leaves the 'Multiple Conditions' Categorization Method as the only method allowing me to MANUALLY change groups/tag AS WELL AS have some automated rules where necessary. The dilemma is this:

'Multiple Conditions' allows for multiple groups/tags to be applied, by default. This is the main issue because a majority of our vSphere Tags are set up as ONE TAG PER OBJECT. (In vCenter, this is set at the 'Add Category' stage with 'Tags Per Object: One tag'). VeeamONE DOES NOT SEEM TO ACCOUNT FOR THIS SETTING and allows multiple groups to be applied to an object, regardless of the setting in vCenter. In my humble opinion, THIS NEEDS TO BE FIXED. If vCenter says 'one tag per object' VeeamONE should be aware of this and any change to an object with an existing group should overwrite the existing group, NOT just add another group to the object. Adding more than one group when vCenter only allows one tag, breaks all kinds of processes and skews reports. (more on this, below)

The biggest use case for this in our environment is in server offboarding/decommissioning. Onboarding is simple; we add the VM and within a short period of time, we are able to apply categorization in VeeamONE, which then syncs to vCenter and drives all our data protection services. Our process for decommissioning a server involves renaming the VM object in vCenter with the text 'DECOMMISSIONED' as a prefix to the VM name. Prior to Update 4, we would rename the VM object in vCenter and VeeamONE's rules would see this change. Based on the new name, a variety of tags would be reassigned as 'Decommissioned' - thereby AUTOMATICALLY removing the VM from nightly backups, retention policies, replication processes and from fail over plans in VAO. Additionally, the VM would be removed from reports that are generated in VeeamONE Reporter. When a server is decommissioned, it no longer needs to be monitored or reported on, but it needs to remain in the vCenter infrastructure for a period of time in case we find an issue with powering off the machine.

This was working flawlessly until Update 4. Update 4 Business View simple ADDS the 'Decommissioned' group to the VeeamONE object, but the vCenter object retains its old tag in all the different categories. Because of this, it causes all kinds of Veeam B&R errors; because now the VM is powered off and cannot be reached for app aware processing, backup copy jobs continue looking for new backup data, but aren’t finding any and therefore report errors or warnings; replication processes error out because there is no backup data to replicate; and VAO Plans still think the VM is part of the plan and error because the replica is out of date, when it is no longer needed. In a best case scenario, where a backup still happens, we still end up with decommissioned servers permeating unnecessarily through our data protection services grid.

EXAMPLE: VM object 'SERVER-123' has the following Business View Category>Group assignments:

Backup Policy > 'Business Critical'
Replication Policy > 'Business Critical'
Data Retention Policy > 'MTR - Mid Term Retention'
VAO Fail Over Plan > 'VAO Test Plan 1'
and many more...


Each of the above Business View Categories is tied to an individual process in our data protection environment. Each of the Categories also has a Group/Tag named 'Decommissioned' with an automated Grouping Condition that reclassifies an object based on its name in vCenter. E.g. "[NAME] > [CONTAINS] > [DECOM]" With this Grouping Condition, when the name on VM object 'SERVER-123' is changed to 'DECOMMISSIONED_Server-123' VeeamONE (Update 4) simply ADDS the corresponding 'Decommissioned' Group to each Category. Previous versions used to REPLACE the existing Group in each of the affected Categories. In this case, this new 'Decommissioned' Group addition does not get synced to vCenter, because vCenter only allows one tag per object and the object VM already contained a tag. VeeamONE, on the other hand, shows both Group boxes as checked, thereby skewing reports and causing havoc with the backup/replication/etc. processes that are dependent on these tags.

Further Examples:

VeeamONE (Update 4) now looks like this for object "DECOMMISSIONED_SERVER-123":

Backup Policy > 'Business Critical', 'Decommissioned'
Replication Policy > 'Business Critical', 'Decommissioned'
Data Retention Policy > 'MTR - Mid Term Retention', 'Decommissioned'
VAO Fail Over Plan > 'VAO Test Plan 1', 'Decommissioned'
Etc...

vCenter still looks like this for object "DECOMMISSIONED_SERVER-123":
Backup Policy > 'Business Critical'
Replication Policy > 'Business Critical'
Data Retention Policy > 'MTR - Mid Term Retention'
VAO Fail Over Plan > 'VAO Test Plan 1'
and many more...

VeeamONE (Update 3a and Older) would have looked like for object "DECOMMISSIONED_SERVER-123":

Backup Policy > 'Decommissioned'
Replication Policy > 'Decommissioned'
Data Retention Policy > 'Decommissioned'
VAO Fail Over Plan > 'Decommissioned'
Etc...

As I have been thinking all of this through, I have thought of a temproarty workaround. When we decommission a machine, we can 'Reset categorization' on the VeeamONE object before changing the name. This *should* cause all tags to be wiped clean and then when VeeamONE sees the new object name, it will apply the individual 'Decommissioned' Group/Tag to each Category. Ultimately, though, I think VeeamONE should take into account the vCenter setting for the Category and determine whether vCenter allows for 'One tag' or 'Many tags' per VM. If only one is allowed, then all existing tags should be overwritten. If many tags are allowed, then all tags should get written to vCenter. In our case, we will likely never use a multiple tags per VM setup; at least not for the processes that I am concerned with here, which are the main ones that are being affected by Update 4's new method of sync.

I understand, I think, more fully where VeeamONE's processing stands at this point in time. I really hope that changes can be made to accommodate the vCenter setup of Categories and Tags, as that seems to be a more logical arrangement. But, all this said, it does nothing to really help me in my environment today. It is looking more and more like I just need to set up a new server, because at this point, so many things are wonky in VeeamONE that I need a fresh start and, I think, will make the change to a dedicated on-board SQL server dedicated to VeeamONE for performance reasons. With that in mind, I still am not finding in the VeeamONE documentation in a way that speaks to me, an answer to this question:

How do I connect vCenter and import all the EXISTING tag infrastructure (and associated tags-per-object) into VeeamONE, THEN allow VeeamONE to manage (add, remove, modify) it all?

Connecting the synchronization to vCenter only allows for changes to be made in vCenter. The other options don't seem to 'map' anything in VeeamONE to vCenter the way the old VeeamONE Business View allowed me to. If I created a new tag in vCenter, I was able to map it to VeeamONE and VeeamONE would create the Category and Group that I could then assign to objects. OR, I could create a new Category/Group in VeeamONE and it would export to vCenter. There doesn’t seem to be any way to do both of these things, now and I don't want to have to recreate 15+ categories and 100+ tags, all with custom properties in VeeamONE, then apply all these new values to 1000 objects and change all my backup/replication/VAO jobs to tie into them. How do VMware users who are new to VeeamONE add their systems with all their existing tagging infrastructure to a new VeeamONE instance and then allow ONE to manage it all? I am either completely blind, or it is not called out in the documentation as to how this is best done.

Thank you for your time and consideration in reading this. I am sure it will take some time to really digest and I would appreciate it if you can escalate this on to the development team for any answers that might be out there.
The final line of that also applies to anyone else who reads this and cares to weigh in. Thanks so much for looking!

Shestakov
Veeam Software
Posts: 6836
Liked: 697 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: VeeamONE Update 4 Performance Issues with Business View

Post by Shestakov » Apr 07, 2019 9:24 pm 1 person likes this post

Thanks for the update, Jason.
I'll ask the support management to escalate the case and involve developers if needed.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest