Maintain control of your Microsoft 365 data
agrob
Veteran
Posts: 380
Liked: 48 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

VBO 3.0 Upgrade Repository

Post by agrob »

Good Day

Just upgraded VBO 2.x to 3.0 this morning. As in the release notes described, i need to upgrade the Repository. I have 3 Repositories. The biggest one with over 3TB is for Exchange Backup. Upgrade took about 30 minutes. Second repository is about 400GB and is used for Sharepoint Backup. This upgrade is running now for about 3-4hours and still running. Gui shows 10% complete. Also the update did use about 30GB of extra diskspace until now (and still using more and more of the free disk space). I leave it running until complete, but i wondering if this is really normal? (havent read about that in the relase notes, or have i missed something)

I also got this messag when i did try to update sharepoint backup job
<Backup Job Name> Unable to identify the template of some sites (see logs for details). Personal sites exclusion can work incorrectly after this upgrade. Consider job reconfiguration.
When i go to this Backup job and go to the selection list and click "add" after some seconds i get the message "Exception has been thronw by the garget of an invocation". Could those two "error" message related to the fact that the repo is still in the upgrade process?

Repo Volumes are located on a NAS (8x4TB Disk, connected over iSCSI over 10Gbit LAN)

Thanks
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: VBO 3.0 Upgrade Repository

Post by nielsengelen »

Database upgrades can take some time and therefore I suggest to wait until everything has finished before adjusting jobs and settings. Once it has finished, please try to update the job again and if you keep getting the Exception error, please contact support.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
agrob
Veteran
Posts: 380
Liked: 48 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: VBO 3.0 Upgrade Repository

Post by agrob » 1 person likes this post

Thanks Niels. I'll wait for the upgrade until it is complete. Maybe it would be a good idea to mention that the upgrade can take hours and that it can need a bigger amount from free space somewhere in the release notes... (at least i havent seen it in the notes)
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Polina »

Andre,

It takes more time to upgrade SharePoint databases because of the numerous changes introduced in v3 for storing and processing SharePoint items. Databases are upgraded consequently year by year, and those with many smaller items will take more time to be upgraded (see your 10% progress), while others could be updated much quicker.

Next, the "Unable to identify the template of some sites" warning is thrown, when VBO is not able to automatically identify if a site in a job is personal (i.e. based on a template for personal sites) or not. Remember that in v3, it's now possible to select or exclude personal sites via a personal sites node, and this drives the need for sites templates verification. To fix this, all you need is to edit the job and re-add these failed sites again.
agrob
Veteran
Posts: 380
Liked: 48 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: VBO 3.0 Upgrade Repository

Post by agrob »

Thanks Polina

About the Error: If i edit the job, go to the slection list and click "add" the window is opening which list me all the available sites. I get the other error bevor the list is complete and therefore i cant select anything... The same happens if i try to create a new job. Anyway, i'll wait until the upgrade is complete and try again. if it still fails i'll update the post here
agrob
Veteran
Posts: 380
Liked: 48 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: VBO 3.0 Upgrade Repository

Post by agrob »

Ok, i really shoud have been more patient...

Error "Exception has been thrown by the garget of an invocation" when editing the job is gone now. It is really necessary to update the repos first after upgrade. If repo upgrade is finished, then update or edit existing backup jobs. otherwise you will get the error described above.
Repository update for sharepoint Backup job (about 500GB of Sharepoint Data) took about 5-6 hours. So if you do an update plan enough time for the repo updates and also make sure you have enough free disk space.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Polina »

Andre,

Thanks for getting back on this. I'm sure the information will be useful for others who upgrade to 3.0.
"Exception has been thrown by the target of an invocation" is not specific to VBO, but nevertheless I'll ask our developers to inspect it.
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: VBO 3.0 Upgrade Repository

Post by m.novelli »

agrob wrote: Apr 10, 2019 11:41 am Thanks Niels. I'll wait for the upgrade until it is complete. Maybe it would be a good idea to mention that the upgrade can take hours and that it can need a bigger amount from free space somewhere in the release notes... (at least i havent seen it in the notes)
Yes, this is an important info that has been skipped in release notes of VBO 3.0 :-/

Ciao
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Polina »

Good point, Marco. We'll update the RN doc
Camso.Inc
Novice
Posts: 6
Liked: 2 times
Joined: Aug 16, 2017 12:52 pm
Full Name: Camso Inc
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Camso.Inc »

I can share my experience with this upgrade from v2 to v3. I was also attempting to see where in the RN it does say something that remotely warns users of this.I found this: "Note that due to many underlying changes introduced for SharePoint Online processing and data storage, it may take time to upgrade repositories containing SharePoint data."

I upgraded to version 3 yesterday and I am currently over 24 hours into doing the "update" to my repo that contains my Sharepoint data.. GUI says 4% completed. In that time it has consumed 215GB of space while performing the "update" and my backup job is currently stopped until it completes. At this rate I should be done with the repo "update" by the end of next week.

Information:
Sharepoint data: 1.9TB worth on the repo
Storage: HPe MSA2050 FC 24x12TB (exclusively used for backup purposes)

VBO server is simple quad core, 16gb, Windows Server 2016. These resources do not seem to be hindering the process as nothing is peaking. All NTFS volumes, no REfs. Veeam Proxy is constant 10-15% CPU usage 10gb RAM.

Case was opened with Veeam support #03552389. Summary of this case was, nothing to do... wait.

Hope this can help some users better plan for this upgrade in terms of space and time requirements.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Mike Resseler »

Camso,
Thanks for the feedback. I obviously am not happy with this delay and this will be discussed internally for further versions
Mike
Camso.Inc
Novice
Posts: 6
Liked: 2 times
Joined: Aug 16, 2017 12:52 pm
Full Name: Camso Inc
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Camso.Inc »

Just to keep the discussion going. Here is an update 24h+ later:

- Still at 4% completion
- Its eaten up about 60GB of free disk space at this time. Based on the time it took to process my 2019 data, it should be close to finishing my 2018 data. The process should accelerate at that point as my biggest ingest of data was from 2018 and 2019.
- Veeam Proxy usage has remained stable 10% with 10gb RAM usage.

Whatever its doing it looks like to going through and reading each and every Sharepoint item one at a time and manipulating the data in the Jet Blue database. When you have 800 and 900GB files for each year I assume it goes slowly
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Polina »

Camso,

Your assumption is correct, when upgrading a repository VBO reads each SharePoint item which is costly time-wise. However, even it has taken 24+ hour to get to the 4% progress it doesn't necessarily mean that the upgrade speed will remain the same for the other 96%. There are good chances for the rest of your repository to be upgraded with a better performance. Please keep up posted with the updates on your case.
Camso.Inc
Novice
Posts: 6
Liked: 2 times
Joined: Aug 16, 2017 12:52 pm
Full Name: Camso Inc
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Camso.Inc » 1 person likes this post

Update:

The "update" of the repository took between 72-96 hours to complete. Total I would estimate at 300GB of additional space used on disk. I cannot seem to find logs to pinpoint exact end-time of the "update". As of right now, each job (I have them separated by data type, so Exchange, Sharepoint, OneDrive, etc...) jobs are running in (Full) mode, this appears to be expected behavior after the upgrade. It seems to be reading each item on the Office 365 side and only copying the changes that occurred. I have definitely been throttled on the tenant side as now I have all my jobs running in full.

This means that since last Wednesday AM, I have not had a successful backup since performing the V2 to V3 upgrade. Hopefully once these (Full) are completed I could start to see a performance improvement from this version VS the previous V2.

I can see going 1 week without backup of our Office 365 tenant due to this upgrade. I am not sure how I feel about that.

As a precaution, due to all this data manipulation, we are forced to perform comprehensive restore tests to make sure the data is still usable. I am honestly not thrilled about this point. It would have been faster to archive this instance of Veeam for Microsoft Office 365 with its data and start from scratch... but hindsight is 20/20....
Camso.Inc
Novice
Posts: 6
Liked: 2 times
Joined: Aug 16, 2017 12:52 pm
Full Name: Camso Inc
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Camso.Inc »

Perhaps another side note on this for users: Since this upgrade manipulates all the items in the Jet Blue database, if you backup your Veeam for Office 365 machine take into consideration that your next incremental backup of this machine will be extremely large as well.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Mike Resseler »

First, to all, I apologize for this. We should have made this much more clear in our Release notes that this upgrade is necessary to gain afterwards the improvements. If we need to do this again (which probably will in some new version, not saying which one :-)) we will make this much more clear and visible that this is necessary and will require maintenance window or something like that.

The biggest question for me... Is it enough to state this in the RN (knowing that probably not a lot of people read those thoroughly...) so please give us suggestions how we can notify you better.
Camso.Inc
Novice
Posts: 6
Liked: 2 times
Joined: Aug 16, 2017 12:52 pm
Full Name: Camso Inc
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Camso.Inc » 1 person likes this post

Reading the RN is really the only location this information should be located. Anybody who does not read it or at least open it and run through it quickly (fix, requirements and known issues) before upgrading is hopeless. I also believe that simply stating "it may take time to upgrade repositories containing SharePoint data" is not enough information.

I know how long it takes to perform Sharepoint restore and even navigate in the Veeam Explorer for Sharepoint, knowing that it was going to read each individual items in the Jet Blue database and rearrange it in some way would have given a heads up. Perhaps its simply stating what this "it may take time to upgrade repositories containing SharePoint data" is actually doing would be enough. I do not believe adding a chart with estimated times is viable as their are too many variables to account for. Perhaps having this additional information will make the customer pause, perhaps seek the forums or Veeam support for clarification about this.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Mike Resseler »

Thanks Camso for your take on it. I appreciate it
poombah
Novice
Posts: 4
Liked: never
Joined: May 23, 2017 5:06 am
Full Name: Russell Cherry-Knight
Contact:

Re: VBO 3.0 Upgrade Repository

Post by poombah »

Did anyone have a repository upgrade that did not complete?

Ours has been sitting on 22% for around 36 hours now - I assume we just keep waiting?
leon.bollerup
Novice
Posts: 5
Liked: never
Joined: May 08, 2019 6:47 am
Full Name: Leon Bollerup
Contact:

Re: VBO 3.0 Upgrade Repository

Post by leon.bollerup »

Camso.Inc wrote: May 13, 2019 7:40 pm Update:

The "update" of the repository took between 72-96 hours to complete. Total I would estimate at 300GB of additional space used on disk. I cannot seem to find logs to pinpoint exact end-time of the "update". As of right now, each job (I have them separated by data type, so Exchange, Sharepoint, OneDrive, etc...) jobs are running in (Full) mode, this appears to be expected behavior after the upgrade. It seems to be reading each item on the Office 365 side and only copying the changes that occurred. I have definitely been throttled on the tenant side as now I have all my jobs running in full.

This means that since last Wednesday AM, I have not had a successful backup since performing the V2 to V3 upgrade. Hopefully once these (Full) are completed I could start to see a performance improvement from this version VS the previous V2.

I can see going 1 week without backup of our Office 365 tenant due to this upgrade. I am not sure how I feel about that.

As a precaution, due to all this data manipulation, we are forced to perform comprehensive restore tests to make sure the data is still usable. I am honestly not thrilled about this point. It would have been faster to archive this instance of Veeam for Microsoft Office 365 with its data and start from scratch... but hindsight is 20/20....
.. and thats excactly what we ended doing going from v2 to v3 - it failed with most repos ... sadly
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Mike Resseler »

Poomba, Leon,

Sitting there for 36 hours seems not normal. I know I am late responding here (sorry... VeeamON was busy ;-)) but if jobs are still not finished, support needs to be called in. If it is just waiting, it might be stuck on something specific that we haven't encountered before. If we know it, then we can take that into account for future versions

Please let us know what your status is at this moment
Thanks
Mike
poombah
Novice
Posts: 4
Liked: never
Joined: May 23, 2017 5:06 am
Full Name: Russell Cherry-Knight
Contact:

Re: VBO 3.0 Upgrade Repository

Post by poombah »

Our upgrade did end up completing after ~ 4 days - however it sat deadlocked on 22% for at least 3.5 days - minimal CPU/Disk activity to indicate anything was actually happening.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Mike Resseler »

Can I ask you to still create a support call and get me the support case ID? I would like to let those logs investigated. As I said, we might discover something very specific which we didn't encounter in our own testing.
poombah
Novice
Posts: 4
Liked: never
Joined: May 23, 2017 5:06 am
Full Name: Russell Cherry-Knight
Contact:

Re: VBO 3.0 Upgrade Repository

Post by poombah »

Case # 03578007

Response I had was :

My name is xxxxxxx and I am the Engineer that will work with you until the issue is resolved.

The process being stuck at 22% is not an issue the Upgrade of the repository is actually continuing and can take up to five days.

Please let the process continue until complete. Stopping the process could result in database corruption.

Thank you,
xxxxxx xxxxxxx
Veeam Support
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Mike Resseler »

Hey Poombah,

Thanks for letting me know. I will use this case for an internal R&D investigation to check what we can do on it.

Thanks again
Mike
MoranJ2000
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 02, 2019 6:29 pm
Full Name: Joseph Moran
Contact:

Re: VBO 3.0 Upgrade Repository

Post by MoranJ2000 »

Coming in a bit late, but I just want to add my two cents to this because I just found it.

Just did the VBO 2.0 -> 3.0 upgrade yesterday, and did it in in the four-hour window between my backups because I had absolutely no idea from any of the Veeam materials that I read (and I did read them) that a lengthy repo upgrade would be required before backups could resume.

It's been many hours now, repo upgrade is sitting at 16%, and based on what I read here, it could be days before it finishes and I can do backups again.

What makes matters worse is that this thread was from April/May and I stepped in the same doo-doo pile in July, so Veeam, you're clearly still not doing a good job of notifying people what's truly involved in performing this upgrade. Had I known, I would have done it over a weekend, although that still might not have been enough time to avoid missing some backups.

Not happy right now.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Polina »

Hi Joseph and welcome to Veeam Forums!

First, I apologize for this inconvenience. As said above, we should have updated the Release Notes earlier to inform you about the lengthy upgrade window.
Please keep us updated on the progress.
MoranJ2000
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 02, 2019 6:29 pm
Full Name: Joseph Moran
Contact:

Re: VBO 3.0 Upgrade Repository

Post by MoranJ2000 »

Hi.

After almost 36 hours, my VBO repository finally upgraded and I was able to resume backups.

Overall I am a happy Veeam customer, but must tell you-- what especially bothers me about this situation is that not only did Veeam not provide any warning about the potential length of this process in either the KB or the release notes that accompanied the software's initial release, event after all the issues recounted in this thread during April and May, I download the release notes more than a month later in early July and the document STILL has not been updated to reflect the reality of this issue.

I quote from the VBO 3.0 release notes currently live on the download site:

"Important note: Due to many underlying changes introduced for SharePoint Online processing and data storage,
it may take time to upgrade repositories containing SharePoint data. The upgrade process requires an additional
15% of free space of the current repository size."

"It may take time" is woefully inadequately to describe a process that can range from many, many, hours to several days, and look like it's stalled virtually the entire time. And I get that there are variables such as size of repo and speed of storage, etc. that prevent you from knowing in advance how long a given upgrade will take, but based on internal testing and formulas you folks should at least be able to provide a rough estimate or general guidelines so that people at least know how very lengthy the upgrade process can be and thus don't attempt to do it during a few hours long maintenance window and wind up with their repo offline for days as a result.

I've said my peace, having learned a valuable lesson.

Joe
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VBO 3.0 Upgrade Repository

Post by Polina »

Joe,

Thank you for the feedback, your concerns are fully understandable. We also have a lesson learned for the future and will keep the documentation up to date and more detailed (on a side note, the latest RN version is available on the website since May, and we hoped it would help).
dustinn3
Influencer
Posts: 22
Liked: 6 times
Joined: Oct 14, 2013 1:53 pm
Full Name: Dustin Newby
Contact:

Re: VBO 3.0 Upgrade Repository

Post by dustinn3 »

Polina, I am having the same issue with my VBO 4.0 upgrade taking multiple days and there is absolutely no documentation in the release notes about the amount of time it will take. I'm already on day 2 and it's still at 20% on a 2 TB disk. I'm also appalled that a backup vendor would knowingly let a customer deploy a product that will take their system offline for multiple days without any type of warning.
Post Reply

Who is online

Users browsing this forum: No registered users and 22 guests