VBO 3.0 Upgrade Repository

[agrob]

Good Day

Just upgraded VBO 2.x to 3.0 this morning. As in the release notes described, i need to upgrade the Repository. I have 3 Repositories. The biggest one with over 3TB is for Exchange Backup. Upgrade took about 30 minutes. Second repository is about 400GB and is used for Sharepoint Backup. This upgrade is running now for about 3-4hours and still running. Gui shows 10% complete. Also the update did use about 30GB of extra diskspace until now (and still using more and more of the free disk space). I leave it running until complete, but i wondering if this is really normal? (havent read about that in the relase notes, or have i missed something)

I also got this messag when i did try to update sharepoint backup job
<Backup Job Name> Unable to identify the template of some sites (see logs for details). Personal sites exclusion can work incorrectly after this upgrade. Consider job reconfiguration.
When i go to this Backup job and go to the selection list and click "add" after some seconds i get the message "Exception has been thronw by the garget of an invocation". Could those two "error" message related to the fact that the repo is still in the upgrade process?

Repo Volumes are located on a NAS (8x4TB Disk, connected over iSCSI over 10Gbit LAN)

Thanks

[nielsengelen]

Database upgrades can take some time and therefore I suggest to wait until everything has finished before adjusting jobs and settings. Once it has finished, please try to update the job again and if you keep getting the Exception error, please contact support.

[agrob]

Thanks Niels. I'll wait for the upgrade until it is complete. Maybe it would be a good idea to mention that the upgrade can take hours and that it can need a bigger amount from free space somewhere in the release notes... (at least i havent seen it in the notes)

[Polina]

Andre,

It takes more time to upgrade SharePoint databases because of the numerous changes introduced in v3 for storing and processing SharePoint items. Databases are upgraded consequently year by year, and those with many smaller items will take more time to be upgraded (see your 10% progress), while others could be updated much quicker.

Next, the "Unable to identify the template of some sites" warning is thrown, when VBO is not able to automatically identify if a site in a job is personal (i.e. based on a template for personal sites) or not. Remember that in v3, it's now possible to select or exclude personal sites via a personal sites node, and this drives the need for sites templates verification. To fix this, all you need is to edit the job and re-add these failed sites again.

[agrob]

Thanks Polina

About the Error: If i edit the job, go to the slection list and click "add" the window is opening which list me all the available sites. I get the other error bevor the list is complete and therefore i cant select anything... The same happens if i try to create a new job. Anyway, i'll wait until the upgrade is complete and try again. if it still fails i'll update the post here

[agrob]

Ok, i really shoud have been more patient...

Error "Exception has been thrown by the garget of an invocation" when editing the job is gone now. It is really necessary to update the repos first after upgrade. If repo upgrade is finished, then update or edit existing backup jobs. otherwise you will get the error described above.
Repository update for sharepoint Backup job (about 500GB of Sharepoint Data) took about 5-6 hours. So if you do an update plan enough time for the repo updates and also make sure you have enough free disk space.

[Polina]

Andre,

Thanks for getting back on this. I'm sure the information will be useful for others who upgrade to 3.0.
"Exception has been thrown by the target of an invocation" is not specific to VBO, but nevertheless I'll ask our developers to inspect it.

[m.novelli]

Thanks Niels. I'll wait for the upgrade until it is complete. Maybe it would be a good idea to mention that the upgrade can take hours and that it can need a bigger amount from free space somewhere in the release notes... (at least i havent seen it in the notes)

Yes, this is an important info that has been skipped in release notes of VBO 3.0 :-/

Ciao

[Polina]

Good point, Marco. We'll update the RN doc

[Camso.Inc]

I can share my experience with this upgrade from v2 to v3. I was also attempting to see where in the RN it does say something that remotely warns users of this.I found this: "Note that due to many underlying changes introduced for SharePoint Online processing and data storage, it may take time to upgrade repositories containing SharePoint data."

I upgraded to version 3 yesterday and I am currently over 24 hours into doing the "update" to my repo that contains my Sharepoint data.. GUI says 4% completed. In that time it has consumed 215GB of space while performing the "update" and my backup job is currently stopped until it completes. At this rate I should be done with the repo "update" by the end of next week.

Information:
Sharepoint data: 1.9TB worth on the repo
Storage: HPe MSA2050 FC 24x12TB (exclusively used for backup purposes)

VBO server is simple quad core, 16gb, Windows Server 2016. These resources do not seem to be hindering the process as nothing is peaking. All NTFS volumes, no REfs. Veeam Proxy is constant 10-15% CPU usage 10gb RAM.

Case was opened with Veeam support #03552389. Summary of this case was, nothing to do... wait.

Hope this can help some users better plan for this upgrade in terms of space and time requirements.

[Mike Resseler]

Camso,
Thanks for the feedback. I obviously am not happy with this delay and this will be discussed internally for further versions
Mike

[Camso.Inc]

Just to keep the discussion going. Here is an update 24h+ later:

- Still at 4% completion
- Its eaten up about 60GB of free disk space at this time. Based on the time it took to process my 2019 data, it should be close to finishing my 2018 data. The process should accelerate at that point as my biggest ingest of data was from 2018 and 2019.
- Veeam Proxy usage has remained stable 10% with 10gb RAM usage.

Whatever its doing it looks like to going through and reading each and every Sharepoint item one at a time and manipulating the data in the Jet Blue database. When you have 800 and 900GB files for each year I assume it goes slowly

[Polina]

Camso,

Your assumption is correct, when upgrading a repository VBO reads each SharePoint item which is costly time-wise. However, even it has taken 24+ hour to get to the 4% progress it doesn't necessarily mean that the upgrade speed will remain the same for the other 96%. There are good chances for the rest of your repository to be upgraded with a better performance. Please keep up posted with the updates on your case.

[Camso.Inc]

Update:

The "update" of the repository took between 72-96 hours to complete. Total I would estimate at 300GB of additional space used on disk. I cannot seem to find logs to pinpoint exact end-time of the "update". As of right now, each job (I have them separated by data type, so Exchange, Sharepoint, OneDrive, etc...) jobs are running in (Full) mode, this appears to be expected behavior after the upgrade. It seems to be reading each item on the Office 365 side and only copying the changes that occurred. I have definitely been throttled on the tenant side as now I have all my jobs running in full.

This means that since last Wednesday AM, I have not had a successful backup since performing the V2 to V3 upgrade. Hopefully once these (Full) are completed I could start to see a performance improvement from this version VS the previous V2.

I can see going 1 week without backup of our Office 365 tenant due to this upgrade. I am not sure how I feel about that.

As a precaution, due to all this data manipulation, we are forced to perform comprehensive restore tests to make sure the data is still usable. I am honestly not thrilled about this point. It would have been faster to archive this instance of Veeam for Microsoft Office 365 with its data and start from scratch... but hindsight is 20/20....

[Camso.Inc]

Perhaps another side note on this for users: Since this upgrade manipulates all the items in the Jet Blue database, if you backup your Veeam for Office 365 machine take into consideration that your next incremental backup of this machine will be extremely large as well.

[Mike Resseler]

First, to all, I apologize for this. We should have made this much more clear in our Release notes that this upgrade is necessary to gain afterwards the improvements. If we need to do this again (which probably will in some new version, not saying which one ) we will make this much more clear and visible that this is necessary and will require maintenance window or something like that.

The biggest question for me... Is it enough to state this in the RN (knowing that probably not a lot of people read those thoroughly...) so please give us suggestions how we can notify you better.

[Camso.Inc]

Reading the RN is really the only location this information should be located. Anybody who does not read it or at least open it and run through it quickly (fix, requirements and known issues) before upgrading is hopeless. I also believe that simply stating "it may take time to upgrade repositories containing SharePoint data" is not enough information.

I know how long it takes to perform Sharepoint restore and even navigate in the Veeam Explorer for Sharepoint, knowing that it was going to read each individual items in the Jet Blue database and rearrange it in some way would have given a heads up. Perhaps its simply stating what this "it may take time to upgrade repositories containing SharePoint data" is actually doing would be enough. I do not believe adding a chart with estimated times is viable as their are too many variables to account for. Perhaps having this additional information will make the customer pause, perhaps seek the forums or Veeam support for clarification about this.

[Mike Resseler]

Thanks Camso for your take on it. I appreciate it

[poombah]

Did anyone have a repository upgrade that did not complete?

Ours has been sitting on 22% for around 36 hours now - I assume we just keep waiting?

[leon.bollerup]

Update:

The "update" of the repository took between 72-96 hours to complete. Total I would estimate at 300GB of additional space used on disk. I cannot seem to find logs to pinpoint exact end-time of the "update". As of right now, each job (I have them separated by data type, so Exchange, Sharepoint, OneDrive, etc...) jobs are running in (Full) mode, this appears to be expected behavior after the upgrade. It seems to be reading each item on the Office 365 side and only copying the changes that occurred. I have definitely been throttled on the tenant side as now I have all my jobs running in full.

This means that since last Wednesday AM, I have not had a successful backup since performing the V2 to V3 upgrade. Hopefully once these (Full) are completed I could start to see a performance improvement from this version VS the previous V2.

I can see going 1 week without backup of our Office 365 tenant due to this upgrade. I am not sure how I feel about that.

As a precaution, due to all this data manipulation, we are forced to perform comprehensive restore tests to make sure the data is still usable. I am honestly not thrilled about this point. It would have been faster to archive this instance of Veeam for Microsoft Office 365 with its data and start from scratch... but hindsight is 20/20....

.. and thats excactly what we ended doing going from v2 to v3 - it failed with most repos ... sadly

[Mike Resseler]

Poomba, Leon,

Sitting there for 36 hours seems not normal. I know I am late responding here (sorry... VeeamON was busy ) but if jobs are still not finished, support needs to be called in. If it is just waiting, it might be stuck on something specific that we haven't encountered before. If we know it, then we can take that into account for future versions

Please let us know what your status is at this moment
Thanks
Mike

[poombah]

Our upgrade did end up completing after ~ 4 days - however it sat deadlocked on 22% for at least 3.5 days - minimal CPU/Disk activity to indicate anything was actually happening.

[Mike Resseler]

Can I ask you to still create a support call and get me the support case ID? I would like to let those logs investigated. As I said, we might discover something very specific which we didn't encounter in our own testing.

[poombah]

Case # 03578007

Response I had was :

My name is xxxxxxx and I am the Engineer that will work with you until the issue is resolved.

The process being stuck at 22% is not an issue the Upgrade of the repository is actually continuing and can take up to five days.

Please let the process continue until complete. Stopping the process could result in database corruption.

Thank you,
xxxxxx xxxxxxx
Veeam Support

[Mike Resseler]

Hey Poombah,

Thanks for letting me know. I will use this case for an internal R&D investigation to check what we can do on it.

Thanks again
Mike

[MoranJ2000]

Coming in a bit late, but I just want to add my two cents to this because I just found it.

Just did the VBO 2.0 -> 3.0 upgrade yesterday, and did it in in the four-hour window between my backups because I had absolutely no idea from any of the Veeam materials that I read (and I did read them) that a lengthy repo upgrade would be required before backups could resume.

It's been many hours now, repo upgrade is sitting at 16%, and based on what I read here, it could be days before it finishes and I can do backups again.

What makes matters worse is that this thread was from April/May and I stepped in the same doo-doo pile in July, so Veeam, you're clearly still not doing a good job of notifying people what's truly involved in performing this upgrade. Had I known, I would have done it over a weekend, although that still might not have been enough time to avoid missing some backups.

Not happy right now.

[Polina]

Hi Joseph and welcome to Veeam Forums!

First, I apologize for this inconvenience. As said above, we should have updated the Release Notes earlier to inform you about the lengthy upgrade window.
Please keep us updated on the progress.

[MoranJ2000]

Hi.

After almost 36 hours, my VBO repository finally upgraded and I was able to resume backups.

Overall I am a happy Veeam customer, but must tell you-- what especially bothers me about this situation is that not only did Veeam not provide any warning about the potential length of this process in either the KB or the release notes that accompanied the software's initial release, event after all the issues recounted in this thread during April and May, I download the release notes more than a month later in early July and the document STILL has not been updated to reflect the reality of this issue.

I quote from the VBO 3.0 release notes currently live on the download site:

"Important note: Due to many underlying changes introduced for SharePoint Online processing and data storage,
it may take time to upgrade repositories containing SharePoint data. The upgrade process requires an additional
15% of free space of the current repository size."

"It may take time" is woefully inadequately to describe a process that can range from many, many, hours to several days, and look like it's stalled virtually the entire time. And I get that there are variables such as size of repo and speed of storage, etc. that prevent you from knowing in advance how long a given upgrade will take, but based on internal testing and formulas you folks should at least be able to provide a rough estimate or general guidelines so that people at least know how very lengthy the upgrade process can be and thus don't attempt to do it during a few hours long maintenance window and wind up with their repo offline for days as a result.

I've said my peace, having learned a valuable lesson.

Joe

[Polina]

Joe,

Thank you for the feedback, your concerns are fully understandable. We also have a lesson learned for the future and will keep the documentation up to date and more detailed (on a side note, the latest RN version is available on the website since May, and we hoped it would help).

[dustinn3]

Polina, I am having the same issue with my VBO 4.0 upgrade taking multiple days and there is absolutely no documentation in the release notes about the amount of time it will take. I'm already on day 2 and it's still at 20% on a 2 TB disk. I'm also appalled that a backup vendor would knowingly let a customer deploy a product that will take their system offline for multiple days without any type of warning.