-
- Service Provider
- Posts: 182
- Liked: 26 times
- Joined: Apr 23, 2021 6:40 am
- Full Name: Sumeet P
- Contact:
Proxy service stops for full sync run
Hi,
The setup is a fresh client with 100+ TB of data to be backed up. I felt of using version 7.0.0.3604, my mistake. Depending on the data size, the setup is designed to have VBO server and a proxy server. Initially the setup was reduced to have 32 threads and the backup for ARchive and exchange was started (as the jobs on different servers), the jobs ran fine for 20+ hours.
The next day we had to enable Teams export API, was not aware this cannot be enabled with jobs running. So I had to stop the jobs which had processed 9+ TB of data across both the jobs.
I got greedy and decided to increase the threads to 64, hoping to push the backups.
But since Thursday night, the proxy service keeps stopping on its own on both the VBO and proxy server.
I have this case 06330388 for three days now, but no much help.
Have provided logs multple times along with system event logs, but each time a different support person picks up the case with different sets of inputs. Increased the resources on the server from 16GB to 24 GB, 8 vCPUs to 12 vCPUs, reduced the threads back to 32. Disabled SentinelOne on the server, which I don't think is a issue, but had to do it for what the support person's observation.
Since yesterday I have disabled Archive job on the VBO server and only have onedrive job running, which has been running more than 24 hours without causing any issues. But the mailbox backup on the proxy keeps failing.
What I do not understand is the obsession from support to look at all other factors apart from the fact that the proxy service fails/dies after consuming all the memory on the server. If the resources are a concern on the server, shouldn't the backups run slow? Why would the service die? I do not open the case after one observation of failure - only after few failures and behavior being consistent, I reach out for help. But even after that no much good.
Is it something with the changes done to Archive/mailbox in version 7 that is causing this issue?
-Sumeet.
The setup is a fresh client with 100+ TB of data to be backed up. I felt of using version 7.0.0.3604, my mistake. Depending on the data size, the setup is designed to have VBO server and a proxy server. Initially the setup was reduced to have 32 threads and the backup for ARchive and exchange was started (as the jobs on different servers), the jobs ran fine for 20+ hours.
The next day we had to enable Teams export API, was not aware this cannot be enabled with jobs running. So I had to stop the jobs which had processed 9+ TB of data across both the jobs.
I got greedy and decided to increase the threads to 64, hoping to push the backups.
But since Thursday night, the proxy service keeps stopping on its own on both the VBO and proxy server.
I have this case 06330388 for three days now, but no much help.
Have provided logs multple times along with system event logs, but each time a different support person picks up the case with different sets of inputs. Increased the resources on the server from 16GB to 24 GB, 8 vCPUs to 12 vCPUs, reduced the threads back to 32. Disabled SentinelOne on the server, which I don't think is a issue, but had to do it for what the support person's observation.
Since yesterday I have disabled Archive job on the VBO server and only have onedrive job running, which has been running more than 24 hours without causing any issues. But the mailbox backup on the proxy keeps failing.
What I do not understand is the obsession from support to look at all other factors apart from the fact that the proxy service fails/dies after consuming all the memory on the server. If the resources are a concern on the server, shouldn't the backups run slow? Why would the service die? I do not open the case after one observation of failure - only after few failures and behavior being consistent, I reach out for help. But even after that no much good.
Is it something with the changes done to Archive/mailbox in version 7 that is causing this issue?
-Sumeet.
-
- Product Manager
- Posts: 10099
- Liked: 2696 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Proxy service stops for full sync run
Hello Sumeet
Have you already used our Talk to a manager option to escalate the case?
We have a known issue for the latest v7 build (7.0.0.3968). Therefore we retracted that version to fix the issue.
Is it possible that you have installed this version instead of 7.0.0.3604?
https://www.veeam.com/kb4481
Best,
Fabian
Have you already used our Talk to a manager option to escalate the case?
We have a known issue for the latest v7 build (7.0.0.3968). Therefore we retracted that version to fix the issue.
Is it possible that you have installed this version instead of 7.0.0.3604?
https://www.veeam.com/kb4481
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Service Provider
- Posts: 182
- Liked: 26 times
- Joined: Apr 23, 2021 6:40 am
- Full Name: Sumeet P
- Contact:
Re: Proxy service stops for full sync run
Hello Mildur,
No I have not used talk to a manager option.
I know about the known issue with the latest build and hence was sure to double check that I install 7.0.0.3604.
Having reduced the threads to 24 on the proxy server which has exhange job seems to have helped as for the first time the exchange backup job has been running for 38 hours without the proxy service failing.
But I see loads of “Cannot access a disposed object. Object name ‘SslStream’” errors for OneDrive job. I see another discussion where people have reported this and that it resolves in subsequent runs. But do you know or anyone else know if any additional details available for this error?
Thanks,
-Sumeet.
No I have not used talk to a manager option.
I know about the known issue with the latest build and hence was sure to double check that I install 7.0.0.3604.
Having reduced the threads to 24 on the proxy server which has exhange job seems to have helped as for the first time the exchange backup job has been running for 38 hours without the proxy service failing.
But I see loads of “Cannot access a disposed object. Object name ‘SslStream’” errors for OneDrive job. I see another discussion where people have reported this and that it resolves in subsequent runs. But do you know or anyone else know if any additional details available for this error?
Thanks,
-Sumeet.
-
- Service Provider
- Posts: 182
- Liked: 26 times
- Joined: Apr 23, 2021 6:40 am
- Full Name: Sumeet P
- Contact:
Re: Proxy service stops for full sync run
We got this to work only by reducing the threads to 16, increasing the memory to 32 GB and vCPU to 12. It took 59 days to backup 159 TB of data for 124xx mailboxes on a dedicated proxy server. We had to do several restarts of the proxy service to get the first full sync to complete.
Sharepoint and onedrive worked fine with 5 additional backup apps and total data of 120 TB.
Not easy to manage the client while the backup keeps running for 2 months to get the first full copy.
During our initial design, microsoft listed total mailbox storage usage as 44.5 TB for primary mailboxes and archive/shared as 18 TB. But without archive, the total data transferred by veeam is 159 TB. More than three times of what msft listed.
We ran powershell commands to get mailbox statistics to compare the size of mailboxes (along with items in the mailbox) to what was downloaded in the backup. These two do not match. Most of the mailbox sizes are 4-5 more in full backup report by veeam as compared to what get-mailbox-statistics reports. The number of items in most cases are 2-3 times more in the backup.
Has anyone seen such discrepencies between the total backup data to what is listed by msft?
Does Veeam have any answers to this?
Sharepoint and onedrive worked fine with 5 additional backup apps and total data of 120 TB.
Not easy to manage the client while the backup keeps running for 2 months to get the first full copy.
During our initial design, microsoft listed total mailbox storage usage as 44.5 TB for primary mailboxes and archive/shared as 18 TB. But without archive, the total data transferred by veeam is 159 TB. More than three times of what msft listed.
We ran powershell commands to get mailbox statistics to compare the size of mailboxes (along with items in the mailbox) to what was downloaded in the backup. These two do not match. Most of the mailbox sizes are 4-5 more in full backup report by veeam as compared to what get-mailbox-statistics reports. The number of items in most cases are 2-3 times more in the backup.
Has anyone seen such discrepencies between the total backup data to what is listed by msft?
Does Veeam have any answers to this?
-
- Service Provider
- Posts: 182
- Liked: 26 times
- Joined: Apr 23, 2021 6:40 am
- Full Name: Sumeet P
- Contact:
Re: Proxy service stops for full sync run
I missed mentioning that the total size for OneDrive and Sharepoint as gathered during design phase, matched (with couple of TBs more) with the first full backup run. It is only the total mailboxes transferred size that is way too off.
-
- Veeam Software
- Posts: 3293
- Liked: 794 times
- Joined: Oct 21, 2011 11:22 am
- Full Name: Polina Vasileva
- Contact:
Re: Proxy service stops for full sync run
Hi Sumeet,
There's some data, for example, attachments to emails, that Veeam has to download and store separately even if there's the same file sent to multiple recipients. Imagine an email with an attachment sent to 10 different users; when you protect these 10 mailboxes, Veeam will download this attachment 10 times and store it 10 times, because there's no indication that this data is the same everywhere.
I can't say if this explains the significant difference in sizes that you're seeing, but it definitely impacts the size of Exchange backups.
There's some data, for example, attachments to emails, that Veeam has to download and store separately even if there's the same file sent to multiple recipients. Imagine an email with an attachment sent to 10 different users; when you protect these 10 mailboxes, Veeam will download this attachment 10 times and store it 10 times, because there's no indication that this data is the same everywhere.
I can't say if this explains the significant difference in sizes that you're seeing, but it definitely impacts the size of Exchange backups.
-
- Service Provider
- Posts: 182
- Liked: 26 times
- Joined: Apr 23, 2021 6:40 am
- Full Name: Sumeet P
- Contact:
Re: Proxy service stops for full sync run
Hi Polina,
Thanks, your answer definitely helps.
Also, do you have additional details on what the behavior is for in-place hold and Litigation hold items. If this is enabled, does the items that are saved with these two settings, list in the users mailbox size as reported by Microsoft?
Regards,
-Sumeet.
Thanks, your answer definitely helps.
Also, do you have additional details on what the behavior is for in-place hold and Litigation hold items. If this is enabled, does the items that are saved with these two settings, list in the users mailbox size as reported by Microsoft?
Regards,
-Sumeet.
-
- Service Provider
- Posts: 182
- Liked: 26 times
- Joined: Apr 23, 2021 6:40 am
- Full Name: Sumeet P
- Contact:
Re: Proxy service stops for full sync run
Hi all,
We figured out the difference between the total mailbox capacity reported by M365 and what we actually backed-up, which was three times more.
For the client we had In-hold and litigation hold items included in the backup.
According to this documentation from MSFT - https://learn.microsoft.com/en-us/excha ... box-quotas
There is separate 100 GB quota allocated when these holds are enabled. This is what it says - Items in the Recoverable Items folder aren't calculated toward the user's mailbox quota.
For smaller clients, this did not make a difference, but with larger ones, where in they have retention policy set to hold items forever, this will make a huge difference.
We learnt this the hard way. Have included this section in our initial discussions with our clients, so that correct expectations are set for the backup.
It will help if Veeam documentation points this out.
Thanks,
-Sumeet.
We figured out the difference between the total mailbox capacity reported by M365 and what we actually backed-up, which was three times more.
For the client we had In-hold and litigation hold items included in the backup.
According to this documentation from MSFT - https://learn.microsoft.com/en-us/excha ... box-quotas
There is separate 100 GB quota allocated when these holds are enabled. This is what it says - Items in the Recoverable Items folder aren't calculated toward the user's mailbox quota.
For smaller clients, this did not make a difference, but with larger ones, where in they have retention policy set to hold items forever, this will make a huge difference.
We learnt this the hard way. Have included this section in our initial discussions with our clients, so that correct expectations are set for the backup.
It will help if Veeam documentation points this out.
Thanks,
-Sumeet.
Who is online
Users browsing this forum: sfey and 19 guests