-
- Influencer
- Posts: 11
- Liked: never
- Joined: Nov 05, 2009 12:18 pm
- Full Name: AJ GB
- Contact:
Azure Data Box Issue
We have had an Azure Data Box for a week and have been gradually uploading 60TB (100 million files) from two separate veeam 10.5 servers.
However the offloads were getting slower over the week and eventually started timing out. Azure storage Explorer was also timing out.
I logged a call with Microsoft as I didn't believe Veeam was at fault and after a couple of calls they sent the log files off to engineering.
On the last call they told us that they had not seen our use case for Azure Databox before ? They said the box was optimised for writing files and not deleting files. Obviously Veeam is deleting the older .vib blocks. Essentially there is a database on the the Azure Databox that holds a list of all the files and it becomes fragmented when files are deleted.
It seems the Azure Databox does not work well with large amounts of files/data when you are also removing files.
Alex
However the offloads were getting slower over the week and eventually started timing out. Azure storage Explorer was also timing out.
I logged a call with Microsoft as I didn't believe Veeam was at fault and after a couple of calls they sent the log files off to engineering.
On the last call they told us that they had not seen our use case for Azure Databox before ? They said the box was optimised for writing files and not deleting files. Obviously Veeam is deleting the older .vib blocks. Essentially there is a database on the the Azure Databox that holds a list of all the files and it becomes fragmented when files are deleted.
It seems the Azure Databox does not work well with large amounts of files/data when you are also removing files.
Alex
-
- Product Manager
- Posts: 14844
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Azure Data Box Issue
Hello,
removal only happens if retention applies. So maybe you could extend retention to stop the deletes?
In general, it's known that Azure Data Box has limited scalability, because it's just one box instead of a classic scale-out system
Best regards,
Hannes
removal only happens if retention applies. So maybe you could extend retention to stop the deletes?
In general, it's known that Azure Data Box has limited scalability, because it's just one box instead of a classic scale-out system
Best regards,
Hannes
-
- Influencer
- Posts: 11
- Liked: never
- Joined: Nov 05, 2009 12:18 pm
- Full Name: AJ GB
- Contact:
Re: Azure Data Box Issue
Thanks Hannes, increasing the retention time may well be a good short term fix.
I was also considering :
1) copying the files to the Data Box in SMB format so there were a lot fewer of them.
2)Then Sending it back to Azure and spinning up a Veeam server in Azure. -
3)Uploading the backups to blob storage.
4) Finally enabling SOBR offload back on premise and letting it synch the changes.
Alex
I was also considering :
1) copying the files to the Data Box in SMB format so there were a lot fewer of them.
2)Then Sending it back to Azure and spinning up a Veeam server in Azure. -
3)Uploading the backups to blob storage.
4) Finally enabling SOBR offload back on premise and letting it synch the changes.
Alex
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Azure Data Box Issue
Am I correct in assuming that you will be:
* deploying a backup repository in Azure for the restore points transferred over Azure Data Box
* adding it as a performance extent to scale-out backup repository
* adding object storage as capacity extent
* letting offload process copy restore points from performance to capacity extent
* switching performance extent to local repository
If so, the described scenario should work fine.
Thanks!
* deploying a backup repository in Azure for the restore points transferred over Azure Data Box
* adding it as a performance extent to scale-out backup repository
* adding object storage as capacity extent
* letting offload process copy restore points from performance to capacity extent
* switching performance extent to local repository
If so, the described scenario should work fine.
Thanks!
-
- Influencer
- Posts: 11
- Liked: never
- Joined: Nov 05, 2009 12:18 pm
- Full Name: AJ GB
- Contact:
Re: Azure Data Box Issue
Yes that's exactly what I was planning. Thankyou for confirming!
Alex
Alex
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Azure Data Box Issue
You're welcome. Let us know how the described process works for you and ask for additional help, if needed. Thanks!
-
- Influencer
- Posts: 11
- Liked: never
- Joined: Nov 05, 2009 12:18 pm
- Full Name: AJ GB
- Contact:
Re: Azure Data Box Issue
In the end I wiped the databox and started again with blob transfer but had extended the retention points on the backups as you suggested.
This worked well and we transferred 20TB in two days at roughly 1.5Gbs a second.
However having shipped the Databox back to Microsoft they are now uploading to the object store at only 1Gbs so that's another 2 days of wait time while my backups on-prem are incrementing all the time.
I had assumed that Microsoft would upload the data in a few hours at 10Gbs.
I'm learning that there are several gotchas with the Data Box.
Alex
This worked well and we transferred 20TB in two days at roughly 1.5Gbs a second.
However having shipped the Databox back to Microsoft they are now uploading to the object store at only 1Gbs so that's another 2 days of wait time while my backups on-prem are incrementing all the time.
I had assumed that Microsoft would upload the data in a few hours at 10Gbs.
I'm learning that there are several gotchas with the Data Box.
Alex
Who is online
Users browsing this forum: No registered users and 7 guests