Tape Backup of MS File Cluster from S3 Compatible (on-site) Object Storage Bloating

[Stupots]

Hi, probably not the most common setup, but we're finding that the 76TB (single .vbk) tape backup of our cluster is bloating when writing to tape. This was working fine when backing up the file from traditional windows ReFS repositories or a copy job on XFS, but not from Object storage (we've decommissioned ReFS and XFS now). The job seemed to be progressing normally up to 76TB when it reached 99%, but then it has carried on at 99% for 7 more days and has currently backed up 195TB! Viewing the file on disk shows 76TB and the cluster only contains 95TB before dedupe at any rate, so we have a weird bloating situation going on.

I currently have a case open which is not progressing much so far: 07428316 and I'm running 12.2 and latest agents 6.2.0.121 and Volume CT Driver 12.0.0.75

Anyone else seeing this?

Thanks

Stu

[Dima P.]

Hello Stuart,

Which B&R version you are running? Thank you!

[Stupots]

Ah yes, I updated the original post with that info, Thanks @Dima

[Dima P.]

Stuart,

Understood. There were some issues with progress time stamps and backup job retries in the v12 which were addressed with 12.1 and 12.2 versions. I recommend to update to the latest v12.2 at your best convenience as job progress and unnecessary task retries (causing extra data processing) were reworked and updated due to similar reports like you've posted. Thank you!

[Stupots]

I'm already on Veeam BR 12.2

[Dima P.]

Oops, sorry for some reason I've got attracted by the driver version. I've asked QA team to review your case details, stay tuned.

[Stupots]

Hi @Dima, This went back in the queue as the original support representative was off, so after a week we're back to asking the same questions, which have already been answered in previous replies to the ticket.

The tapes job is still writing out to tape. The cluster only had 95TB of data (Deduped down to a single 76TB .vbk during the active full disk stage backup), so even if it's "undoing" dedupe, how can the tape backup now be at 235TB? (after 12 days)

[Stupots]

There's definitely some oddness going on in the figures of the original active full. The Total Size/Data Read/Transferred summary appear to be 2x actual figures... Why is it adding the individual node size to the total cluster size, when the total cluster size is the sum of the individual node size, not in addition to it. To reiterate... our resultant .vbk file is 76.2TB in size, we have 98TB of data contained on this cluster out of a total of 155TB is disk provisioned to it.

Below screenshot (parts redacted) shows the active full (bottom) and the first incremental (top)

Thanks

Stu

[Dima P.]

Stu,

Thank you for the details. I've asked support team to escalate the case to T2 support team and investigate the logs. QA team proposed to perform detailed log analysis to understand the root cause.

[Stupots]

Interesting (but maybe not relevant to the issue in this thread) stats completely changed from v12.0 to v12.1. I carried out that upgrade on March 21st; note the before and after "Total Size" in the summary stats and the top line in the Details part for the cluster object… a GUI bug or an actual bug?



Apologies for the poor quality image.

Thanks

Stu

[Dima P.]

Hi Stu,

Unfortunately I cant say - that's why I've asked support team to review the logs. Thank you!

[Stupots]

Hi, my case has been open a month now, with no answers yet and now sitting with QA. I have another monthly full disk stage backup waiting to write out to tape; that will tie up a tape drive for the 14 day duration if I let it run. I understand it is probably a complicated and rare issue, but I would hope to have an update by now.

[Dima P.]

Hello Stuart,

Sorry to hear that, I've asked support folks to share the update. Stay tuned!

[Stupots]

I got a workaround a few days ago. Just waiting to try it out as our Object Storage solution is being rebuilt due to other issues.