BackupSync fails if vm disk size increases

[vennerstrand]

Hi,

I think I may have stumbled on a big issue with BackupSync. My original Backup job is working great. When I change vm disk size, I get warnings since CBT cannot be used (naturally). This is just for the first backup after changing size. All following jobs work great with CBT as well.

However, the BackupSync is a different story. I had a running, working BackupSync going, but after changing vm disk size, it failed with:

Error: Unable to initialize remote target session connection: all aliases have failed.
--tr:Failed to open WAN synchronization target.
--tr:Failed to create target task

Digging deeper into the log files at the remote repository:

Code: Select all

stg| WARN|Safe reader ReadBlockEx error.
stg| >>  |The block with the specified ID  [102368] does not exist in file restore snapshot. File: [visma_1-flat.vmdk]. Blocks number: [51200].
cli| ERR |Command 'WanWriteCommand' has failed.
cli| >>  |Cannot allocate memory for an array. Array size: [18446744020057063424].
cli| >>  |--tr:Failed to read block [102368] from restore point
cli| >>  |--tr:Failed to read blocks [102368,...,102399] for update.
cli| >>  |--tr:Wan random writer failed to process WriteEx request.
stg| Metadata were flushed.
cli| Verifying signatures.
cli| WARN|Corrupt blocks were detected. Corrupt blocks count: '51200'.
cli| WARN|DEBUG: corrupt blocks regions: 51200-102399
cli| Verifying signatures. ok.
wani| Corrupt extents were enumerated. [51200] corrupt blocks were detected.

The disk was resized from 50GB to 100GB (50*1024 is... exactly! 51200)..

I created a support case not knowing the above but mentioned that it COULD be vm disk size dependent. I feel that I get no or very little help from support so I wnt to my backup and removed the VM. Rerunning the sync is then successful (disk size is correct since there is no old backup to match)

No,I had to change disk sizes for our file sever (~1TB data) and now the BackupSync has failed in the exact same way. This is a HUGE problem. I must be able to increase the size of the disks in the future without loosing all backup data..

This problem seems similar to the replica job issue when resizing disks. However, this is no replica..

Please HELP. How can i get the BackupSync going and accepting the new disk size?

BR
/Anders

[vennerstrand]

Oh, I use WAN accelerators. (If not apparent from the log above..)

[foggy]

Anders, please provide your support case ID, as it is required according to the forum rules. Thanks.

[Gostev]

Actually, don't worry about the support case ID (this particular time), because we've recently ran into the same issue internally. We already have a hot fix for this issue, and it will be included in the next patch.

[vennerstrand]

This is good new Gostev. Any ETA?

Can I install Patch 1 without making it worse?

BR
/Anders

[vennerstrand]

Sorry for breaking the rules, for completeness, here's the case id: # 00464107.

I liked the forums here before we invested in Veeam, and this makes it even better. A bit disappointed in the official support though. 5 hours to get progress on the forums, 5 days with 3 replies from support without the slightest progress. Just "do an active full" which is not an option for our wan setup..

No more rants. Going from BackupExec to Veeam is a wonderful experience. Now I'm just waiting for the fix..

Thanks!

[Gostev]

I believe you should be able to get the hotfix from our support right now, because we do have it installed in our environment for sure (tell them to talk to me if they have issues locating one). This code will both prevent the issue from appearing AND cleanup the current "mess" automatically for you. You should be able to get it, unless this code contains some other changes and is not yet fully tested (we may have received a pilot hot fix).

And actually, for real issues like this one, official support is usually much faster than forum. The only reason I was able to provide you the response so fast, is because I personally have ran into this issue (I am receiving notifications from our production environment), and drove it through R&D for resolution. Normally though, I would have to defer you to support for them to perform debug log investigation, which for real issues also requires escalating the research into development, so here is your 5 days easily.

[fmt1962]

I presume when stating "backupSync" you are meaning the new Backup Copy job.....???

Anyway, I have found what I believe is the same problem, when I have changed the VM disk size, the backup works fine (CBT not valid for one backup only, then all OK as always)..... However the backup copy job crashes out on this VM.

I have an open case #00466370 for this, but appears to be getting nowhere....
After reading this post, I contacted support and reported what I believe was the same problem and that Gostev says there is a private hotfix available....
Support seem a little baffled and cannot find any confirmed hotfix for this issue.......

Any further info there Gostev????

[Gostev]

Yes, we are talking about the Backup Copy jobs. You can tell them to ask me. We do have a fix installed in our production, but just as I've said, I am just not sure if it is approved for public distribution at this time.

[vennerstrand]

I can confirm that the fix solves the issue. However, a lot of time is spent creating the digest for virtual disks (i assume this is part of the fix).

[veremin]

Most likely, change of disk size has caused digest re-calculation. Thanks.