{Transform.Patch} failures

[NetBSS]

We have 2 remote servers which we backup with the Agent for Windows (free) to locally connected USB disks. (The important data is replicated via DFSR and backed up elsewhere, so we use just a single disk for each server.) The disks are currently connected to the servers, but the plan has been to eventually share them off another server so we reference them in the backup job via UNC path. These servers are essentially identical and hosting almost identical data. Both servers have occasionally failed with {Transform.Patch} errors, e.g.,

12/5/2019 11:29:22 PM :: Full backup file merge failed Error: Agent: Failed to process method {Transform.Patch}: An unexpected network error occurred.

but the backups have eventually succeeded after some number of retries. (Again, the destination is a locally connected disk accessed via UNC path.)

Recently one of the servers has been logging these errors much more often than other, and for the first time the Thursday backup there gave up after failing 48 successive retries.

SMART tests of the WDC USB backup disks have shown no errors.

We opened Support Case 03869739 but it was archived without getting any response.

The logs refer to "bad allocation" which suggests a resource issue, but what resource?

Thanks.

[Adrian1980]

Have you done SMART und FS-Check on the source?

[NetBSS]

No SMART on the source, as it's a RAID-10 array, and no FS check (until now) on the source as there's been no indication of problems on the source: the source snapshot succeeds and the incremental backup is successfully created. It's the subsequent merge which is failing.

But to confirm: I just did a r/o chkdsk on both volumes and no issues were found.

Until the recent failure the backups have eventually succeeded despite initial {Transform.Patch} errors. This also indicates to me that the backup itself is ok, but the merge process is running into intermittent (and increasing) resource issues.

But what resource? I just watched retry #29 (for last night's backup) fail, a minute or two after the merge reached 79% complete:

• 5.3GB of the 8GB total RAM was available
• paged and unpaged pool together were under 800MB
• the 4TB backup disk has 1.5TB free

[Dima P.]

Hello NetBSS,

Mind me asking, if its possible to use USB drive as a regular USB target (with Veeam Agent for Windows job settings). I'd recommend to use the local disk in such case and we eliminate the middle man (i.e. SMB protocol which may cause disconnects and time outs)? You still can use UNC as an external resource. Thanks!

[NetBSS]

Perhaps configuring as a local disk is the next step, but the question is why we are having problems on this server but no problems on another essentially identical server (not to mention many other servers also configured to backup to a local disk via UNC.) Or more important, how can we troubleshoot this error, which has become very reproducible?

Will changing the destination type will break the backup chain and require starting a new backup?

Thanks.

[NetBSS]

As the backup has continued to fail I tried changing the destination to the equivalent local path. Other job settings were retained.

The backup and merge succeeded, though the job is still doing a health check. The backup chain was retained, and a normal size incremental backup created for last night.

The good news is that the backup has succeeded. But it would be helpful to be able to troubleshoot why the UNC path method, which is working on other servers and had been working on this server for some time, suddenly stopped working.

Thanks.

[Dima P.]

Hello NetBSS,

Thank you for sharing the update - glad to hear that you are up and running. SMB issues might be related to firewall configuration, routing problems or connectivity issues (all these issues are still relevant eventhough you are using the locally mounted drive shared over smb). Please raise a support ticket if you want this issue to be investigated and do not forget to share the case ID. Cheers!