Veeam errors after about 72 hours into running a backup job

[Squeezer]

I am running Veeam 6.5 for Hyper-V. I am attempting to backup some rather large virtual servers over a T-1 line. After about 72 hours the backup jobs always fails. Is there a limitation within veeam that a job cannot run more than 72 hours? Here is the error:

[Gostev]

I am not aware of such limitation. Please open a support case for investigation, and include support case ID in this topic for further reference.

[ravelin]

I have the exact same error. 72 hours and a couple of minutes.

Code: Select all

METZFIL	Error	14:08:39	14:10:55 (+3)	1,2 TB	27,1 GB	12,2 GB	72:02:15 	Error: Client error: The device is not ready. Failed to read data from the file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1519\Hyper-V\Virtual Hard Disks\METZFILE_D.vhd]. Failed to read data from the file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1519\Hyper-V\Virtual Hard Disks\METZFILE_D.vhd]." Unable to retrieve next block transmission command. Number of already processed blocks: [4704744].

Then tried Again on NeXT Schedule:
METZFIL	Error	02:46:44	02:49:03 (+3)	1,2 TB	27,1 GB	12,2 GB	72:02:18 	Guest is still being processed by the previous session
Error: Client error: The device is not ready. Failed to read data from the file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1533\Hyper-V\Virtual Hard Disks\METZFILE_D.vhd]. Failed to read data from the file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1533\Hyper-V\Virtual Hard Disks\METZFILE_D.vhd]." Unable to retrieve next block transmission command. Number of already processed blocks: [4562044].

[Squeezer]

I am not aware of such limitation. Please open a support case for investigation, and include support case ID in this topic for further reference.

I have opened support case 00199309.

[Gostev]

To me, both errors look like dropped network connection between proxy and repository.

[Amyd]

Any progress on this issue, maybe a workaround? I have (almost) the same behaviour: long backup of a large VM (in my case 24 TB with 17 TB used), on a CSV, fails regularly at about 72 hours with a similar error to those mentioned above.

OE-VM-DATEN-02 Error 17:39:55 17:41:59 (+3) 23,6 TB 0,0 KB 0,0 KB 72:02:04 Error: Client error: Der Datentr�ger einer Datei wurde extern so ge�ndert, dass die ge�ffnete Datei nicht mehr g�ltig ist. Error code: 1006 Failed to read data from the file [\\?\GLOBALROOT\Device\CSV{890ba7f8-1a2c-4473-99ff-e4652ed9bb2c}\HyperV\Virtuelle Computer\OE-VM-Daten-02\OE-VM-Daten-02-LW_D.vhdx]. Failed to read data from the file [\\?\GLOBALROOT\Device\CSV{890ba7f8-1a2c-4473-99ff-e4652ed9bb2c}\HyperV\Virtuelle Computer\OE-VM-Daten-02\OE-VM-Daten-02-LW_D.vhdx].') Unable to retrieve next block tra

From a previous run:

OE-VM-DATEN-02 Error 19:11:18 19:18:03 (+3) 23,6 TB 0,0 KB 0,0 KB 72:06:44 Error: Client error: Der Datentr�ger einer Datei wurde extern so ge�ndert, dass die ge�ffnete Datei nicht mehr g�ltig ist. Error code: 1006 Failed to read data from the file [\\?\GLOBALROOT\Device\CSV{8754bc4c-d5e1-4221-853d-313c143d484e}\HyperV\Virtuelle Computer\OE-VM-Daten-02\OE-VM-Daten-02-LW_D.vhdx]. Failed to read data from the file [\\?\GLOBALROOT\Device\CSV{8754bc4c-d5e1-4221-853d-313c143d484e}\HyperV\Virtuelle Computer\OE-VM-Daten-02\OE-VM-Daten-02-LW_D.vhdx].') Unable to retrieve next block tra

I doubt it is network connectivity, in my case it is not a WAN backup, but a local LAN backup.

[ravelin]

I moved my backup server to the lan of the big machine, and got a full backup in less than 72 hours.
I then moved it back to the original location. I never got it fixed, and renegotiation of the VPN tunnel takes place much more often than 72 hours. I doubt it was a connectivity problem.
If it is, than it must be Veeam or MS. Maybe max of a SMB session?

[veremin]

According to the OP’s ticket, support team has developed a fix for him that increases this interval from 72 to 168 hours. So, I strongly recommend you contacting them in order to get the aforesaid solution.

Thanks.

[ravelin]

Ok, would have been nice of him to inform the thread viewers of a fix
I will ask for it, if we get the problem on other customers.

[veremin]

I believe the reason that OP has never come back was that the proposed solution didn’t work for him, since in his case a given job couldn’t finish even in 168-hour period. Thanks.

[Squeezer]

Veeam technical support sent me a patch which increased the backup timeout to 168 hours. That still was not enough time. Veeam support replied that a limitation in Microsoft C# prevented them from creating a job that could run more than 168 hours.

So, instead I am using Microsoft DFS to push the data to the off-site servers for disaster recovery.

[Gostev]

It is quite scary to run VM off snapshots for over 168 hours in the first place.
However, this seems like a perfect use case for Backup Copy functionality of v7.