Weird error messages during daily forward incremental backup

[massimiliano.rizzi]

Hello experts,

all of a sudden we have been getting the following error messages on a small production environment where backups are being targeted to a Buffalo TeraStation III Rackmount NAS device equipped with 4 x 1 TB SATA hard drives on a RAID 5 3+1 configuration:

==================================================
- "The process cannot access the file because it is being used by another process. Error code: 32 Failed to open file [forward incremental backup file name] in readonly mode"
- "Error: Client error: The specified network name is no longer available. Failed to read data from the file..."
- "Error: Client error: The specified network name is no longer available. Error code: 64 Failed to flush file buffers. File: [forward incremental backup file name]."
==================================================

The Veeam backup server is a virtual machine configured as follows:

==================================================
- Microsoft Windows Server 2008 R2 Standard Edition SP1 running Veeam Backup & Replication 7.0 R2 update Patch #4
- 4x Intel Xeon E5-2620 v2 vCPUs configured as 1 socket and 4 cores
- 4 GB RAM
- Hot-add is the transport mode that is used
- the Veeam primary backup repository is a CIFS share on the Buffalo TeraStation III Rackmount NAS device
==================================================

The weird thing is that we’ve been getting the above errors starting 2 weeks ago after adding a new Windows file server to the list of VMs being backed up. What’s even weirder is that we’re getting the above errors only during the first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completes successfully.

Before adding the new Windows file server on the 4th of July
==================================================
- the list of VMs being backed up comprised 3 VMs (approximately 120 GB full backup file size)
- we were using reversed incremental backup
- no one backup has been missed starting November 2013
- on the 2nd of July we abandoned the existing backup chain and started a new one using forward incremental backup
- no one backup has been missed from the 2nd of July
==================================================

After adding the new Windows file server on the 4th of July
==================================================
- the list of VMs being backed up now comprises 4 VMs (approximately 700 GB full backup file size)
- we are using forward incremental backup with active full every Saturday
- on the 4th of July we added the new Windows file server to the list of VMs being backed up and. As a result, the new file server has been backed up the first time. The job completed successfully during the first scheduled run
- on the 5th of July (Saturday) the weekly active full completed successfully during the first scheduled run
- from the 6th of July (Sunday) until the 11th of July (Friday) we got the above error during each first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completed successfully.
- on the 12th of July (Saturday) the weekly active full completed successfully during the first scheduled run
- from the 13th of July (Sunday) until today we got the above error during each first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completed successfully.
==================================================

Any help would be greatly appreciated.

Thanks and Regards,

Massimiliano

[Gostev]

Hi, Massimiliano. Please include the support case ID for this issue, as requested when you click New Topic. Otherwise, this topic will be removed by moderators eventually. Thank you!

[massimiliano.rizzi]

Hi Gostev,

thank you for your prompt reply.

I will file a support case and report back its ID in a few minutes.

Thanks !

Massimiliano

Edit // Case # 00603666

[Gostev]

Thank you. I will guess here that the issue here is caused by your backup storage misbehaving. Generally speaking, low end NAS devices are the top source of support cases involving backup file corruption or access issue. But let's see what our support says after close review of debug logs. I assume you have already tried rebooting your NAS. Thanks!

[massimiliano.rizzi]

Hi Gostev,

I apologize for the late reply, I have been very busy lately and, as a result, I have been unable to follow up on this issue immediately.

I will guess here that the issue here is caused by your backup storage misbehaving. Generally speaking, low end NAS devices are the top source of support cases involving backup file corruption or access issue.

I fully agree with you. That was my first thought too.

But let's see what our support says after close review of debug logs.

Like you said, the Support Engineer pointed out that the issue may be caused by the Buffalo storage device backup misbehaving especially when used as a SMB repository. The Support Engineer also pointed out that it is not uncommon to have to set an ingestion rate on the repository configuration when using Buffalo storage devices as a SMB repository.

I assume you have already tried rebooting your NAS.

I confirm I have already tried rebooting the NAS. I also have updated it to the latest product firmware, I have initialized it to its factory default and disabled every useless feature.

As part of running through the checklist of KB1735 as requested by the Support Engineer and for performance reasons, I am about to reconfigure the backup job to use a different primary storage that will be responsible for receiving backups from the production VMs, while the Buffalo SMB repository will receive backup copies from the primary backup storage using the BackupCopy feature.

I am thinking of presenting to the Veeam B&R VM a LUN (using a RDM in virtual compatibility mode) large enough to accommodate a backup chain with the number of restore points we want to retain.

Does it make sense to you ?

Thanks again !

[Gostev]

Sure, as long as this LUN is not located on the same storage that runs your virtual machines

And you don't really need vRDM in this case, you can go with pRDM as well.
In-guest iSCSI is also a common approach in case of iSCSI-based storage.

[massimiliano.rizzi]

Hi Gostev,

thank you for your reply.

Unfortunately this LUN is located on the same storage that runs the virtual machines, so I like to just consider it a “helper” primary storage responsible for receiving backups from the production VMs, while the Buffalo SMB repository will receive backup copies from this primary backup storage using the BackupCopy feature right after the the primary backup job completes successfully.

During first run the newly-created Backup Copy job targeted to the Buffalo SMB repository created the first restore point/full backup successfully. From this point on, we'll have to wait and see how it goes with the forever-incremental backup. It has to be said, all weekly active fulls completed successfully during the first scheduled attempt when we were using the Buffalo SMB repository as the primary storage responsible for receiving backups from the production VMs.

Is it correct to say that a BackupCopy job puts less stress on a backup repository rather than a regular backup job ?

Thank you again for your support.

Massimiliano

[foggy]

Is it correct to say that a BackupCopy job puts less stress on a backup repository rather than a regular backup job ?

It's actually the opposite, due to its synthetic nature, backup copy job requires more I/O on the target storage than the forward incremental backup job.

[dellock6]

As always, it depends
Backup Copy jobs uses 2 IO per saved block while running transform operations (the first write operations are 1 IO like forward incremental until it reaches the retention limit and starts transforming).
So, it' less stressful than reversed incremental (3 IO), but more than forward (1 IO).
Let's say it's a great compromise between IO and space savings (it uses only one full backup just like reversed).

[massimiliano.rizzi]

Hi guys,

thank you for your replies.

I'd like to take a few minutes to follow up on this issue after taking a deeper look into it with the Support Engineer.

As part of running through the checklist of KB1735 as requested by the Support Engineer, I have checked the system event log on the proxy server and searched for events with Source=MUP // Event ID: 139 and I confirm there have been events with Source=MUP // Event ID: 139 until a day before we configured a helper repository on local drive and the Backup Copy to the Buffalo storage. After targeting the backup job to a helper repository on a local drive/LUN and the Backup Copy to the Buffalo storage I have seen no errors.

The issue was definitely being caused by the Buffalo storage misbehaving when used as primary storage responsible for receiving backups from the production VMs.

So far so good, let’s see if things will go as well when the Backup Copy will reach the retention limit and will start transforming, thus putting more stress on the Buffalo storage.

Thanks everyone for the help.

Massimiliano

[veremin]

Thank you Massimiliano for taking your time and updating the topic with the found resolution; we do appreciate it!