Host-based backup of VMware vSphere VMs.
Post Reply
massimiliano.rizzi
Service Provider
Posts: 205
Liked: 26 times
Joined: Jan 24, 2012 7:56 am
Full Name: Massimiliano Rizzi
Contact:

Weird error messages during daily forward incremental backup

Post by massimiliano.rizzi »

Hello experts,

all of a sudden we have been getting the following error messages on a small production environment where backups are being targeted to a Buffalo TeraStation III Rackmount NAS device equipped with 4 x 1 TB SATA hard drives on a RAID 5 3+1 configuration:

==================================================
- "The process cannot access the file because it is being used by another process. Error code: 32 Failed to open file [forward incremental backup file name] in readonly mode"
- "Error: Client error: The specified network name is no longer available. Failed to read data from the file..."
- "Error: Client error: The specified network name is no longer available. Error code: 64 Failed to flush file buffers. File: [forward incremental backup file name]."
==================================================

The Veeam backup server is a virtual machine configured as follows:

==================================================
- Microsoft Windows Server 2008 R2 Standard Edition SP1 running Veeam Backup & Replication 7.0 R2 update Patch #4
- 4x Intel Xeon E5-2620 v2 vCPUs configured as 1 socket and 4 cores
- 4 GB RAM
- Hot-add is the transport mode that is used
- the Veeam primary backup repository is a CIFS share on the Buffalo TeraStation III Rackmount NAS device
==================================================

The weird thing is that we’ve been getting the above errors starting 2 weeks ago after adding a new Windows file server to the list of VMs being backed up. What’s even weirder is that we’re getting the above errors only during the first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completes successfully.

Before adding the new Windows file server on the 4th of July
==================================================
- the list of VMs being backed up comprised 3 VMs (approximately 120 GB full backup file size)
- we were using reversed incremental backup
- no one backup has been missed starting November 2013
- on the 2nd of July we abandoned the existing backup chain and started a new one using forward incremental backup
- no one backup has been missed from the 2nd of July
==================================================

After adding the new Windows file server on the 4th of July
==================================================
- the list of VMs being backed up now comprises 4 VMs (approximately 700 GB full backup file size)
- we are using forward incremental backup with active full every Saturday
- on the 4th of July we added the new Windows file server to the list of VMs being backed up and. As a result, the new file server has been backed up the first time. The job completed successfully during the first scheduled run
- on the 5th of July (Saturday) the weekly active full completed successfully during the first scheduled run
- from the 6th of July (Sunday) until the 11th of July (Friday) we got the above error during each first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completed successfully.
- on the 12th of July (Saturday) the weekly active full completed successfully during the first scheduled run
- from the 13th of July (Sunday) until today we got the above error during each first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completed successfully.
==================================================

Any help would be greatly appreciated.

Thanks and Regards,

Massimiliano
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Weird error messages during daily forward incremental ba

Post by Gostev »

Hi, Massimiliano. Please include the support case ID for this issue, as requested when you click New Topic. Otherwise, this topic will be removed by moderators eventually. Thank you!
massimiliano.rizzi
Service Provider
Posts: 205
Liked: 26 times
Joined: Jan 24, 2012 7:56 am
Full Name: Massimiliano Rizzi
Contact:

Re: Weird error messages during daily forward incremental ba

Post by massimiliano.rizzi »

Hi Gostev,

thank you for your prompt reply.

I will file a support case and report back its ID in a few minutes.

Thanks !

Massimiliano

Edit // Case # 00603666
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Weird error messages during daily forward incremental ba

Post by Gostev »

Thank you. I will guess here that the issue here is caused by your backup storage misbehaving. Generally speaking, low end NAS devices are the top source of support cases involving backup file corruption or access issue. But let's see what our support says after close review of debug logs. I assume you have already tried rebooting your NAS. Thanks!
massimiliano.rizzi
Service Provider
Posts: 205
Liked: 26 times
Joined: Jan 24, 2012 7:56 am
Full Name: Massimiliano Rizzi
Contact:

Re: Weird error messages during daily forward incremental ba

Post by massimiliano.rizzi »

Hi Gostev,

I apologize for the late reply, I have been very busy lately and, as a result, I have been unable to follow up on this issue immediately.
Gostev wrote:I will guess here that the issue here is caused by your backup storage misbehaving. Generally speaking, low end NAS devices are the top source of support cases involving backup file corruption or access issue.
I fully agree with you. That was my first thought too.
Gostev wrote:But let's see what our support says after close review of debug logs.
Like you said, the Support Engineer pointed out that the issue may be caused by the Buffalo storage device backup misbehaving especially when used as a SMB repository. The Support Engineer also pointed out that it is not uncommon to have to set an ingestion rate on the repository configuration when using Buffalo storage devices as a SMB repository.
Gostev wrote:I assume you have already tried rebooting your NAS.
I confirm I have already tried rebooting the NAS. I also have updated it to the latest product firmware, I have initialized it to its factory default and disabled every useless feature.

As part of running through the checklist of KB1735 as requested by the Support Engineer and for performance reasons, I am about to reconfigure the backup job to use a different primary storage that will be responsible for receiving backups from the production VMs, while the Buffalo SMB repository will receive backup copies from the primary backup storage using the BackupCopy feature.

I am thinking of presenting to the Veeam B&R VM a LUN (using a RDM in virtual compatibility mode) large enough to accommodate a backup chain with the number of restore points we want to retain.

Does it make sense to you ?

Thanks again !
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Weird error messages during daily forward incremental ba

Post by Gostev »

Sure, as long as this LUN is not located on the same storage that runs your virtual machines ;)

And you don't really need vRDM in this case, you can go with pRDM as well.
In-guest iSCSI is also a common approach in case of iSCSI-based storage.
massimiliano.rizzi
Service Provider
Posts: 205
Liked: 26 times
Joined: Jan 24, 2012 7:56 am
Full Name: Massimiliano Rizzi
Contact:

Re: Weird error messages during daily forward incremental ba

Post by massimiliano.rizzi »

Hi Gostev,

thank you for your reply.

Unfortunately this LUN is located on the same storage that runs the virtual machines, so I like to just consider it a “helper” primary storage responsible for receiving backups from the production VMs, while the Buffalo SMB repository will receive backup copies from this primary backup storage using the BackupCopy feature right after the the primary backup job completes successfully.

During first run the newly-created Backup Copy job targeted to the Buffalo SMB repository created the first restore point/full backup successfully. From this point on, we'll have to wait and see how it goes with the forever-incremental backup. It has to be said, all weekly active fulls completed successfully during the first scheduled attempt when we were using the Buffalo SMB repository as the primary storage responsible for receiving backups from the production VMs.

Is it correct to say that a BackupCopy job puts less stress on a backup repository rather than a regular backup job ?

Thank you again for your support.

Massimiliano
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Weird error messages during daily forward incremental ba

Post by foggy »

massimiliano.rizzi wrote:Is it correct to say that a BackupCopy job puts less stress on a backup repository rather than a regular backup job ?
It's actually the opposite, due to its synthetic nature, backup copy job requires more I/O on the target storage than the forward incremental backup job.
dellock6
VeeaMVP
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Weird error messages during daily forward incremental ba

Post by dellock6 »

As always, it depends :)
Backup Copy jobs uses 2 IO per saved block while running transform operations (the first write operations are 1 IO like forward incremental until it reaches the retention limit and starts transforming).
So, it' less stressful than reversed incremental (3 IO), but more than forward (1 IO).
Let's say it's a great compromise between IO and space savings (it uses only one full backup just like reversed).
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
massimiliano.rizzi
Service Provider
Posts: 205
Liked: 26 times
Joined: Jan 24, 2012 7:56 am
Full Name: Massimiliano Rizzi
Contact:

Re: Weird error messages during daily forward incremental ba

Post by massimiliano.rizzi » 2 people like this post

Hi guys,

thank you for your replies.

I'd like to take a few minutes to follow up on this issue after taking a deeper look into it with the Support Engineer.

As part of running through the checklist of KB1735 as requested by the Support Engineer, I have checked the system event log on the proxy server and searched for events with Source=MUP // Event ID: 139 and I confirm there have been events with Source=MUP // Event ID: 139 until a day before we configured a helper repository on local drive and the Backup Copy to the Buffalo storage. After targeting the backup job to a helper repository on a local drive/LUN and the Backup Copy to the Buffalo storage I have seen no errors.

The issue was definitely being caused by the Buffalo storage misbehaving when used as primary storage responsible for receiving backups from the production VMs.

So far so good, let’s see if things will go as well when the Backup Copy will reach the retention limit and will start transforming, thus putting more stress on the Buffalo storage.

Thanks everyone for the help.

Massimiliano
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Weird error messages during daily forward incremental ba

Post by veremin »

Thank you Massimiliano for taking your time and updating the topic with the found resolution; we do appreciate it!
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 87 guests